Building and maintaining automated tests can often be challenging and cumbersome.
In fact, the term “flaky” is often associated with tests — especially tests created at the UI layer.
Even the smallest amount of changes in the UI can cause automated tests created at the UI level to break, resulting in a lot of maintenance overhead. This problem hasn’t gotten any easier with the explosion of browsers, mobile, third-party controls, and other IoT devices.
Hence, unless there is a solid test automation strategy, UI tests can result in false negatives — a situation where UI a test fails on a proper code. Or even false positives — a situation where a UI test passes even on a bad code.
Deciding whether or not to file a bug in such cases often involves a lot of time spent debugging and figuring out what’s wrong with the UI test.
Do you have a problem with flaky tests?
It’s common knowledge that flaky tests shouldn’t be tolerated. A test with a pass rate of 99.5% often looks impressive, but is it? In fact, a 99.5% pass rate can look less extraordinary when looked across a suite containing 300 tests. Take the following for example:
- A test suite has 300 tests
- Each test has a .5% failure rate
- Pass rate for each tests: 99.5%
- The pass rate for the entire suite in this scenario is: (99.5%)^300 = 22.23%
As a result, a 99.5% pass rate for a particular test comes to be a modest >25% when looked across the suite.
The bottom line here is that testing teams cannot, and should not, just look at the high rate of pass percentage for individual tests. You instead need to look at the entire test suite.
There are other reasons why flakiness shouldn’t be tolerated and how flaky tests can be overcome.
That’s why we created this eBook: Managing Test Flakiness: How to Make Tests Stable, Scalable, & Maintainable
In this eBook, we’ll answer some important questions:
- Root causes of flaky tests
- Solutions to test flakiness
- Flaky vs. Healthy Tests: How to Find a Balance
- Tools to scale automated tests
- How to separate signal from noise when looking at flaky tests