Why Too Many UI Tests Prevents Change

Feeds

Monday, 3rd September 2012

Having seen attempts to reduce the time and cost of testing in many companies, I have had the opportunity to learn a great deal from other people's mistakes.

The first lesson I learned is that unit testing is absolutely essential. Not only does it give you the confidence to make large-scale refactorings knowing that you haven't changed the expected behavior, but using test-driven methods can actually lead you to better code structures in the first instance. As long as you follow a strict pattern of creating very easy unit tests, you can ensure that all calling code can use the class under test easily. The first sign of poor code is a complicated unit test as it suggests that your class has too many dependencies, too many parameters, is doing too much or is too complex to use.

The second lesson I learned is that integration tests are really very useful indeed. Despite having 100% unit test coverage, there is no guarantee that your end-to-end integration is going to work. My favorite integration test was actually built into the product. A special administration page allowed you to kick off the integration test, which called down through each layer to ensure it could get a response from each system and component. If there were any problems, the exact integration point was reported back. A simple UI test was used to run this test automatically, but it could also be used to determine the source of any outage by running it manually.

I have been exposed to UI testing for years. From QTP in a large financial organisation to Selenium at a start up. The problem in all of these companies is the number of UI tests. Very often, UI tests are created to reduce the manual effort of testing a release. This is certainly the case in large waterfall projects where the test cycle is measured in weeks and had to be repeated in full for each bug-fixing release. The instinct was to simply automate what was already being done. A better first step would have been to break the monolithic product into smaller deploy-able elements, especially where those elements are entirely unrelated to the rest of the product as so many often are. This means the test plan only needs to cover what is being deployed, rather than the entire product. Of course there needs to be automation, but where possible the business rules should be automated in unit tests and only a very small amount of testing should be performed by automating the UI.

When you grow a large set of UI tests, it makes it very hard to change your product. Once the large financial product was partially covered by UI tests, each time anything on the UI was moved or renamed it would invalidate the look-ups used to find the UI elements. Tests would fail when UI elements were renamed to clarify their purpose, or to correct spelling errors and when something was moved to improve usability. It increased the levels of paranoia because small changes would have a large knock-on effect, even though the only thing that was broken was the testing strategy.

If you want to automate what the testers are doing, they should be paired with developers in order to convert their boilerplate regression tests into unit tests and sometimes integration tests and very rarely, a UI test.