Last week, at the London .NET User Group meeting, Ian Cooper talked about Test-driven development, focusing on both good and bad practices. I’m a big fan of learning from anti-patterns and mistakes of other people, so the second part of his session was very interesting to me. Here is a short list of things that Ian identified as symptoms that TDD has gone bad in a project, along with my comments:
- Disabling tests to make a build pass:
If the build is failing because of a test, developers disable the test to make the build pass. This is a bad practice because tests become irrelevant or get lost — people don't remember to fix and enable them later. If the test is deprecated, then delete it completely. If it is not deprecated, don't disable it but make it pass.
- Continuous integration not failing when unit tests fail:
The point of continuous integration is to automate problem checking and prevent a big-bang integration before a release. Broken unit tests should raise an alarm and get fixed before problems pile up. If the CI server does not break the build when a unit test fails, then CI configuration must be changed.
- Not monitoring customer tests:
Ian put integration and acceptance tests under the "customer tests" group. These tests don't break the build because they will not pass for most of the development, but they still might go down from 30% to 20%, for example. That is a sure sign that something bad happened, yet if nobody is monitoring the reports, this will again lead to a big bang integration on the end. In my eyes, integration tests and customer tests should be split into two parts: the first one (integration) should break the build, and the second one (acceptance) should not, but it should still be monitored.
- UI change causes tests to fail (interface sensitivity):
If tests depend on the UI heavily, then they will be brittle and hard to maintain. I wrote about this earlier in Effective User Interface Testing.
- (3rd party) API changes cause tests to fail:
If changes to 3rd party APIs propagate to tests, then tests are again hard and expensive to maintain. I guess that the bigger underlying problem here is that the business logic is not isolated properly from 3rd party libraries.
- Many tests fail when a single behaviour changes:
This applies to unit tests, and signals that tests are not properly granulated and focused on code units, but try to test too much. Again, the issue arising from this is high cost of test maintenance.
- Data-sensitive tests:
Tests that depend on some data pre-conditions (such as certain records existing in the database) are also brittle and will break when the data changes. Test harness should ideally set up all the pre-conditions for a test. A telling sign of this are tests that use hard-coded database IDs.
- (Shared) Context Sensitivity:
If tests depend on other tests to set up the context, then the order of test execution becomes important and you can no longer run individual tests in isolation. This can lead to big problems, especially if the test runner does not guarantee the order of tests. Again, the test harness should ideally set up all the pre-conditions for a test and individual tests should be independent.
- Conditional test logic:
The telling sign of this problem is that tests choose validations based on run-time context (if (...) test this... else test that...). This signals that the test is not clearly focused on a single thing, and that the author does not really understand what he is testing.
A quick summary
Tests that break without anyone reacting do not prevent problems from piling up. This defeats the point of tests being a traffic light that keeps the problems small and prevents a big-bang integration on the end. When tests fail, alarm bells should ring.
High maintenance cost of tests defeats the whole point of having them, as a way to guarantee that code changes are easy and cost-effective. Keep unit tests independent and focused on a single code unit. Then they will be easy to maintain and will support change rather then inhibit it.
