Last week, at the London .NET User Group meeting, Ian Cooper talked about Test-driven development, focusing on both good and bad practices. I’m a big fan of learning from anti-patterns and mistakes of other people, so the second part of his session was very interesting to me. Here is a short list of things that Ian identified as symptoms that TDD has gone bad in a project, along with my comments:
If the build is failing because of a test, developers disable the test to make the build pass. This is a bad practice because tests become irrelevant or get lost — people don't remember to fix and enable them later. If the test is deprecated, then delete it completely. If it is not deprecated, don't disable it but make it pass.
The point of continuous integration is to automate problem checking and prevent a big-bang integration before a release. Broken unit tests should raise an alarm and get fixed before problems pile up. If the CI server does not break the build when a unit test fails, then CI configuration must be changed.
Ian put integration and acceptance tests under the "customer tests" group. These tests don't break the build because they will not pass for most of the development, but they still might go down from 30% to 20%, for example. That is a sure sign that something bad happened, yet if nobody is monitoring the reports, this will again lead to a big bang integration on the end. In my eyes, integration tests and customer tests should be split into two parts: the first one (integration) should break the build, and the second one (acceptance) should not, but it should still be monitored.
If tests depend on the UI heavily, then they will be brittle and hard to maintain. I wrote about this earlier in Effective User Interface Testing.
If changes to 3rd party APIs propagate to tests, then tests are again hard and expensive to maintain. I guess that the bigger underlying problem here is that the business logic is not isolated properly from 3rd party libraries.
This applies to unit tests, and signals that tests are not properly granulated and focused on code units, but try to test too much. Again, the issue arising from this is high cost of test maintenance.
Tests that depend on some data pre-conditions (such as certain records existing in the database) are also brittle and will break when the data changes. Test harness should ideally set up all the pre-conditions for a test. A telling sign of this are tests that use hard-coded database IDs.
If tests depend on other tests to set up the context, then the order of test execution becomes important and you can no longer run individual tests in isolation. This can lead to big problems, especially if the test runner does not guarantee the order of tests. Again, the test harness should ideally set up all the pre-conditions for a test and individual tests should be independent.
The telling sign of this problem is that tests choose validations based on run-time context (if (...) test this... else test that...). This signals that the test is not clearly focused on a single thing, and that the author does not really understand what he is testing.
Tests that break without anyone reacting do not prevent problems from piling up. This defeats the point of tests being a traffic light that keeps the problems small and prevents a big-bang integration on the end. When tests fail, alarm bells should ring.
High maintenance cost of tests defeats the whole point of having them, as a way to guarantee that code changes are easy and cost-effective. Keep unit tests independent and focused on a single code unit. Then they will be easy to maintain and will support change rather then inhibit it.
Get practical knowledge and speed up your software delivery by participating in hands-on, interactive workshops:
Get future articles, book and conference discounts by e-mail.