Long term maintenance cost is one of the biggest issues that teams face today when implementing agile acceptance testing. Tests that are just written and automated without any long term planning are guaranteed to cost you more than they are worth. But then again, a properly designed testing framework saves a lot of money, time and effort in the long run. It seems that the community is now going through the same learning cycle as we went through with unit tests, with people writing any crap code in unit tests at first, then learning that testing code maintenance hurts as much as it would hurt for normal code, and cleaning up their act. The ongoing research for my new book has helped me understand that, in the case of acceptance tests, the problem is much deeper.
At the recent open space session on the future of acceptance testing at Open Volcano 10, we discussed treating test code as second class as one of the biggest problems with agile acceptance testing implementation and one of the biggest challenges for teams in the future. It also appeared on the top 10 reasons why teams fail with acceptance testing at CITCON Europe 09. Long term test maintenance costs also sparked a recent discussion on tools in the blogosphere (see Jim Shore’s original article and responses from George Dinwiddie, Ron Jeffries and my reply and comments on those posts). I now believe that most of these issues come from the same problem.
An acceptance testing framework is a product on its own
Although most of them don’t realise it themselves, the most successful teams I interviewed actually treat their acceptance testing framework as a product in its own right. I’m not talking about the automation tool – though some of the teams have developed their own tools or extended opensource tools significantly – but about the test specifications, the infrastructure around them and the integration layer for automation (fixtures, step definitions, keyword implementations).
Many teams have ended up designing a domain specific language for specifications and tests (implemented using FIT, Cucumber, Robot Framework or even directly in code), structuring their acceptance tests around it to keep them consistent, easy to understand and minimise long term maintenance costs. When retrofitting acceptance testing into already existing products, they often had to make the system and the architecture more testable, which required senior people to plan and implement serious design changes. Implementing agile acceptance testing successfully, and getting the most benefits out of it, requires a significant investment in all of this. It requires structure, design and planning. It requires thinking about how to maximise the return on investment in the framework. It requires on-going cultivation and maintenance.
If we think about tests as by-products of the development process, this investment is very hard to justify because it doesn’t give direct value to any of the project stakeholders. The stakeholders of the framework, however, aren’t the customers or the business users. They are the members of the development teams (and maintenance teams if separate). Many teams I interviewed have at some point ripped out the heart of their system and replaced it, or rewritten the entire system, while keeping their acceptance tests and using them to guide the whole effort. This is where the investment in live documentation really pays off. Such a framework is genuinely a separate product, with a different lifecycle and a different group of stakeholders.
Understanding that the acceptance testing framework is actually a product resolves a ton of things I’ve considered genuine problems in the past, but now look at them as just symptoms of this deeper issue. For example, once you start thinking about the framework as a product, the question of whether to put the tests in a version control system or not goes away immediately. Cleaning up “test code” no longer requires a separate explanation. Working on enhancing the structure or clarity of tests is no longer something to put on the technical debt list – it is part of a backlog for a completely separate product. The flaw in delegating the work on acceptance tests to junior developers and testers suddenly becomes obvious. On the other side of this equation, looking at the testing framework as a separate product also prevents teams from falling into the trap of goldplating the tests at the expense of the primary product.
|
|
Just two days ago, I met another fallacy that testers new to Acceptance Testing are facing. I called it the Second System Effect in Software Test Automation:
http://blog.shino.de/2010/04/27/the-second-system-effect-in-software-test-automation/
In short, most often when I see testers coming up with a way to automate their tests, they start to build a second system – a system similar to the one they are going to test.
Treating the automation code (fixtures, …) as a separate product, thereby becomes a second implementation of the system under test – with all the problems regarding maintenance of the two systems. Your blog provides the right cure on it.
Gojko – It’s interesting you bring this up. Markus Gärtner blogged about “The Second System Effect in Software Test Automation” yesterday, and mentioned in the past he has found a second system being developed along side the primary software project, in the form of a test system. If due care is not taken, then a change in the primary system can leave the test system in a non-useful state and in need of serious changes.
The key point which you have both made is to be aware that the test automation project is just as important as the primary project, and the same software development rules should be applied to both – otherwise they will both suffer from the lowest common denominator effect.
See Markus’s blog post – http://blog.shino.de/2010/04/27/the-second-system-effect-in-software-test-automation/
Out of curiosity, would you recommend practising TDD and ensuring good unit test coverage of your acceptance test framework (assuming it’s something you develop yourself) – in the same way that you would expect it on the primary project?
Depending on how complicated it is. The last place where I introduced FIT required a lot of custom fixtures to work with their reporting frameworks. We developed those with TDD and covered with unit tests. Simpler fixtures weren’t covered by unit tests.