By using better names, we can avoid a lot of completely meaningless discussions and get people started on the right path straight away. So here is what I propose:

One of the biggest issues teams have with acceptance testing is who should write what and when. So we need to come up with a good name for the start of the process that clearly says that everyone should be involved, and that this needs to happen before developers start developing and testers start testing, because we want to use acceptance tests as a target for development. Test first is a good technical name for it, but business users don’t get it. I propose we talk about Specifying Collaboratively instead of test first or writing acceptance tests.

It sounds quite normal to put every single numerical possibility into an automated functional test, why wouldn’t you do it if it is automated. But such complex tests are unusable as a communication tool. So instead of writing functional tests, let’s talk about Illustrating with Examples (thanks, Dave) and expect the output of that to be Key Examples to point out that we only want enough to explain the context properly.

Key examples are raw material, but if we just talk about acceptance testing then why not just dump all those complicated 50-column 100-row tables into an acceptance test without any explanation, it’s going to be tested by a machine anyway. But these tests are for humans as well for machines, so let’s talk about a whole new step after this, about the process of extracting the minimal set of attributes and examples to specify a business rule, and about adding a title, description and so on. I propose we call this Distilling the specification.

I just don’t want to spend any more arguing with people who already paid a license for QTP that it is completely unusable for acceptance tests. As long as we talk about test automation, there is always going to be a push to use whatever horrible contraption testers already use for automation, because it’s logical to use a single tool. Acceptance testing tools don’t compete with QTP or things like that, they address a completely different problem. Instead of talking about test automation, let’s talk about automating a specification without distorting any information – Literal Automation. Literal automation also avoids the whole scripting horror and using technical libraries directly in test specifications. If it’s literal, it should look as it looked on the whiteboard, it should not be translated to selenium commands.

After Literal Automation, we get a specification that can be checked against the code automatically, an Executable Specification.

We want to run all the acceptance tests frequently to make sure that the system still does what it is supposed to do (and equally more important to check that the specification still says what the system does). If we call this regression testing, it’s very hard to explain to testers why they should not go and add five million other test cases to a previously nice, small and focused specification. If we talk about continuous integration, then we get into the trouble of explaining why these tests should not always be end-to-end and run the whole system. On the top of that, for some legacy systems we need to run acceptance tests against a live, deployed environment. Technical integration tests run before deployment. So let’s not talk about regression testing or continuous integration, let’s talk about
Continuous Validation (or even Frequent Validation).

The long term pay-off from agile acceptance testing is having a reference on what the system does that is as relevant as the code itself, but much easier to read. That makes development much more efficient long term, facilitates collaboration with business users, leads to an alignment of software design and business models and just makes everyone’s life much easier. But to do this, the reference really has to be relevant, it has to be maintained, it has to be consistent with itself and with code. we should not have silos of tests that use terms we had three years ago, and those we used a year ago, and so on. Going back and updating tests is a very hard thing to sell to teams who are busy, but going back to update documentation after a big change is expected. So let’s not talk about folders filled with hundreds of tests, let’s talk about a Living Documentation system. That makes it much easier to explain why things should be self-explanatory, why business users need access to this as well and why it has to be nicely organised so that things are easy to find.

What do you think about this? Does it make sense? Will it help? Do you have a better name for one of these concepts, that explains it more clearly?

The problem is that manual and automated checks are affected by a completely different set of constraints. With manual testing, the time spent preparing the context is often a key bottleneck. With automated testing, people spend most time on understanding what is wrong when a test fails.

For example, to prepare for a manual test that checks user account management rules, you might have to log on to an administrative application, create a user, log on as that new user to the client application, and change the password after first use. To avoid doing this several times during the test, manual scripts often reuse the context. So you would create the user once, block that account and verify that the user cannot log on, reset the password to verify that it is re-enabled, then set some user preferences and verify they change the home page correctly. This approach helps a tester run through the script quicker.

With automated testing, time spent on setting up the user is no longer a problem. Automated tests generally go through many more cases than manual tests, and when they run correctly nobody is really looking at them. Once a test fails, someone has to go in and understand what went wrong. If a test is described as a sequence of interdependent steps, it will be very hard to understand what exactly caused the problem, because the context changes throughout the script. The fact that a single script is checking ten different things also makes it more probable that the test will fail because it is affected by lots of different areas of code. In the previous example with user account management, if the password reset function stops working, we won’t be able to set the user preferences correctly. If we had ten different, smaller, focused and independent tests instead of one big script, a bug in the password reset function won’t affect the test results for user preferences. That makes tests more resilient to change and reduces the cost of maintenance. It also helps us pin-point the problems quicker.

Instead of plainly automating manual test scripts, think about what the script is testing and describe that with a group of independent, focused tests. This will significantly reduce the automation overhead and maintenance costs.

The five most important things to think about are:

• It needs to be self explanatory
• It needs to be focused
• It needs to be a specification, not a script
• It needs to be in domain language
• It needs to be about business functionality, not about software design

Here is a really good example:

It has all the elements listed above. Whenever I’ve shown it to people in the workshops, I did not have to use a single word to explain it. The title and the introductory paragraph explain the structure of the test data enough that readers don’t need to work back from the data to understand the rule. But the examples are there to make it actually testable and explain the behaviour in edge cases. It is focused on a very particular rule of free delivery availability, does not explain how the books are purchased but just what the available delivery mechanism is, and does not try to talk about any implementation specifics.

Here is a really bad example (taken from http://fitnesse.org/FitNesse.UserGuide.PayrollTests.AddAndPayTest)

This test has so many bad things in it that it is actually a good example to demonstrate what happens when people don’t take care about writing good tests.

First of all, although it has a title and some text around the tables to seemingly explain what’s going on, the effect of that is marginal. Why is this test “simple”, and what exactly in payroll is it checking? It fails straight away on the self-explanatory litmus-test.

Second, it’s not really clear what this test is checking. We need to work backwards. It seems to verify that the cheques are printed with unique numbers, starting from the next available number that’s configurable in the system. It also seems to validate the data printed on each cheque. And that there is one cheque printed per employee (I’ll come back to this later).

There is a lot of seemingly incidental complexity there – names and addresses aren’t really used anywhere in the test apart from setting up the employees. There are some database IDs there which are completely irrelevant for the business rules, but they are used to match up employees and the paycheck inspector.

Paycheck inspector is obviously an invented thing just for testing. No company is going to have Peter Sellers in a Clouseau outfit inspecting cheques as they go out. If you have enough employees to have to print cheques, you don’t want to inspect them manually. That’s what this test is about, anyway.

There is also a very interesting issue of those blank cells in the assertion part of the test, and the two Paycheck inspector tables which seem unrelated. Blank cells in Fitnesse are used to print test results for debugging and troubleshooting, they don’t check anything. So this is an automated test that a human has to look over — pretty much defeating the purpose of automation. Blank cells are typically a sign of instability in tests (more on this a bit later) and they are often a signal that you’re missing something – either testing in the wrong place or missing a rule that would help make the system process repeatable and testable.

The language is inconsistent, which makes it hard to make a connection between inputs and outputs at first. What is the 1001 value in the table below? The column header tells us that it is a number – well thanks for that, I though it was a sausage. There is a “cheque number” above, but what kind of a cheque number is that? What is the relationship between these two things?

So this test is very very bad.

Presuming that the addresses are there because cheques are printed as part of a statement with an address ready for automated envelope packaging, this test fails to check at least one very important thing (and I’ll come back to this later as well): that the right people got paid the right amounts. If the first person got both cheques, this test would happily pass. If they both got each-others salaries, this test would pass. If a date far in the future was printed on the cheque, our employees might not be able to cash it in but the test would still pass. The reason these cells are blank is because there is another hidden rule there: ordering of cheques coming out. There is nothing specifying that. So a technical workaround for a functional gap is to create a test that gives us loads of false positives.

Is this test checking one thing or many things? Without the context information it’s hard to tell. If the cheque printing system is used for anything else, I’d pull out the fact that cheque numbers are unique and start from a configured number into a separate page. If we only ever print salary cheques, it’s probably part of a single thing (salary cheque printing).

Now, let’s clean it up. Let’s try to work back and drop all the incidental stuff. Let’s use a nice descriptive title, such as “Payroll Cheque Printing”. And let’s add a paragraph that explains the structure of the test.

A cheque has the payee name, amount, payment date. A cheque does not have a name and a salary. If the cheque is printed on a statement, it also has an address that will be used for automatic envelope packaging. A combination of a name and address should be enough for us to match the employee with his cheque — we don’t really need the database IDs. We can make the system more testable by agreeing on an ordering rule, whatever it is. For example, alphabetic by name.

Let’s also pull the context to the start of the test. Our context is the payroll date and the next available cheque number, along with employee salary data. Let’s make it explicit what the number is for, so that the readers don’t have to figure this out for themselves in the future. We can also make this block stick out visually to show that it is about the context.

The action that gets kicked off doesn’t necessarily need to be listed in the test. A payroll run can be executed implicitly by the table that checks payroll results. This is an example of focusing on what’s being tested instead of how it is being checked. There is no need to have a separate step that says “Next we pay them”.

Let’s also rename that paycheck inspector to something that makes more sense. Because we want whoever automates this to ensure that we check for all cheques printed, let’s put that in the header. Otherwise, someone might use subset matching and the system might print every cheque twice and we won’t notice.

And here is the cleaned version.

Much easier to understand, without all that incidental stuff. And now comes the punchline. When we look at the test like this, without database IDs, without all the unnecessary clutter, we can have a nice shot at answering the question “Are we missing something?”. What are the edge cases that might break this? We don’t really need to ensure validity of employee data, hopefully that’s done in another part of the system. But is there any kind of valid employee data that would be an edge case for this test – can we play with the numbers to make this illogical in some way?

An obvious answer to that is – what happens if an employee has a salary of 0? Do we still print the cheque? The rule as we described it says “One cheque per employee” – so any employees that have been fired years ago and no longer receive salaries would still get cheques printed, with zeroes on them. We could then have a discussion with the business on making this rule stronger and ensuring that cheques don’t go out when they don’t need to do.

FitNesse gets a lot of bad reputation because of this kind of broken tests. Concordion was built as an answer. Some people at my recent workshop on this example pointed out that Given-When-Then structure of Cucumber would be better at preventing some of these problems. I don’t think so. It’s not about the tools – people can do this kind of bad test design with any tool, and likewise they could do nice and clean tests with FitNesse. I guess the fact that the basic examples coming with FitNesse are so bad doesn’t help, but the problem is not in the tool. It’s about the process and effort put into making the tests easy to understand. The funny thing is it doesn’t take a lot more effort to make the test nice and clean, but it brings a lot more value that way.

If you found this article helpful, you’ll love my three day workshop on effective specification by example and agile acceptance testing

We ran a variant of the Product Box game, described by Luke Hohmann in Innovation Games: Creating Breakthrough Products and Services, with five teams competing to design the best agile test management tool. They gave their products the following names: Silver Bullet, Agile Manager, Perfect 10, Nimble and FleXT.

Getting all kinds of different tools to talk to each other seems to be the biggest problem in the community at the moment. Instead of a monolithic all-in-one solution, all groups went for integration with other tools. Perfect 10 has “extensive integration with unit test tools, build tools, source code repositories”. Agile manager is able to “pull the requirements from whatever the requirements management tool you are using”. FleXT even integrates with third party defect management systems. The ability to pull the data out of a management tool also seemed very important, with FleXT offering REST and MS office interfaces and Nimble having a “unique universal adapter”. With such a wide range of integrated collaborators, the main feature of these tools seems to be rich reporting. Agile Manager automatically offers traceability to requirements and allows you to “dig in, find out the whole trend analysis, what the developers are doing, whether it’s working, whether it’s not working, who’s working hard and who is not”. FleXT also has coverage management – other teams didn’t seem to think that this was a key feature.

Improved collaboration was an interesting topic as well. Agile Manager offers continuous integration and analysis: “While developers are running the code we are telling them whether it is going to work or not. We’re going to put red squiggly lines below the code.” Nimble allows people to “block and tackle any problem”. It also performs continuous testing and has a “soft wall for code review”. FleXT has a virtual story board with virtual sticky notes.

A very surprising outcome of this workshop for me was that most tools seemed to provide a holistic view at a range of projects instead of focusing on managing tests. Many of them were more about story management than tests or defects. Perfect 10, for example, has a product catalogue and, given the priorities, works out automatically which sprint a feature will go into and automatically produces velocity and burndown charts. It has a current sprint view with a quick status overview of the requirements and tests for an on-going iteration. It also has a product catalogue dashboard, with a quick overview of several projects and the ability to drill down to identify problems. Nimble has “cross-project control, resource monitoring, defect tracking, task progress and graphical dashboards” and “shows you exactly where you are and where you want to be”. Agile Manager goes even further to produce a “a cost/benefit analysis model – how you’re moving against your budget” and predicts what the implications might be if someone comes in and changes one of the requirements, integrated with the cost benefit model. “if you put that feature in, that’s great but it’s going to set the project back this by this much time and it’s going to cost this amount of money”.

Coming back from the future into present time, many of these features already exist in a bunch of different tools, but the workshop suggested that we are lacking a universal aggregator for all the data that provides a holistic view at the whole product range. Sonar seems to be the closest thing to an universal integrator I’ve seen so far – it provides heuristics and analytics on code coverage, unit testing and code quality with a plug-in extension model. So we could easily extend it to pull in functional test statistics and defect management metrics as well.

I wouldn’t be surprised if many story management features mentioned exist in one of the popular agile project management tools, but what those tools are probably lacking at the moment is the holistic approach.

Photos: Nathan Bain

At the recent open space session on the future of acceptance testing at Open Volcano 10, we discussed treating test code as second class as one of the biggest problems with agile acceptance testing implementation and one of the biggest challenges for teams in the future. It also appeared on the top 10 reasons why teams fail with acceptance testing at CITCON Europe 09. Long term test maintenance costs also sparked a recent discussion on tools in the blogosphere (see Jim Shore’s original article and responses from George Dinwiddie, Ron Jeffries and my reply and comments on those posts). I now believe that most of these issues come from the same problem.

An acceptance testing framework is a product on its own

Although most of them don’t realise it themselves, the most successful teams I interviewed actually treat their acceptance testing framework as a product in its own right. I’m not talking about the automation tool – though some of the teams have developed their own tools or extended opensource tools significantly – but about the test specifications, the infrastructure around them and the integration layer for automation (fixtures, step definitions, keyword implementations).

Many teams have ended up designing a domain specific language for specifications and tests (implemented using FIT, Cucumber, Robot Framework or even directly in code), structuring their acceptance tests around it to keep them consistent, easy to understand and minimise long term maintenance costs. When retrofitting acceptance testing into already existing products, they often had to make the system and the architecture more testable, which required senior people to plan and implement serious design changes. Implementing agile acceptance testing successfully, and getting the most benefits out of it, requires a significant investment in all of this. It requires structure, design and planning. It requires thinking about how to maximise the return on investment in the framework. It requires on-going cultivation and maintenance.

If we think about tests as by-products of the development process, this investment is very hard to justify because it doesn’t give direct value to any of the project stakeholders. The stakeholders of the framework, however, aren’t the customers or the business users. They are the members of the development teams (and maintenance teams if separate). Many teams I interviewed have at some point ripped out the heart of their system and replaced it, or rewritten the entire system, while keeping their acceptance tests and using them to guide the whole effort. This is where the investment in live documentation really pays off. Such a framework is genuinely a separate product, with a different lifecycle and a different group of stakeholders.

Understanding that the acceptance testing framework is actually a product resolves a ton of things I’ve considered genuine problems in the past, but now look at them as just symptoms of this deeper issue. For example, once you start thinking about the framework as a product, the question of whether to put the tests in a version control system or not goes away immediately. Cleaning up “test code” no longer requires a separate explanation. Working on enhancing the structure or clarity of tests is no longer something to put on the technical debt list – it is part of a backlog for a completely separate product. The flaw in delegating the work on acceptance tests to junior developers and testers suddenly becomes obvious. On the other side of this equation, looking at the testing framework as a separate product also prevents teams from falling into the trap of goldplating the tests at the expense of the primary product.