The problem is that manual and automated checks are affected by a completely different set of constraints. With manual testing, the time spent preparing the context is often a key bottleneck. With automated testing, people spend most time on understanding what is wrong when a test fails.

For example, to prepare for a manual test that checks user account management rules, you might have to log on to an administrative application, create a user, log on as that new user to the client application, and change the password after first use. To avoid doing this several times during the test, manual scripts often reuse the context. So you would create the user once, block that account and verify that the user cannot log on, reset the password to verify that it is re-enabled, then set some user preferences and verify they change the home page correctly. This approach helps a tester run through the script quicker.

With automated testing, time spent on setting up the user is no longer a problem. Automated tests generally go through many more cases than manual tests, and when they run correctly nobody is really looking at them. Once a test fails, someone has to go in and understand what went wrong. If a test is described as a sequence of interdependent steps, it will be very hard to understand what exactly caused the problem, because the context changes throughout the script. The fact that a single script is checking ten different things also makes it more probable that the test will fail because it is affected by lots of different areas of code. In the previous example with user account management, if the password reset function stops working, we won’t be able to set the user preferences correctly. If we had ten different, smaller, focused and independent tests instead of one big script, a bug in the password reset function won’t affect the test results for user preferences. That makes tests more resilient to change and reduces the cost of maintenance. It also helps us pin-point the problems quicker.

Instead of plainly automating manual test scripts, think about what the script is testing and describe that with a group of independent, focused tests. This will significantly reduce the automation overhead and maintenance costs.

The five most important things to think about are:

• It needs to be self explanatory
• It needs to be focused
• It needs to be a specification, not a script
• It needs to be in domain language
• It needs to be about business functionality, not about software design

Here is a really good example:

It has all the elements listed above. Whenever I’ve shown it to people in the workshops, I did not have to use a single word to explain it. The title and the introductory paragraph explain the structure of the test data enough that readers don’t need to work back from the data to understand the rule. But the examples are there to make it actually testable and explain the behaviour in edge cases. It is focused on a very particular rule of free delivery availability, does not explain how the books are purchased but just what the available delivery mechanism is, and does not try to talk about any implementation specifics.

Here is a really bad example (taken from http://fitnesse.org/FitNesse.UserGuide.PayrollTests.AddAndPayTest)

This test has so many bad things in it that it is actually a good example to demonstrate what happens when people don’t take care about writing good tests.

First of all, although it has a title and some text around the tables to seemingly explain what’s going on, the effect of that is marginal. Why is this test “simple”, and what exactly in payroll is it checking? It fails straight away on the self-explanatory litmus-test.

Second, it’s not really clear what this test is checking. We need to work backwards. It seems to verify that the cheques are printed with unique numbers, starting from the next available number that’s configurable in the system. It also seems to validate the data printed on each cheque. And that there is one cheque printed per employee (I’ll come back to this later).

There is a lot of seemingly incidental complexity there – names and addresses aren’t really used anywhere in the test apart from setting up the employees. There are some database IDs there which are completely irrelevant for the business rules, but they are used to match up employees and the paycheck inspector.

Paycheck inspector is obviously an invented thing just for testing. No company is going to have Peter Sellers in a Clouseau outfit inspecting cheques as they go out. If you have enough employees to have to print cheques, you don’t want to inspect them manually. That’s what this test is about, anyway.

There is also a very interesting issue of those blank cells in the assertion part of the test, and the two Paycheck inspector tables which seem unrelated. Blank cells in Fitnesse are used to print test results for debugging and troubleshooting, they don’t check anything. So this is an automated test that a human has to look over — pretty much defeating the purpose of automation. Blank cells are typically a sign of instability in tests (more on this a bit later) and they are often a signal that you’re missing something – either testing in the wrong place or missing a rule that would help make the system process repeatable and testable.

The language is inconsistent, which makes it hard to make a connection between inputs and outputs at first. What is the 1001 value in the table below? The column header tells us that it is a number – well thanks for that, I though it was a sausage. There is a “cheque number” above, but what kind of a cheque number is that? What is the relationship between these two things?

So this test is very very bad.

Presuming that the addresses are there because cheques are printed as part of a statement with an address ready for automated envelope packaging, this test fails to check at least one very important thing (and I’ll come back to this later as well): that the right people got paid the right amounts. If the first person got both cheques, this test would happily pass. If they both got each-others salaries, this test would pass. If a date far in the future was printed on the cheque, our employees might not be able to cash it in but the test would still pass. The reason these cells are blank is because there is another hidden rule there: ordering of cheques coming out. There is nothing specifying that. So a technical workaround for a functional gap is to create a test that gives us loads of false positives.

Is this test checking one thing or many things? Without the context information it’s hard to tell. If the cheque printing system is used for anything else, I’d pull out the fact that cheque numbers are unique and start from a configured number into a separate page. If we only ever print salary cheques, it’s probably part of a single thing (salary cheque printing).

Now, let’s clean it up. Let’s try to work back and drop all the incidental stuff. Let’s use a nice descriptive title, such as “Payroll Cheque Printing”. And let’s add a paragraph that explains the structure of the test.

A cheque has the payee name, amount, payment date. A cheque does not have a name and a salary. If the cheque is printed on a statement, it also has an address that will be used for automatic envelope packaging. A combination of a name and address should be enough for us to match the employee with his cheque — we don’t really need the database IDs. We can make the system more testable by agreeing on an ordering rule, whatever it is. For example, alphabetic by name.

Let’s also pull the context to the start of the test. Our context is the payroll date and the next available cheque number, along with employee salary data. Let’s make it explicit what the number is for, so that the readers don’t have to figure this out for themselves in the future. We can also make this block stick out visually to show that it is about the context.

The action that gets kicked off doesn’t necessarily need to be listed in the test. A payroll run can be executed implicitly by the table that checks payroll results. This is an example of focusing on what’s being tested instead of how it is being checked. There is no need to have a separate step that says “Next we pay them”.

Let’s also rename that paycheck inspector to something that makes more sense. Because we want whoever automates this to ensure that we check for all cheques printed, let’s put that in the header. Otherwise, someone might use subset matching and the system might print every cheque twice and we won’t notice.

And here is the cleaned version.

Much easier to understand, without all that incidental stuff. And now comes the punchline. When we look at the test like this, without database IDs, without all the unnecessary clutter, we can have a nice shot at answering the question “Are we missing something?”. What are the edge cases that might break this? We don’t really need to ensure validity of employee data, hopefully that’s done in another part of the system. But is there any kind of valid employee data that would be an edge case for this test – can we play with the numbers to make this illogical in some way?

An obvious answer to that is – what happens if an employee has a salary of 0? Do we still print the cheque? The rule as we described it says “One cheque per employee” – so any employees that have been fired years ago and no longer receive salaries would still get cheques printed, with zeroes on them. We could then have a discussion with the business on making this rule stronger and ensuring that cheques don’t go out when they don’t need to do.

FitNesse gets a lot of bad reputation because of this kind of broken tests. Concordion was built as an answer. Some people at my recent workshop on this example pointed out that Given-When-Then structure of Cucumber would be better at preventing some of these problems. I don’t think so. It’s not about the tools – people can do this kind of bad test design with any tool, and likewise they could do nice and clean tests with FitNesse. I guess the fact that the basic examples coming with FitNesse are so bad doesn’t help, but the problem is not in the tool. It’s about the process and effort put into making the tests easy to understand. The funny thing is it doesn’t take a lot more effort to make the test nice and clean, but it brings a lot more value that way.

If you found this article helpful, you’ll love my three day workshop on effective specification by example and agile acceptance testing

Online articles and resources

• US Army experiment
• B2 bomber crash report
• F-16 design project
• Tests and Requirements, Requirements and Tests: A Möbius Strip
• Describe-demonstrate-develop
• Rick Mugridge: Doubling the value of automated tests from Google Tech Talks in London in 2006.
• Concordion
• FitNesse
• Agile Testing UK
• Agile acceptance testing info portal

Books

• Bridging the Communication Gap: Specification by Example and Agile Acceptance Testing, my book (also see the PDF edition)
• Sources of Power: How People Make Decisions by Gary Klein
• Implementing Lean Software Development: From Concept to Cash (Addison-Wesley Signature) by Mary and Tom Poppendieck
• Exploring Requirements: Quality Before Design by Gerald Weinberg and Donald C. Gause

Events

• Agile Specifications, Testing and BDD exchange, November 27th, London
• Intro to Specification by Example and Agile Acceptance Testing with Fitnesse , November 30th, London
• Writing software that matters, December 4th, London

• How to communicate effectively with project stakeholders
• Value of specification by example
• How to write good scenarios
• How to spot problems with scenarios and examples and how to fix them
• How to integrate BDD and specification by example into your development process and organisation
• Focusing on writing software that really matters
• Managing code after it is released, using examples and scenarios as live documentation

This will be a unique opportunity to learn BDD and specification by example straight from the horse’s mouth. We’re ironing out the final details and I’ll publish them early next week. For now, I can only say that the workshop will take place in central London mid December this year and that the number of places will be very limited (definitely not more than 30), so if this sounds interesting send me an e-mail or ping me on twitter and I’ll keep you posted.

This was the first time I did something similar at a public event. The workshop is one of the central parts of my one-day agile acceptance testing course but I always presented that on-site in companies, where we work on the domain of that particular company and all the participants know quite a bit about it. With a public event, that is a challenge as participants work for different companies and on different domains. I chose the Blackjack game for the domain as it is one of the most popular casino games, but not a lot of people go to casinos, so I supposed that there will be 5 or 6 people in the room that know how to play and they will be able to act as domain experts or customer representatives. Blackjack rules are fairly simple so one hour of demonstrating with playing cards will be enough to introduce the rules to people who had no previous exposure to the game. It was a bit of a challenge to buy ten packs of playing cards on a Sunday evening where I live, but I think that it was worth it.

The experiment

To simulate a situation where a customer points the development team to a competitor site and asks them to copy some functionality, I went to a popular blackjack online games site and copied the game rules. The interesting thing about these rules is that, although they fit on a single A4 paper, there were quite a few inconsistencies and functional gaps there. For example, one rule stated that the house always wins if the dealer has a Blackjack and another rule stated that the player gets his money back if both the player and the dealer have the same cards, so a case where both the player and the dealer have a Blackjack was ambiguous. There were some edge cases that were not properly explained which left a lot of space for ambiguity and misunderstanding. One particular rule dealing with the value of the Ace card was very uncommon, as if it was intended to give more advantage to house over players. I’m pretty sure that it was not implemented like that even on the original site, but I was interested in finding out whether the workshop participants will complain about it.

The workshop

There were seven people in the room who were blackjack players, so I asked them to be domain experts and other people to form teams around them. Teams were then given the task to demonstrate chosen blackjack rules for financial payout using a pack of playing cards and realistic examples. During the first hour I mostly let the people in the teams explain the rules to each-other. During the second hour I asked them to focus more on specifying financial outcome rules with realistic examples and demonstrating individual rules from the requirements document. If they found any inconsistencies or missing functionality, the domain expert on the team was supposed to resolve the issue and in particular write that down as an example. On the end, one representative from each team – not the domain expert – presented the examples and answered the following three questions:

• was there any functionality missing from the requirements that you discovered?
• were there any conflicting rules that you discovered?
• was there something that you scrapped and decided not to do?

The results

Although none of these people had any previous experience with example-driven specification workshops, the examples that all teams produced were pretty good which shows that it is fairly easy to get started with specification workshops. Most of the examples ended up being in the form of “with this hand of the player and this hand of the dealer, the outcome is this”. One team used examples to define domain terms such as “bust” and “blackjack” and then used them in the examples demonstrating game outcome. Other teams only used examples of game outcome and defined the domain terms implicitly using that. With specification workshops on real projects, we often end up cleaning up examples to produce the specification (I call this distilling the specification in my book Bridging the communication gap), so this would give people a chance to refine examples and focus them.

One of the teams used the rules on the paper just as a guideline, they worked on Blackjack rules that the domain expert demonstrated during the workshop and ignored the details on the paper, which is an interesting example of the team ignoring written requirements and going with their understanding of the matter. A domain expert in another team pulled out a card with Blackjack rules and card counting strategies from somewhere and they used that to clear up any questions. Two teams decided to ignore the offending rule about Ace values and just specify the Ace value as it was in a normal Blackjack game, which is an example of a domain expert overruling a requirement that makes no sense. One team had not reached that rule during the workshop at all so they haven’t noticed it. All the teams discovered almost all the inconsistencies and functional gaps, and one team even noticed something that I missed while preparing the workshop. It turned out that one of the designated domain experts wasn’t really a Blackjack player so they spent lots of time discussing cases which are irrelevant for the game, such as the dealer splitting cards. That might seem logical from a perspective of someone who values symmetry (and developers typically do) so this demonstrated what can happen when developers try to implement acceptance testing on their own without customers or domain experts.

Measuring the effectiveness

I asked for a quick show of hands to count who considered that discussing realistic examples helped flush out inconsistencies and functional gaps. All the workshop participants except one raised their hands, and the one that did not raise his hand said that he had not heard the question, then said that he agrees with that as well.

I also ran a quick feedback exercise. Feedback exercises are a very effective way of measuring shared understanding of the domain, working on the same principle as Planning poker. Team members write down answers to a question that requires understanding of the domain, typically a difficult edge case, and then compare the results. Differences in answers point to differences in understanding. Before the workshop, I selected three particularly difficult edge cases that were not explained precisely on the requirements sheet. After the workshop, I asked all the participants to write down the outcome in those three cases and then compare it with other members of their team. Different teams will probably have different answers because their domain experts made different decisions, but people in the same team should have the same answers to these questions if they share the understanding of the domain. After they compared the results, I asked people to raise their hands if they had different answers within the team – only one team raised their hands. The explanation was that they still have not had time to discuss a particular rule that was demonstrated by the edge case.

As six out of seven teams had a shared understanding good enough to come up with the same answers to really difficult edge cases after a quick workshop, even though most people on the team had no previous exposure to the target domain, for me the experiment was a very effective demonstration of how specification workshops and specification by example help improve clarity in software projects and give us a much better foundation for implementation and testing than traditional requirements.

Feedback

If you participated in this workshop – I’d really love to hear your feedback. Please post a comment below or contact me. To find out more about specification workshops and agile acceptance testing, have a look at the agile acceptance testing portal and the resources page on my site.