My experience with Fit and other agile acceptance testing tools is that they cost more than they’re worth. There’s a lot of value in getting concrete examples from real customers and business experts; not so much value in using “natural language” tools like Fit and similar.

The two failure patterns that Shore describes in his post are falling back on testers to write everything and merging acceptance and integration tests. I’ve experienced both of these myself, and it seems that they are common in general. We discussed both during the top 10 ways to fail with acceptance testing openspace session at CITCON Europe last year. However, there are good ways to solve both problems.

I never really expected customers to write anything themselves, but I was relatively successful in persuading them to participate in specification workshops that led to examples which were then converted to acceptance tests later. Another idea I discovered while doing the research for my new book is discussing the key examples with customers and then going off to write detailed test pages, which then get verified by the customers. The third good idea is doing ad-hoc specification writing sessions when a developer needs information, by involving a tester and a business analyst. This is a lot less formal than a specification workshop and gives you similar benefits if you have all the knowledge in the room (or readily available) most of the time.

Not preserving acceptance tests as a separate group and mixing quick and slow tests is something that most people, at least according to my ongoing research, get burned with at some point but again teams learn from that and improve.

One of the biggest benefits from acceptance testing for me was that the teams finally get a source of information on what goes on in the system as reliable as the code itself. Without acceptance tests, code is the only thing you can really trust and any other documentation gets outdated very quickly. And such tests are much easier to read and understand than the code because they are on a higher level and in a natural language. Having a live specification helps me quite a lot when change requests come in later. It also helps with handing over and taking over code. Acceptance tests stay relevant throughout the project because they are automated, and automated tests are kept up to date in order for them to pass. Automation, and consequently a tool, are necessary to get this benefit. With informal agreements and on-site reviews that Jim Shore describes, I guess something else needs to be in place to facilitate this.

I agree with Shore that it takes a while for the problems with tools such as FIT to surface, but I’m not sure whether that is tool related or not. Most people I spoke to so far said that it took them between six months and a year to discover that acceptance testing isn’t about the tools but about communication, and that the biggest benefit is in the examples as Shore wrote. A notable exception to six months to a year rule was Lisa Crispin‘s team who generally started out knowing what they need (but that’s because she had done it before). Clear examples and improved communication are the biggest benefits of the process, but using a tool brings some additional nice benefits as well. A tool gives us an impartial measure of progress. Ian Cooper said during the interview for my new book that “the tool keeps developers honest”, and I can certainly relate to that. With tests that are evaluated by an impartial tool, “done” is really “what everyone agreed on”, not “almost done with just a few things to fill in tomorrow”. I’m not sure whether an on-site review is enough to guard against this completely.

update (2009-01-05): a more polished version of this article is published in my new book, Bridging the Communication Gap as part of the “Implementing Agile Acceptance Testing” chapter!

update (2008-09-18): the implementation and polishing phases explained better, link to Lisa Crispin’s process description added

Agile acceptance testing, at least in the form in which I use it, does not say anything about how to develop software or organise iterations. There is nothing stopping us from fitting it into a two-week iteration. In fact, it will work best with short iterations. Short iterations enforce us to keep the stories
which should be discussed in the workshop short, allowing us to keep the workshop short and focused. This is a simplified version of the iteration flow that Andy Hatoum and I came up with recently. Take this just as a guideline, not as something that has to be implemented exclusively that way. Hopefully it will make it easier to understand how to apply these ideas in different environments.

The brackets point out to the iteration that a step relates to. N is the current one, N-1 is the previous one and N+1 is the next iteration.

1. Potential Release (N-1): We can release the result of the previous iteration on Monday (marked N-1). Not every iteration has to be shipped to customers, but we like to have the code as a potentially shippable software package at any point. We don’t release on Fridays to reduce the risk of causing production problems over weekends, while people are not readily available to fix issues.
2. Implementation (N): On Monday, the implementation of the current iteration (N) starts. This includes all normal agile development practices such as unit testing and continuous integration. An important change for teams already practising agile development is that the development is focused on implementing acceptance tests specified for the iteration. Implementation includes running and cleaning up acceptance tests, identifying and discussing missing cases, writing more tests for those cases or modifying existing tests. It also includes releasing to the QA environment often, so that on-site customer and testers can play with the software and provide quick feedback. Implementation continues until Wednesday next week.
3. Pre-planning (N+1): The clients, the business analyst and the project manager and the development team leaders work out roughly what will go into the next iteration (not the current one, but the one after). This is intentionally a rough plan, not carved in stone, but good enough to give the business analyst and the customers a starting point for discussion and allow them to stay one step ahead. This happens on Monday as well.
4. Acceptance test clean-up and review (N): Not all acceptance tests will be perfect straight away, so a business analyst chases open questions, gets remote domain specialists to review the tests and the PM can get any sort of sign-off required for tests. Acceptance tests can be simplified, cleaned up and organised better by testers, developers or business analysts. We expect this in practice to span the first few days of an iteration. Changes might be introduced into the acceptance tests later, but we expect the bulk of it to be done in the first few days and for the tests to stabilise after that. The diagram shows that this should end by Thursday, but that is not a specifically important deadline, it is more an estimate. Clean-up ends when it ends, sometimes it may not be needed at all, sometimes it will end sooner, sometimes it will spill over to the next week.
5. Preparing examples (N+1): Once the bulk of acceptance test clean-up it is done, the business analyst can start working with the clients and the QA on the examples for the next iteration, preparing a good starting point for the workshop. The time-lines for this are again not enforced, so the diagram is a rough estimate. In general, after the work on the current iteration is done, the business analyst should focus on the next one and keep doing that until the polishing starts. The important thing is that these examples don’t need to be absolutely complete at this point. The goal is to get started and prepare for the workshop, not to iron out all the acceptance tests.
6. Integration testing, Exploratory testing, Polishing and Demo (N): The development should close by Tuesday in the second week of the iteration. At that point, we start polishing the result. We expect developers to run packaging and release to the internal QA environment as soon as there is something for others to see. That allows the on-site client and QA to check what developers are doing during the development and provide feedback, but those intermediate releases do not necessarily need to be production quality. At this point we start really focusing on producing a clean, polished package. This involves cleaning up database scripts, packages, configuration, testing the installation and upgrades from the previous version. Everyone switches from adding new features into exploratory testing and cleaning up the results. Issues, UI bugs and ideas for small improvements get resolved during during Wednesday and Thursday. By Thursday afternoon, we should have a polished version that we can demonstrate to off-site customers remotely. Polishing may spill over on to Friday, but that should be an exception rather than a rule.
7. Retrospective (N): A normal retrospective, looking at the last two weeks of development and possibly a wider scope. Everyone is involved.
8. Planning (N+1): Planning poker, estimating stories, selecting the scope for the next iteration. This becomes the official plan, and the results from the pre-planning session are used to start of the discussion. Everyone is involved.
9. Example-writing workshop (N+1): After the scope has been selected, the workshop is organised to nail down the specifications and facilitate the transfer of knowledge. The aim of the workshop is to build a shared understanding among developers, business people and testers about the aims for the next two weeks of work. A more tangible result of the workshop are realistic examples which can be converted to acceptance tests.

   Selected stories are introduced by the business analyst, then we all go through the examples that the business analysts have prepared with the customers. The other workshop participants then ask questions and suggest edge cases for discussion. Developers need to think about how they will implement the stories and identify functional gaps and inconsistencies. Testers need to think about breaking the system and suggest important testing cases for discussion. We can organise feedback exercises during the workshop to check whether we all understand the same thing. Because realistic examples are discussed and written down, inconsistencies and gaps should be easy to identify at that stage and we will get a solid foundation for the development. The workshop ends when everyone involved agrees that there are enough examples and that everything is clear enough to start the work. Some questions may remain open, because they require further approval of a more senior stakeholder or an opinion of a particular domain expert, but in general the workshop should define the precise scope and ensure that we all agree on the same things. A representative set of examples is chosen to be an acceptance criteria at the end of the workshop, and formalised into acceptance tests.

After this, we start all over with the next iteration.

For an alternative view, see Lisa Crispin’s process description.
Image credits: Arte Ram and Keith Syvinski

They conducted the experiment by showing a seven-point star picture before one of their seminars, then after a lecture and a coffee break asked the attendees to tell them how many points the picture had. They got quite a wide spread of results. According to Gause and Weinberg, everyone knows what a five-point star looks like but seven-point star was not that usual, so people remembered it differently and recalled different images, especially after the coffee break. That accounted for the spread. People also understood the task differently, which explained clusters in the answers.

I am not so interested in people not remembering things off the top of their head, but I wanted to check the problem of differences in understanding. After my talk on Selenium yesterday, I handed out index cards and put a picture of a standard five-point star on the projector, asking people to write down how many points they see in the picture there and then. I used an image of a familiar figure (the one on the right) and it was on display while people wrote down their answers. So any differences in answers could only be caused by people understanding the task differently.

After initial reluctance and my explaining that the question might sound stupid and obvious, but I’d really like people to answer it anyway, I got about 50 cards back. And the results were simply amazing. I expected that most people would write 10 as the number, and that a few would possibly write “infinite” referring to the mathematical axioms that any line contains an infinite number of points. But the answers were much more surprising.

Most people (25) voted for 10 points in the star, counting inner and outer points. The second most popular answer (7) was five points, where people counted just the outer points. Number 14 got a lot of votes as well — one of the people at the pub after the talk explained to me that this is probably the ten points on the star and four corners of the picture. There was a single vote for number 9, which can theoretically be explained by five outer points and four corners of the picture, but there were also four cards which I can not explain at all: numbers 11 and 15 got two votes each.

If anything, this experiment has confirmed that even a simple thing as a familiar image and a straight-forward question can cause a lot of misunderstanding, or rather differences in understanding. Some people thought that the edges of the image counted as part of the image, some did not consider them. Some people just counted outer points, some counter inner as well. This is essentially why giving people screenshots or wireframes with simple descriptions does not work as an effective technique to pass knowledge. This is a very effective demonstration why we need workshops that stimulate discussion to define requirements and specifications, ideally using realistic examples which could then be converted to acceptance tests.

Surowiecki started with a story about Francis Galton, a prominent English Victorian-age statistician infamous as the founder of eugenics, who organised a contest at a country fair in west England. There was an ox on display and people guessing the weight of the ox. Galton took the average of all guesses and expected a deeply flawed estimate, but the average was in fact spot on. Surowiecki also mentioned jelly bean counting experiments that are apparently a modern example of the same idea. The researcher holds a jar filled with jelly beans and asks the audience to estimate the number of beans in the jar. According to Surowiecki, the average of all answers is always very close to the true number of beans, often closer than any individual guess. The organisers of Agile 2008 conducted a similar experiment, asking conference attendees to estimate the number of lines of code in Visual Studio. The average value of all guesses was 47 million and the actual number is 43.3 million of code. The interesting thing as well is that only two people guessed better than the group average.

According to Surowiecki, Microsoft now uses a similar technique to estimate how long projects are going to take. He talked about an early experiment with implementation team members effectively betting on when the project is going to complete. Although the project was scheduled to end in November, most people were “buying shares” for February, sending a strong signal that the project is not on track. Bestbuy ran an experiment with polling 100 people from the company how much Christmas card sales are going to jump and that estimate was more precise than the estimate of their expert team. Experiments like the ones in Microsoft and Bestbuy also show that tapping into group intelligence allows a more efficient flow of information in organisations. People closer to the problem often have a better sense of what is going on than those higher in the chain, but hierarchical organisations often impede an efficient flow of information. Surowiecki quoted Lewis Platt, a former CEO of HP, “If HP knew what HP knows, we would be three times more profitable”.

Surowiecki’s comment about those experiments was that “under the right conditions, groups of people can be remarkably intelligent, can be smarter than even the most intelligent person in the group”. But that leaves us with the question of discovering the criteria you need to make your group a wise one. The answer to that question, Surowiecki argued, can be broken down into three categories.

1. aggregation: there has to be an efficient way to aggregate lots of different judgments into a single collective judgment. The final product has to represent the conclusion of the group.
2. diversity: “the more diverse the team is, the more likely it is going to be smart and produce good work”. In this example, the focus is on cognitive diversity, getting input from people who are thinking about the problem from different perspectives and use different approaches and heuristics.
3. Independence: people have to offer their own judgment and knowledge rather than spend time looking at what other people want. Surowiecki said that “groups are smartest when people in them work as individuals”.

Diversity on the team

Although it works best for larger groups, from 100 people onwards, Surowiecki said that there is a lot of research that proves great effects even with groups of 6 or 7 people. The challenge with smaller groups is to work harder to foster diversity and individuality. Small groups also often have a problem with flushing out the knowledge that is not shared, as the discussion tends to focus on topics that everyone knows. So one of the key practices to create a smart small group is to focus on putting individual insights and knowledge into the open as well. During the Storytelling Skills for Agile Teams workshop in the evening, Rachel Davies and Rebecca Wirfs-Brock suggested writing designer stories that reflect concerns and ideas about the project from individual viewpoints and then making people read them out to the group, which might be an interesting solution to the problem of voicing individual knowledge.

Diversity is especially valuable in small groups because it expands the range of information. Bring people together who are not going to look at the problem from the same angle, and they will not make the same mistakes. You want people that make mistakes at worst not correlated, at best negatively correlated so that they cancel each other out. Without diversity, groupthink becomes a problem. Surowiecki said that “the more homogeneous groups talk to each other, the dumber they become”. Because of the same opinions, there is sort of an echo effect, and people become convinced that their ideas are correct. At that point, it becomes really hard to see blind spots. A possible solution for this is to appoint the Devil’s advocate, someone who should intentionally try to find flaws in common opinions. Surowiecki said that a single different opinion can often make a big difference but that a pitfall with this approach is that, if the same person is always the Devil’s advocate, other team members will start discarding that opinion. Having some less experienced people on the team can also help because they will ask questions that experts might overlook.

Group diversity is also important to eliminate the effects of peer pressure. Surowiecki quoted an experiment where eight people were asked to identify which of the three lines shown to them are of the same length, with seven people actually being told to intentionally give a wrong answer. The eight person (the one that was actually the subject of the experiment) often selected the same wrong answer to get along with the group. When the same experiment was repeated with six people giving the wrong answer and the seventh giving the right one, then the subject chose the right answer as well.

Surowiecki argued that the best group decisions do not emerge out of immediate consensus, but out of conflict, and that companies should promote a culture that encourages a “good fight”. Diversity is very good for that, but it can also pose a challenge, as it takes some work to make a diverse group function well.

Group intelligence is not a war on expertise — you need people who have a good idea of the problem in the first place. Polling random people on the street about the number of lines of code in visual studio would not produce a high-quality estimate. The underlying idea is how to expand the power of experts and get something better than would be produced by any particular person in the group.

How can we apply this to software projects?

An immediate thing that comes to mind is to build teams out of people with different opinions. Have people on the team that think differently, use different tools and approach the problems from a different angle. This will help the team spot blind spots easier and avoid the echo effect, increasing the team collective intelligence.

Getting a diverse group of people involved with flushing out specifications is one of the key practices I advocate to effectively implement agile acceptance testing. Business domain experts, developers and testers all have to be involved in discussing examples because they will all come with different concerns and ideas. By discussing all those ideas upfront we get a more detailed picture of the system and identify potential pitfalls much earlier. When the specifications are written only by a single person or a homogeneous group, lots of things pass unnoticed until much later in development.

The best way to quickly see lots of landmarks in London is to take the boat ride from Westminster to Greenwich. The boat operators do not hire professional tourist guides, but someone from the staff typically introduces important buildings with a mix of history, popular culture and humour. Not too far from the place where suspected pirates were chained to the river bead and left to the tide to decide whether they were guilty or not, on most of those boats you will hear the story how the London Bridge got shipped to United States. The original London Bridge was bought by some folks from US as a marketing stunt in 1968 for a few million dollars. It was taken down, every stone was numbered, packaged and transported to Arizona, then re-assembled there and it now stands on top of Lake Havasu. A new bridge, also called the London Bridge, was put in its place over Thames river, where this story typically gets told. Although this seems like a huge undertaking, it would not be important for this story if it weren’t for the wrong assumptions.

If the story is to be believed, the buyers assumed that they were buying the “London bridge” — you know, the one from all the postcards with two big medieval towers and moving platforms. In fact, they got the London Bridge. Although famous around the world because of the nursery rhyme, the original London Bridge is a very dull and ordinary construction. It does not stand out at all from any of the many anonymous bridges you cross every year without giving them a second thought. When you think about it, the nursery rhyme is actually about the bridge falling down, not about it being great or historically significant. No surprise the Brits were more than happy to sell it.

The new London bridge is equally uninspiring. In fact, most people are quite disappointed when they see the (new) bridge for the first time — somehow its fame leads us to believe that it is in some way special.

The landmark bridge of London, which the tourists flock to, is just half a mile down the river from the London Bridge, and it is called the Tower Bridge. But even at first sight, they are very, very different.

Somehow this story is a bit comforting for programmers, because it suggests that the software industry is not the only one plagued by the terrible effects of wrong assumptions. Although, I must admit, even if this story is true to the last word, problems caused by wrong or misunderstood assumptions are much more commonplace in software then in the world of second-hand bridge trade.

Ask “why”, not “what”

I had a unique privilege of chatting to Eric Evans last week about the danger of (wrong) assumptions. He suggested asking “why” instead of “what” to flush out the assumptions and minimise the dangers of misunderstood requirements in software projects. This reminded me of the Commanders Intent story, where the conclusion was to focus on “why” and “watch out for” much more than on “what” and “how”.

Understanding why something is required, instead of just blindly following the tasks, is definitely one of the key practices to improve the chances of success in a software project. In my article The magic of goals, I suggested that whenever a client comes with a list of tasks, we should instead ask them for the problems that they are trying to solve and their business goals. In the same way that understanding underlying business goals helps us improve the whole project, understanding the business reasons behind a task can help us make sure that we are on the same page. It may not give us a framework to make business decisions, but it will give us a foundation to spot when something is fishy. If the person negotiating the purchase of the London Bridge understood that the goal was to get something magnificent to attract the tourists, he or she would have spotted the flaw in the plan straight away after seeing the real bridge. I’m sure that the old London Bridge, dull as it is, attracts tourists to Lake Havasu in Arisona, but I am equally sure that the Tower Bridge would have done a much better job. It is unlikely that the Tower Bridge was for sale as well, but for the money that these people invested in the whole project, they could have probably built something much much more interesting.

Likewise, a few misunderstood requirements and wrong assumptions are not going to doom a software project, but spending time to understand why something is needed and how that fits into the big picture will make the difference between a mediocre delivery and great results.

Image credits: wallyg, John McCall, Lim CK.