Online articles and resources

• US Army experiment
• B2 bomber crash report
• F-16 design project
• Tests and Requirements, Requirements and Tests: A Möbius Strip
• Describe-demonstrate-develop
• Rick Mugridge: Doubling the value of automated tests from Google Tech Talks in London in 2006.
• Concordion
• FitNesse
• Agile Testing UK
• Agile acceptance testing info portal

Books

• Bridging the Communication Gap: Specification by Example and Agile Acceptance Testing, my book (also see the PDF edition)
• Sources of Power: How People Make Decisions by Gary Klein
• Implementing Lean Software Development: From Concept to Cash (Addison-Wesley Signature) by Mary and Tom Poppendieck
• Exploring Requirements: Quality Before Design by Gerald Weinberg and Donald C. Gause

Events

• Agile Specifications, Testing and BDD exchange, November 27th, London
• Intro to Specification by Example and Agile Acceptance Testing with Fitnesse , November 30th, London
• Writing software that matters, December 4th, London

• How to communicate effectively with project stakeholders
• Value of specification by example
• How to write good scenarios
• How to spot problems with scenarios and examples and how to fix them
• How to integrate BDD and specification by example into your development process and organisation
• Focusing on writing software that really matters
• Managing code after it is released, using examples and scenarios as live documentation

This will be a unique opportunity to learn BDD and specification by example straight from the horse’s mouth. We’re ironing out the final details and I’ll publish them early next week. For now, I can only say that the workshop will take place in central London mid December this year and that the number of places will be very limited (definitely not more than 30), so if this sounds interesting send me an e-mail or ping me on twitter and I’ll keep you posted.

This was the first time I did something similar at a public event. The workshop is one of the central parts of my one-day agile acceptance testing course but I always presented that on-site in companies, where we work on the domain of that particular company and all the participants know quite a bit about it. With a public event, that is a challenge as participants work for different companies and on different domains. I chose the Blackjack game for the domain as it is one of the most popular casino games, but not a lot of people go to casinos, so I supposed that there will be 5 or 6 people in the room that know how to play and they will be able to act as domain experts or customer representatives. Blackjack rules are fairly simple so one hour of demonstrating with playing cards will be enough to introduce the rules to people who had no previous exposure to the game. It was a bit of a challenge to buy ten packs of playing cards on a Sunday evening where I live, but I think that it was worth it.

The experiment

To simulate a situation where a customer points the development team to a competitor site and asks them to copy some functionality, I went to a popular blackjack online games site and copied the game rules. The interesting thing about these rules is that, although they fit on a single A4 paper, there were quite a few inconsistencies and functional gaps there. For example, one rule stated that the house always wins if the dealer has a Blackjack and another rule stated that the player gets his money back if both the player and the dealer have the same cards, so a case where both the player and the dealer have a Blackjack was ambiguous. There were some edge cases that were not properly explained which left a lot of space for ambiguity and misunderstanding. One particular rule dealing with the value of the Ace card was very uncommon, as if it was intended to give more advantage to house over players. I’m pretty sure that it was not implemented like that even on the original site, but I was interested in finding out whether the workshop participants will complain about it.

The workshop

There were seven people in the room who were blackjack players, so I asked them to be domain experts and other people to form teams around them. Teams were then given the task to demonstrate chosen blackjack rules for financial payout using a pack of playing cards and realistic examples. During the first hour I mostly let the people in the teams explain the rules to each-other. During the second hour I asked them to focus more on specifying financial outcome rules with realistic examples and demonstrating individual rules from the requirements document. If they found any inconsistencies or missing functionality, the domain expert on the team was supposed to resolve the issue and in particular write that down as an example. On the end, one representative from each team – not the domain expert – presented the examples and answered the following three questions:

• was there any functionality missing from the requirements that you discovered?
• were there any conflicting rules that you discovered?
• was there something that you scrapped and decided not to do?

The results

Although none of these people had any previous experience with example-driven specification workshops, the examples that all teams produced were pretty good which shows that it is fairly easy to get started with specification workshops. Most of the examples ended up being in the form of “with this hand of the player and this hand of the dealer, the outcome is this”. One team used examples to define domain terms such as “bust” and “blackjack” and then used them in the examples demonstrating game outcome. Other teams only used examples of game outcome and defined the domain terms implicitly using that. With specification workshops on real projects, we often end up cleaning up examples to produce the specification (I call this distilling the specification in my book Bridging the communication gap), so this would give people a chance to refine examples and focus them.

One of the teams used the rules on the paper just as a guideline, they worked on Blackjack rules that the domain expert demonstrated during the workshop and ignored the details on the paper, which is an interesting example of the team ignoring written requirements and going with their understanding of the matter. A domain expert in another team pulled out a card with Blackjack rules and card counting strategies from somewhere and they used that to clear up any questions. Two teams decided to ignore the offending rule about Ace values and just specify the Ace value as it was in a normal Blackjack game, which is an example of a domain expert overruling a requirement that makes no sense. One team had not reached that rule during the workshop at all so they haven’t noticed it. All the teams discovered almost all the inconsistencies and functional gaps, and one team even noticed something that I missed while preparing the workshop. It turned out that one of the designated domain experts wasn’t really a Blackjack player so they spent lots of time discussing cases which are irrelevant for the game, such as the dealer splitting cards. That might seem logical from a perspective of someone who values symmetry (and developers typically do) so this demonstrated what can happen when developers try to implement acceptance testing on their own without customers or domain experts.

Measuring the effectiveness

I asked for a quick show of hands to count who considered that discussing realistic examples helped flush out inconsistencies and functional gaps. All the workshop participants except one raised their hands, and the one that did not raise his hand said that he had not heard the question, then said that he agrees with that as well.

I also ran a quick feedback exercise. Feedback exercises are a very effective way of measuring shared understanding of the domain, working on the same principle as Planning poker. Team members write down answers to a question that requires understanding of the domain, typically a difficult edge case, and then compare the results. Differences in answers point to differences in understanding. Before the workshop, I selected three particularly difficult edge cases that were not explained precisely on the requirements sheet. After the workshop, I asked all the participants to write down the outcome in those three cases and then compare it with other members of their team. Different teams will probably have different answers because their domain experts made different decisions, but people in the same team should have the same answers to these questions if they share the understanding of the domain. After they compared the results, I asked people to raise their hands if they had different answers within the team – only one team raised their hands. The explanation was that they still have not had time to discuss a particular rule that was demonstrated by the edge case.

As six out of seven teams had a shared understanding good enough to come up with the same answers to really difficult edge cases after a quick workshop, even though most people on the team had no previous exposure to the target domain, for me the experiment was a very effective demonstration of how specification workshops and specification by example help improve clarity in software projects and give us a much better foundation for implementation and testing than traditional requirements.

Feedback

If you participated in this workshop – I’d really love to hear your feedback. Please post a comment below or contact me. To find out more about specification workshops and agile acceptance testing, have a look at the agile acceptance testing portal and the resources page on my site.

The book is about improving communication between customers, business analysts, developers and testers on software projects, especially by using specification by example and agile acceptance testing. It is primarily intended for product owners, business analysts, software developers and testers who want to learn about agile acceptance testing and implement it. It should also prove to be interesting to project managers working on software projects, both within the implementation team and on the customer side. It is intended both for people already working with agile processes and for people who wish to migrate to them.

I’m really enthusiastic about this book because I think that it fills an important void in the market today, helping different roles look at specification by example and agile acceptance testing from their perspective and figure out how to cooperate better as a team. The book is intentionally not very technical so that business people can read and understand it. It got some really nice reviews, so I encourage you to have a look at it by downloading the sample chapters and checking out the table of contents.

The book is (again) published by my company using a print-on-demand system. It should soon appear on all major online bookshops and will hopefully be picked up by traditional bookstores as well. It is immediately available online from my new acceptance testing info portal. You can order the paperback (they will start shipping on Jan 5th) or download the PDF immediately.

Collaborative requirements and specifications

Specifications written by a single person do not harness the knowledge and experience of all people on the team, in particular technical people are often left out of the loop. Collaborative specifications are often better in the sense that they are more complete, easier to implement and more consistent with the technical infrastructure.

The best way to get collaborative specifications is to organise a workshop at the start of each iteration. The goal of the workshop is to discuss examples of stories that are planned for the iteration. The discussion leads to investigating edge cases, identifying functional gaps and inconsistencies before the development starts. A tangible output of the workshop is a set of realistic examples demonstrating important cases for the user stories planned for the iteration, which can be easily converted to acceptance tests.

An even more important output of the workshop is coordinating the thoughts and the understanding of the domain in the heads of business people, developers and testers. The workshop also facilitates a transfer of domain knowledge to developers and testers, allowing them to learn about the domain first hand and avoid the effects of the telephone game. This shared understanding is, for me, even more important than the physical acceptance tests. Unfortunately, this output is not tangible, which is why the workshop often gets skipped and replaced by just a single person writing tests.

A new name

Regular readers of my blog or people who have seen my acceptance testing presentations will probably recognise these ideas under several names, because until today I did not really have a good name for the concept.

I initially called this “the acceptance testing threesome”, suggesting that business people, developers and testers need to work together. This was labeled as offensive by some people so I changed it to acceptance testing meeting, but having the word “testing” in the name caused a lot of confusion as testers thought that they should take charge and business people often did not understand why they need to participate. I later changed this to “example-writing workshop” to emphasise that the workshop should be focused on writing and discussing examples. This was better in the sense that “testing” was no longer an issue.

Participation of the business, both in terms of domain experts and stakeholders, is crucial to get these workshops right. Domain experts will know how the system should work, but they rarely control the scope. Someone has to be able to say yes or no to edge cases, ideas and identified functional gaps. Workshops can also provide quite a good project progress visibility to such stakeholders, but “example-writing workshop” is not something that they feel obliged to participate in.

After one persuasion session with a project stakeholder this week, I think that I finally have a good name that is self-explanatory enough to ensure that stakeholders want to be there. So I propose yet another new name for the concept. Specification workshop makes it was clear that the business needs to be present there, both in terms of defining the contents of the specification and controlling what goes in or out.

As I’m now in the final stages of preparing my new book for printing, this change came in just in time for one last refactoring of the book. The example-writing workshop was one of the key concepts in the book, so this is a major refactoring. I’m now really happy that I used docbook again so that this change will not hurt at all.

Image credits: erdogan ergun