My experience with Fit and other agile acceptance testing tools is that they cost more than they’re worth. There’s a lot of value in getting concrete examples from real customers and business experts; not so much value in using “natural language” tools like Fit and similar.

The two failure patterns that Shore describes in his post are falling back on testers to write everything and merging acceptance and integration tests. I’ve experienced both of these myself, and it seems that they are common in general. We discussed both during the top 10 ways to fail with acceptance testing openspace session at CITCON Europe last year. However, there are good ways to solve both problems.

I never really expected customers to write anything themselves, but I was relatively successful in persuading them to participate in specification workshops that led to examples which were then converted to acceptance tests later. Another idea I discovered while doing the research for my new book is discussing the key examples with customers and then going off to write detailed test pages, which then get verified by the customers. The third good idea is doing ad-hoc specification writing sessions when a developer needs information, by involving a tester and a business analyst. This is a lot less formal than a specification workshop and gives you similar benefits if you have all the knowledge in the room (or readily available) most of the time.

Not preserving acceptance tests as a separate group and mixing quick and slow tests is something that most people, at least according to my ongoing research, get burned with at some point but again teams learn from that and improve.

One of the biggest benefits from acceptance testing for me was that the teams finally get a source of information on what goes on in the system as reliable as the code itself. Without acceptance tests, code is the only thing you can really trust and any other documentation gets outdated very quickly. And such tests are much easier to read and understand than the code because they are on a higher level and in a natural language. Having a live specification helps me quite a lot when change requests come in later. It also helps with handing over and taking over code. Acceptance tests stay relevant throughout the project because they are automated, and automated tests are kept up to date in order for them to pass. Automation, and consequently a tool, are necessary to get this benefit. With informal agreements and on-site reviews that Jim Shore describes, I guess something else needs to be in place to facilitate this.

I agree with Shore that it takes a while for the problems with tools such as FIT to surface, but I’m not sure whether that is tool related or not. Most people I spoke to so far said that it took them between six months and a year to discover that acceptance testing isn’t about the tools but about communication, and that the biggest benefit is in the examples as Shore wrote. A notable exception to six months to a year rule was Lisa Crispin’s team who generally started out knowing what they need (but that’s because she had done it before). Clear examples and improved communication are the biggest benefits of the process, but using a tool brings some additional nice benefits as well. A tool gives us an impartial measure of progress. Ian Cooper said during the interview for my new book that “the tool keeps developers honest”, and I can certainly relate to that. With tests that are evaluated by an impartial tool, “done” is really “what everyone agreed on”, not “almost done with just a few things to fill in tomorrow”. I’m not sure whether an on-site review is enough to guard against this completely.

How would you effectively define a sufficient set of If-When-Then scenarios to test for correctness what is potentially an extremely large set of transformations?

BDD scenarios (and agile acceptance tests in general) are not about full regression testing – they work best as a set of examples that demonstrate business rules which need to be developed, and as the software matures become live documentation of what the system does. So they typically should contain:

• A representative example demonstrating each important aspect of a business rule. Business users, analysts or customers will typically lead defining these.
• An example demonstrating each important technical edge case (technical boundary conditions) developers will typically suggest cases here. Business users, analysts or customers will define the correct expected behaviour.
• An example demonstrating each particularly troublesome area of the expected implementation (such as cases that caused bugs in the past, boundary conditions that might not be explicitly demonstrated by previous examples etc). Testers will typically suggest these and business users, analysts or customers will define the correct behaviour.

I would start by asking business users to define important examples, then developers and testers to comment on these examples and suggest important conditions and cases not covered by existing examples. I would do this in a specification workshop to allow everyone to benefit from a discussion and build a shared understanding of the problem. At the point where everyone is happy that they have enough information to work, we have enough examples. Then I would capture the examples discussed during the workshop, identify potential duplicates, remove information not directly related to the business rules for the cases and formalise them as acceptance tests/BDD scenarios.

This is not to say that additional examples cannot be added later to provide a more complete functional regression check. This can even be automated using the same scenario step format and automation tools and helpers as the other examples. But I like to keep the two separate so that there is a small and focused specification that can be easily read and understood later, when things change. When using tools such as FitNesse, you can link the two sets of examples with an “For Additional Examples, See …” link to point people to a full regression set later. Having them separate also allows you to quickly run the entire specification and get faster feedback.

They are thinking about using Concordion here. I remember you made some comments about using it. Can it be used sensibly by a tester without lots of dev experience?

Concordion is a great tool for acceptance testing as support for development. Whether it can be used sensibly by a tester without lots of development experience, that depends on what the intended use is. Concordion tests/results are HTML files, so anyone can read them using a browser. I’ve never tried to write Concordion tests using Word or anything similar, only by hand-coding HTML, so I don’t know whether tests can be maintained with a visual tool. However, anyone with basic HTML knowledge should be able to write tests as well. In terms of running the tests, Concordion runs within JUnit/NUnit, so this should be fairly simple as well.

Concordion does not have a test management tool and intentionally doesn’t allow you to reuse or share automation components, so it requires a fair bit of cooperation between developers and testers. I think this is very good for tests that support development, but it might be a problem for retro-fitting regression test packs into an existing product if testers are expected to do the bulk of work.

I wrote a review of concordion last year, which will give you a bit more detail on how it works. I also have a short video about it (.NET oriented but still a good introduction to capabilities):

David Peterson, the author of Concordion, spoke about it at the agile acceptance testing tools round-up event, and that video is also online.

1. No collaboration — as a specification of a system that acts as the shared definition of done, acceptance tests need to be collaboratively produced and agreed upon. Business people, developers and testers should all provide input, each from their own area of expertise. The discussion that happens around the specification is actually more valuable than the tests themselves, as people build a shared understanding of the domain and the problem at hand. If this discussion doesn’t happen, then teams still feel the effects of the telephone game. Symptoms of this problem are developers writing acceptance tests themselves, testers being expected to handle everything about acceptance testing and business dictating tests without any feedback from developers or testers. The solution for this are collaborative specifications and specification workshops.
2. Focusing on how, not on what — business specifications should be clear and understandable and allow the team to implement the best possible solution without prescribing the design or the implementation. Too often tests describe how something should be implemented rather than what should be implemented. Tests such as these are too low level, hard to understand and always turn out to be a pain to maintain long-term. Instead of facilitating system change they inhibit it as people will be reluctant to introduce changes that break a lot of tests. Symptoms of this problem are action-oriented acceptance tests, lots of workflows or duplication in tests and tests that are very technical. The solutions for this are communicating intent and distilling the specification.
3. Tests unusable as live documentation — one of the greatest benefits of agile acceptance testing is that we are gradually building a human readable specification of the system. This specification is ideally automated to a great degree so that we can be confident that it is in sync with the code. This solves the problem of the running code being the only thing you can really trust about what the system does. Correct, easily understandable and accessible documentation is crucial for future change. If tests don’t serve as a live documentation that those benefits are lost. Symptoms of this problem are very technical tests, long tests, hard to understand tests, tests that are poorly organised so that you can’t quickly find relevant tests for a particular piece of functionality. Solutions for this is to ensure that tests can be used as a live documentation, reorganising them and updating when the ubiquitous language changes.
4. Expecting acceptance tests to be a full regression suite — acceptance tests are primarily a specification of what the system should do and should deal with exemplars that represents whole sets of cases. Specifying too much and covering all possible edge cases will make them hard to understand and might introduce an effect similar to paralysis by analysis. So acceptance tests cannot provide a full regression test suite and additional tests are required (including exploratory testing). Symptoms of this problem are tests that are very long and verify a large number of similar cases. Note that this doesn’t mean that regression tests can’t be written and automated using the same tools as acceptance tests.
5. Focusing on tools — some teams focus too much on tools and features of particular tools, disregarding things that do not naturally fit into their chosen toolset. This makes the specification vague in parts that aren’t covered by a particular tool. Instead of that, teams should be focusing on the business specifications and then choosing the right tool for the job. In particular, tools should not play an important part in the specification workshop. Symptoms of this problem are ignoring the UI (as UI tests are hard to automate) and wasting time on converting specifications in a format suitable for a particular tool during the specification workshops (which breaks the flow of the discussion and wastes valuable time).
6. Not considering acceptance testing as value added activity — agile acceptance testing is not about QA, but more about specifying and agreeing on what needs to be done. Teams that consider it part of QA often delegate the responsibility to it to junior programmers and testers, which effectively means that junior team members will be writing the specification (which is wrong on so many levels that I don’t know where to start). The solutions for this are specification workshops.
7. “Test code” not maintained with love — tests and code to automate them are often considered as less important than production code, so given to less capable developers and testers to implement and maintain. This leads to huge maintenance problems later and can cause teams to completely abandon full suites of tests. Acceptance tests are the specification of a system so they are crucial to the success of a project. They also need to be maintained and kept in sync with the live project language and design in order to serve as a live documentation.
8. Objectives of team members not aligned — if the job of business analysts is only to deliver the specifications, the job of developers only to ship the system and the job of testers to ensure quality, then people won’t pull in the same direction or invest time in producing good acceptance tests. This is a management issue but failures with acceptance testing might make it more visible.
9. No management buy-in — successful acceptance testing requires an active participation of the business and with no management buy-in the benefits will simply not be there (also related to #1).
10. Underestimating the skill required to do this well — introducing acceptance testing is a tall order and often means changing the way teams are organised and approach specifications. This requires a lot of time and investment in skill-up, mastering tools, facilitating workshops and dealing with resistance to change. Underestimating the effort required to do this properly might cause teams to think that they failed and get disappointed too early.

Xavier Warzee filmed the session so watch the videos if you are interested to see more of this discussion (scroll down on the page, it’s the last batch of videos). You can also check out two more write-ups by Nicolas Martignole and Xavier Bourguignon (both in French) and session notes from the CITCON wiki.

If applying acceptance testing and BDD effectively is something that you want to learn more about, you might want to register for the Writing Software that Matters workshop (Dec 4th, central London). I also run on-site workshops that help teams get started and implement agile acceptance testing.

Image credits: Cirilo Wortel