My experience with Fit and other agile acceptance testing tools is that they cost more than they’re worth. There’s a lot of value in getting concrete examples from real customers and business experts; not so much value in using “natural language” tools like Fit and similar.

The two failure patterns that Shore describes in his post are falling back on testers to write everything and merging acceptance and integration tests. I’ve experienced both of these myself, and it seems that they are common in general. We discussed both during the top 10 ways to fail with acceptance testing openspace session at CITCON Europe last year. However, there are good ways to solve both problems.

I never really expected customers to write anything themselves, but I was relatively successful in persuading them to participate in specification workshops that led to examples which were then converted to acceptance tests later. Another idea I discovered while doing the research for my new book is discussing the key examples with customers and then going off to write detailed test pages, which then get verified by the customers. The third good idea is doing ad-hoc specification writing sessions when a developer needs information, by involving a tester and a business analyst. This is a lot less formal than a specification workshop and gives you similar benefits if you have all the knowledge in the room (or readily available) most of the time.

Not preserving acceptance tests as a separate group and mixing quick and slow tests is something that most people, at least according to my ongoing research, get burned with at some point but again teams learn from that and improve.

One of the biggest benefits from acceptance testing for me was that the teams finally get a source of information on what goes on in the system as reliable as the code itself. Without acceptance tests, code is the only thing you can really trust and any other documentation gets outdated very quickly. And such tests are much easier to read and understand than the code because they are on a higher level and in a natural language. Having a live specification helps me quite a lot when change requests come in later. It also helps with handing over and taking over code. Acceptance tests stay relevant throughout the project because they are automated, and automated tests are kept up to date in order for them to pass. Automation, and consequently a tool, are necessary to get this benefit. With informal agreements and on-site reviews that Jim Shore describes, I guess something else needs to be in place to facilitate this.

I agree with Shore that it takes a while for the problems with tools such as FIT to surface, but I’m not sure whether that is tool related or not. Most people I spoke to so far said that it took them between six months and a year to discover that acceptance testing isn’t about the tools but about communication, and that the biggest benefit is in the examples as Shore wrote. A notable exception to six months to a year rule was Lisa Crispin’s team who generally started out knowing what they need (but that’s because she had done it before). Clear examples and improved communication are the biggest benefits of the process, but using a tool brings some additional nice benefits as well. A tool gives us an impartial measure of progress. Ian Cooper said during the interview for my new book that “the tool keeps developers honest”, and I can certainly relate to that. With tests that are evaluated by an impartial tool, “done” is really “what everyone agreed on”, not “almost done with just a few things to fill in tomorrow”. I’m not sure whether an on-site review is enough to guard against this completely.

I’ll be presenting Cucumber a tool for behaviour-driven development and agile acceptance testing. I will demonstrate how to use Cucumber for Java, .NET and Ruby applications, talk about new Cucumber features and best practices for writing and maintaining Cucumber scenarios.

The event is free, but up-front registration is required for capacity planning. For more information and to register click here

How would you effectively define a sufficient set of If-When-Then scenarios to test for correctness what is potentially an extremely large set of transformations?

BDD scenarios (and agile acceptance tests in general) are not about full regression testing – they work best as a set of examples that demonstrate business rules which need to be developed, and as the software matures become live documentation of what the system does. So they typically should contain:

• A representative example demonstrating each important aspect of a business rule. Business users, analysts or customers will typically lead defining these.
• An example demonstrating each important technical edge case (technical boundary conditions) developers will typically suggest cases here. Business users, analysts or customers will define the correct expected behaviour.
• An example demonstrating each particularly troublesome area of the expected implementation (such as cases that caused bugs in the past, boundary conditions that might not be explicitly demonstrated by previous examples etc). Testers will typically suggest these and business users, analysts or customers will define the correct behaviour.

I would start by asking business users to define important examples, then developers and testers to comment on these examples and suggest important conditions and cases not covered by existing examples. I would do this in a specification workshop to allow everyone to benefit from a discussion and build a shared understanding of the problem. At the point where everyone is happy that they have enough information to work, we have enough examples. Then I would capture the examples discussed during the workshop, identify potential duplicates, remove information not directly related to the business rules for the cases and formalise them as acceptance tests/BDD scenarios.

This is not to say that additional examples cannot be added later to provide a more complete functional regression check. This can even be automated using the same scenario step format and automation tools and helpers as the other examples. But I like to keep the two separate so that there is a small and focused specification that can be easily read and understood later, when things change. When using tools such as FitNesse, you can link the two sets of examples with an “For Additional Examples, See …” link to point people to a full regression set later. Having them separate also allows you to quickly run the entire specification and get faster feedback.

Story estimation is one of those things where the process of doing it is more important than the output. Planning poker is a very effective technique to discover differences in understanding. However, BDD achieves the same by getting an agreement on scenarios. Specification workshops go much further than planning poker in building a shared understanding of the tasks at hand and definition of done. Teams that practice BDD or agile acceptance testing could skip planning poker if only there was some other effective way to get a sensible feel of how long a project phase will take. That’s why I find Dan’s idea so appealing.

On a decent-size project, the estimates will average out anyway. Treating all stories as medium size and figuring out how long a medium story will take will not be any less accurate than doing more detailed estimates and adding them up. It will, however, save quite a lot of time.

As a nice side effect, this should also produce a better breakdown of stories. Any story that obviously stands out by complexity will have to be broken down into smaller pieces, as we are not allowed to give it a higher valuation. Evenly-sized stories make it easier to establish a constant pace of delivery.

They are thinking about using Concordion here. I remember you made some comments about using it. Can it be used sensibly by a tester without lots of dev experience?

Concordion is a great tool for acceptance testing as support for development. Whether it can be used sensibly by a tester without lots of development experience, that depends on what the intended use is. Concordion tests/results are HTML files, so anyone can read them using a browser. I’ve never tried to write Concordion tests using Word or anything similar, only by hand-coding HTML, so I don’t know whether tests can be maintained with a visual tool. However, anyone with basic HTML knowledge should be able to write tests as well. In terms of running the tests, Concordion runs within JUnit/NUnit, so this should be fairly simple as well.

Concordion does not have a test management tool and intentionally doesn’t allow you to reuse or share automation components, so it requires a fair bit of cooperation between developers and testers. I think this is very good for tests that support development, but it might be a problem for retro-fitting regression test packs into an existing product if testers are expected to do the bulk of work.

I wrote a review of concordion last year, which will give you a bit more detail on how it works. I also have a short video about it (.NET oriented but still a good introduction to capabilities):

David Peterson, the author of Concordion, spoke about it at the agile acceptance testing tools round-up event, and that video is also online.