I got this question from a reader today:
How would you effectively define a sufficient set of If-When-Then scenarios to test for correctness what is potentially an extremely large set of transformations?
BDD scenarios (and agile acceptance tests in general) are not about full regression testing - they work best as a set of examples that demonstrate business rules which need to be developed, and as the software matures become living documentation of what the system does. So they typically should contain:
- A representative example demonstrating each important aspect of a business rule. Business users, analysts or customers will typically lead defining these.
- An example demonstrating each important technical edge case (technical boundary conditions) developers will typically suggest cases here. Business users, analysts or customers will define the correct expected behaviour.
- An example demonstrating each particularly troublesome area of the expected implementation (such as cases that caused bugs in the past, boundary conditions that might not be explicitly demonstrated by previous examples etc). Testers will typically suggest these and business users, analysts or customers will define the correct behaviour.
I would start by asking business users to define important examples, then developers and testers to comment on these examples and suggest important conditions and cases not covered by existing examples. I would do this in a specification workshop to allow everyone to benefit from a discussion and build a shared understanding of the problem. At the point where everyone is happy that they have enough information to work, we have enough examples. Then I would capture the examples discussed during the workshop, identify potential duplicates, remove information not directly related to the business rules for the cases and formalise them as acceptance tests/BDD scenarios.
This is not to say that additional examples cannot be added later to provide a more complete functional regression check. This can even be automated using the same scenario step format and automation tools and helpers as the other examples. But I like to keep the two separate so that there is a small and focused specification that can be easily read and understood later, when things change. When using tools such as FitNesse, you can link the two sets of examples with an “For Additional Examples, See …” link to point people to a full regression set later. Having them separate also allows you to quickly run the entire specification and get faster feedback.
