Now go to http://watirmelon.com/2011/05/18/specification-by-example-a-love-story/ and enjoy!

I listened to your podcast with Scott Hanselman and the discussion touched on the mix of unit tests and BDD acceptance tests, but didn’t really go as far as to offer guidance on where each is really best suited.

Since the BDD tests are integration tests it would seem that for ultimate confidence in the BDD tests as a regression suite they should cover all of the corner cases that the unit tests cover where possible (strange input cases that lead to null references, divide by zero, zero, one, edge case testing, etc) and be exposed to the BA’s to ensure that the code is handling these as expected. This seems to require essentially a near duplication of the cases handled in unit tests to achieve this level. The tests in the unit tests are still useful as they give guidance to developers on which layer the expectation is failing at, especially if the tests are layered with mocks stuffed with the correct behaviors at higher levels.

All of the examples of BDD I’ve seen follow mostly the happy path with only an occasional nod to the unhappy path and don’t seem to go into as many corner cases and essentially code coverage as unit tests too, but this feels incomplete.

I saw this article referenced in the show note comments, but it also felt incomplete

I’m curious if you have any advice or good examples of projects covered by both BDD and unit tests that achieve a good mix.

Regarding the division of responsibilities, I like Brian Marick’s quandrants as a model, they explain that nicely.

I disagree with the statement that BDD tests are integration tests. BDD tests are a functional specification of a system (in the quadrant model, they are business facing and supporting the team), unit tests guide technical design (in the quadrant model, team facing and supporting). They address two different issues. Unit tests tell you that the system works as the programmers wanted it to in the technical aspects, BDD scenarios tell you that the right functionality is there from a business perspective. They interact with the system on completely different levels and have different key examples. Null pointers don’t exist in most businesses. Integration tests check that the system components are connected correctly and that they talk to each other (including third parties), not that the business functionality is there.

I wrote about this problem of mixing up BDD and integration tests recently.

If the BDD scenarios you’ve seen cover only happy paths, you haven’t seen good examples then. They should contain key examples – positive and negative. See what Cuke4Ninja has to say on the key examples. (in the cucumber language, a feature file is a bdd test).

For examples of projects that got BDD right, see specificationbyexample.com.

The seminar should specifically fit people who are too busy to attend training otherwise, such as team leaders, managers and senior technical people. Click here for more information and to register

The most useful software models are the ones where the technical view of the world is aligned with the relevant business domain model. This ensures that one small change in the business domain (new requirements or changes to existing features) results in one small change in software. This is one of the key ideas of Domain Driven Design. The alignment of the technical and the business models ensures that software evolves nicely with the business and that we never get into a position where a small report change requires a rewrite of the entire system or six months to implement.

The same is true for the other artefacts produced for software — especially tests and documentation. When the conceptual model used in the tests is aligned with the business domain model, one small change in business will result in one small change in tests, making the tests easy to maintain. This applies to the language used in the tests, the abstractions used to describe features and the way that the tests are organised. So I propose the following principle to get good executable specifications/acceptance tests/bdd scenarios/feature files/fitnesse pages or whatever you call them:

The principle of symmetric change
One small change in a business model should require one small change to executable specifications.

To achieve this, ensure that:

• each specification should be focused on a single concept of the business domain model (which might be a business rule, step in a flow, or something higher such as the overall definition of a business process)
• the concepts used in key examples in a specification are closely aligned with the business domain model and reflected in underlying software domain model. this includes naming (concepts are described in the business domain language) but also the relationships between them.
• the (automated) validation procedure is clearly separated from the specification of key examples
• the specifications are organised into a living documentation system according to the conceptual relationships that exist in the business domain model

Thinking about aligning key examples and specifications with the business domain model instantly solves many of classic acceptance testing/bdd problems we identified during a CITCON workshop in Paris last year. For example, “Focusing on How, not What” is a sure way to write tests that are very hard to maintain but lots of teams still do it. By thinking about aligning the language and abstraction concepts in the specifications with the business domain model, we force ourselves to describe what the software should do, not how it does it. This separates the specification (what) from the validation procedure (how can we check it in software). The validation procedure can then be automated using a tool. This is a classic problem that Ward Cunningham and Rick Mugridge have identified in Fit For Developing Software nand advised readers not to automate flows. But that is an overly generalised statement – it confuses people who deal with genuine workflows, for example in payment systems where business workflows are everything. David Peterson advised people to focus on specification over scripting, which is less generalised but still not directly to the point. Flows or scripts aren’t the real problem – it is mixing the business definition of a function with the validation procedure.

Thinking about specifications and tests in this way also prevents people from falling into the trap of The Sine of Death by UI Test Automation. Tests described with low level technical detail, in the language of technical UI interactions, are everything but aligned with a business model. If anything, they are aligned with the current design and layout of user interfaces. A small change in business requirements can have a huge impact on such tests, requiring hours of updates to tests after just a few minutes of changes to the code, which then fails the symmetric change principle.

Ensuring that the specifications are aligned with the business model and in the business language, organised according to the relationships the underlying concepts have in the business model solves all the reasons behind the “Tests unusable as live documentation” family of problems, and ensuring that the previous documents are still aligned with the new model resolves most of the issues we described as “Test code not maintained with love”.

The main author of this e-book is my colleague David de Florinier, and I’ve also contributed a bit.

The first version covers the basics of Cucumber, setting up for Ruby, Java and .NET projects, implementing basic features in all three languages, effective test design and managing complex features efficiently. We plan to develop this guide incrementally, and the next version will include web browser automation topics.

To download the PDF or read the e-Book online, point your browser to cuke4ninja.com. For news and updates, follow cuke4ninja.