The most useful software models are the ones where the technical view of the world is aligned with the relevant business domain model. This ensures that one small change in the business domain (new requirements or changes to existing features) results in one small change in software. This is one of the key ideas of Domain Driven Design. The alignment of the technical and the business models ensures that software evolves nicely with the business and that we never get into a position where a small report change requires a rewrite of the entire system or six months to implement.

The same is true for the other artefacts produced for software — especially tests and documentation. When the conceptual model used in the tests is aligned with the business domain model, one small change in business will result in one small change in tests, making the tests easy to maintain. This applies to the language used in the tests, the abstractions used to describe features and the way that the tests are organised. So I propose the following principle to get good executable specifications/acceptance tests/bdd scenarios/feature files/fitnesse pages or whatever you call them:

The principle of symmetric change
One small change in a business model should require one small change to executable specifications.

To achieve this, ensure that:

• each specification should be focused on a single concept of the business domain model (which might be a business rule, step in a flow, or something higher such as the overall definition of a business process)
• the concepts used in key examples in a specification are closely aligned with the business domain model and reflected in underlying software domain model. this includes naming (concepts are described in the business domain language) but also the relationships between them.
• the (automated) validation procedure is clearly separated from the specification of key examples
• the specifications are organised into a living documentation system according to the conceptual relationships that exist in the business domain model

Thinking about aligning key examples and specifications with the business domain model instantly solves many of classic acceptance testing/bdd problems we identified during a CITCON workshop in Paris last year. For example, “Focusing on How, not What” is a sure way to write tests that are very hard to maintain but lots of teams still do it. By thinking about aligning the language and abstraction concepts in the specifications with the business domain model, we force ourselves to describe what the software should do, not how it does it. This separates the specification (what) from the validation procedure (how can we check it in software). The validation procedure can then be automated using a tool. This is a classic problem that Ward Cunningham and Rick Mugridge have identified in Fit For Developing Software nand advised readers not to automate flows. But that is an overly generalised statement – it confuses people who deal with genuine workflows, for example in payment systems where business workflows are everything. David Peterson advised people to focus on specification over scripting, which is less generalised but still not directly to the point. Flows or scripts aren’t the real problem – it is mixing the business definition of a function with the validation procedure.

Thinking about specifications and tests in this way also prevents people from falling into the trap of The Sine of Death by UI Test Automation. Tests described with low level technical detail, in the language of technical UI interactions, are everything but aligned with a business model. If anything, they are aligned with the current design and layout of user interfaces. A small change in business requirements can have a huge impact on such tests, requiring hours of updates to tests after just a few minutes of changes to the code, which then fails the symmetric change principle.

Ensuring that the specifications are aligned with the business model and in the business language, organised according to the relationships the underlying concepts have in the business model solves all the reasons behind the “Tests unusable as live documentation” family of problems, and ensuring that the previous documents are still aligned with the new model resolves most of the issues we described as “Test code not maintained with love”.

I’ll also be speaking at the next BCS SIGIST conference on December 8th on winning big with agile acceptance testing.

The main author of this e-book is my colleague David de Florinier, and I’ve also contributed a bit.

The first version covers the basics of Cucumber, setting up for Ruby, Java and .NET projects, implementing basic features in all three languages, effective test design and managing complex features efficiently. We plan to develop this guide incrementally, and the next version will include web browser automation topics.

To download the PDF or read the e-Book online, point your browser to cuke4ninja.com. For news and updates, follow cuke4ninja.

By using better names, we can avoid a lot of completely meaningless discussions and get people started on the right path straight away. So here is what I propose:

One of the biggest issues teams have with acceptance testing is who should write what and when. So we need to come up with a good name for the start of the process that clearly says that everyone should be involved, and that this needs to happen before developers start developing and testers start testing, because we want to use acceptance tests as a target for development. Test first is a good technical name for it, but business users don’t get it. I propose we talk about Specifying Collaboratively instead of test first or writing acceptance tests.

It sounds quite normal to put every single numerical possibility into an automated functional test, why wouldn’t you do it if it is automated. But such complex tests are unusable as a communication tool. So instead of writing functional tests, let’s talk about Illustrating with Examples (thanks, Dave) and expect the output of that to be Key Examples to point out that we only want enough to explain the context properly.

Key examples are raw material, but if we just talk about acceptance testing then why not just dump all those complicated 50-column 100-row tables into an acceptance test without any explanation, it’s going to be tested by a machine anyway. But these tests are for humans as well for machines, so let’s talk about a whole new step after this, about the process of extracting the minimal set of attributes and examples to specify a business rule, and about adding a title, description and so on. I propose we call this Distilling the specification.

I just don’t want to spend any more arguing with people who already paid a license for QTP that it is completely unusable for acceptance tests. As long as we talk about test automation, there is always going to be a push to use whatever horrible contraption testers already use for automation, because it’s logical to use a single tool. Acceptance testing tools don’t compete with QTP or things like that, they address a completely different problem. Instead of talking about test automation, let’s talk about automating a specification without distorting any information – Literal Automation. Literal automation also avoids the whole scripting horror and using technical libraries directly in test specifications. If it’s literal, it should look as it looked on the whiteboard, it should not be translated to selenium commands.

After Literal Automation, we get a specification that can be checked against the code automatically, an Executable Specification.

We want to run all the acceptance tests frequently to make sure that the system still does what it is supposed to do (and equally more important to check that the specification still says what the system does). If we call this regression testing, it’s very hard to explain to testers why they should not go and add five million other test cases to a previously nice, small and focused specification. If we talk about continuous integration, then we get into the trouble of explaining why these tests should not always be end-to-end and run the whole system. On the top of that, for some legacy systems we need to run acceptance tests against a live, deployed environment. Technical integration tests run before deployment. So let’s not talk about regression testing or continuous integration, let’s talk about
Continuous Validation (or even Frequent Validation).

The long term pay-off from agile acceptance testing is having a reference on what the system does that is as relevant as the code itself, but much easier to read. That makes development much more efficient long term, facilitates collaboration with business users, leads to an alignment of software design and business models and just makes everyone’s life much easier. But to do this, the reference really has to be relevant, it has to be maintained, it has to be consistent with itself and with code. we should not have silos of tests that use terms we had three years ago, and those we used a year ago, and so on. Going back and updating tests is a very hard thing to sell to teams who are busy, but going back to update documentation after a big change is expected. So let’s not talk about folders filled with hundreds of tests, let’s talk about a Living Documentation system. That makes it much easier to explain why things should be self-explanatory, why business users need access to this as well and why it has to be nicely organised so that things are easy to find.

What do you think about this? Does it make sense? Will it help? Do you have a better name for one of these concepts, that explains it more clearly?

Dan Leong on Clean Acceptance Tests

This presentation discusses how our agile team renewed our focus and understanding of our acceptance tests when the team members changed. Our group found some core shared values in the context of acceptance testing which we expressed in the style of the agile manifesto. We then looked at our existing tests to find bad test smells that we could learn from. The whole exercise was a good experience and we encourage you to try something similar in your teams.

Dan Leong is a team lead at Sky Network Services, where they have been using agile/XP techniques for over 4 years to deliver the company’s broadband and voice provisioning system. He has over 10+ experience working in companies ranging from small .com start-ups to global advertising and media companies. Like the rest of us, he’s trying to figure out how to do things better.

The event is free to attend, but up front registration is required. Register now