By using better names, we can avoid a lot of completely meaningless discussions and get people started on the right path straight away. So here is what I propose:

One of the biggest issues teams have with acceptance testing is who should write what and when. So we need to come up with a good name for the start of the process that clearly says that everyone should be involved, and that this needs to happen before developers start developing and testers start testing, because we want to use acceptance tests as a target for development. Test first is a good technical name for it, but business users don’t get it. I propose we talk about Specifying Collaboratively instead of test first or writing acceptance tests.

It sounds quite normal to put every single numerical possibility into an automated functional test, why wouldn’t you do it if it is automated. But such complex tests are unusable as a communication tool. So instead of writing functional tests, let’s talk about Illustrating with Examples (thanks, Dave) and expect the output of that to be Key Examples to point out that we only want enough to explain the context properly.

Key examples are raw material, but if we just talk about acceptance testing then why not just dump all those complicated 50-column 100-row tables into an acceptance test without any explanation, it’s going to be tested by a machine anyway. But these tests are for humans as well for machines, so let’s talk about a whole new step after this, about the process of extracting the minimal set of attributes and examples to specify a business rule, and about adding a title, description and so on. I propose we call this Distilling the specification.

I just don’t want to spend any more arguing with people who already paid a license for QTP that it is completely unusable for acceptance tests. As long as we talk about test automation, there is always going to be a push to use whatever horrible contraption testers already use for automation, because it’s logical to use a single tool. Acceptance testing tools don’t compete with QTP or things like that, they address a completely different problem. Instead of talking about test automation, let’s talk about automating a specification without distorting any information – Literal Automation. Literal automation also avoids the whole scripting horror and using technical libraries directly in test specifications. If it’s literal, it should look as it looked on the whiteboard, it should not be translated to selenium commands.

After Literal Automation, we get a specification that can be checked against the code automatically, an Executable Specification.

We want to run all the acceptance tests frequently to make sure that the system still does what it is supposed to do (and equally more important to check that the specification still says what the system does). If we call this regression testing, it’s very hard to explain to testers why they should not go and add five million other test cases to a previously nice, small and focused specification. If we talk about continuous integration, then we get into the trouble of explaining why these tests should not always be end-to-end and run the whole system. On the top of that, for some legacy systems we need to run acceptance tests against a live, deployed environment. Technical integration tests run before deployment. So let’s not talk about regression testing or continuous integration, let’s talk about
Continuous Validation (or even Frequent Validation).

The long term pay-off from agile acceptance testing is having a reference on what the system does that is as relevant as the code itself, but much easier to read. That makes development much more efficient long term, facilitates collaboration with business users, leads to an alignment of software design and business models and just makes everyone’s life much easier. But to do this, the reference really has to be relevant, it has to be maintained, it has to be consistent with itself and with code. we should not have silos of tests that use terms we had three years ago, and those we used a year ago, and so on. Going back and updating tests is a very hard thing to sell to teams who are busy, but going back to update documentation after a big change is expected. So let’s not talk about folders filled with hundreds of tests, let’s talk about a Living Documentation system. That makes it much easier to explain why things should be self-explanatory, why business users need access to this as well and why it has to be nicely organised so that things are easy to find.

What do you think about this? Does it make sense? Will it help? Do you have a better name for one of these concepts, that explains it more clearly?

The webinar is tomorrow. Register now

Sounds familiar? Read How to implement UI Testing without shooting yourself in the foot

Dan Leong on Clean Acceptance Tests

This presentation discusses how our agile team renewed our focus and understanding of our acceptance tests when the team members changed. Our group found some core shared values in the context of acceptance testing which we expressed in the style of the agile manifesto. We then looked at our existing tests to find bad test smells that we could learn from. The whole exercise was a good experience and we encourage you to try something similar in your teams.

Dan Leong is a team lead at Sky Network Services, where they have been using agile/XP techniques for over 4 years to deliver the company’s broadband and voice provisioning system. He has over 10+ experience working in companies ranging from small .com start-ups to global advertising and media companies. Like the rest of us, he’s trying to figure out how to do things better.

The event is free to attend, but up front registration is required. Register now

Although we have a long way to go things are going OK at the moment and we feel it is bringing some real focus to the development process. However a lot of the early stories are largely admin type CRUD – for
example functionaility to set up user defined entities and their user defined properties within the system and to provide a mechanism for relating these entities to each other
…
Does anyone have any advice about how to write tests for this sort of stuff or any experiences in starting out with BDD they can share.

CRUD is not a user story, it’s a screen. It’s not a business function, but implementation detail. Why do the business users need a particular CRUD screen? (I know it sounds as a stupid rhetorical question, but I’m serious) What does it allow them to do?

Often you don’t need to implement an entire CRUD screen and deliver them one by one. Sometimes there is value in releasing two screens that allow you to set a subset of properties but together they bring value. You can then automate these tests through the UI and use the CRUD screens, but that will be hidden in the automation layer. Say for example that we have a risk report that lists users and their card numbers:

Scenario: Only regular customers with a specific risk category and 
              card type show up in risk reports
Given the following users
|name| type| card type| card number| risk category|
|Mike |VIP | Mastercard |53111 11111 11111 1111|X|
|Tom |VIP| Visa |41111 11111 11111 1111|Y|
John |Regular |Mastercard |51111 11111 11111 1111|X|
|Steve |Regular |Mastercard |52111 11111 11111 1111|Y|
Then the risk report for Mastercard and category X contains the following data
|John | 51111 11111 11111 1111|

This could, for example, invoke the user CRUD to save a user name, type and risk category and a completely different CRUD to save card details for that user. Any other information that would go on that CRUD (addresses, card expiry dates etc) aren’t part of this story or criteria because they are not important for this particular report.

Start with the outputs, the reports that the system produces, instead of data entry operations. this ensures that you have the data you need to produce the reports at the end, and that you don’t have superfluous data that nobody really cares about. if you don’t do that, the resulting data schemas are often overcomplicated and contain many things that simply aren’t used at the end at all.