Early on we started breaking down the exercises designed to teach basic ideas (such as test first and unit testing tool usage) and exercises designed to teach design or technique subtleties. There wasn’t much interest in discussing basic exercises apart from the idea that they should be as simple as possible, so that people don’t get confused by the problem itself. Someone suggested the String Calculator exercise as a good example of this.

As examples of good exercises that teach advanced TDD techniques and subtleties, the group suggested the following:

• Robocode competitions seem to be good in teaching evolutionary design. Using Google, I found a reference to a TDD Robocode workshop from XPDay 2 organised by Paul Simmons and Duncan Pierce with detailed instructions on how to run the exercise.
• TDD as if you meant it, conceived by Keith Braithwaite and ran for the first time during the Software Craftsmanship 09 conference, is a great way to expose design presumptions and make people think hard about actually driving the design by tests.
• I use the Roulette Wheel exercise during my TDD In Practice workshops to make people run into the trap of trying to apply TDD with temporal constraints and random events. There is no write-up of this exercise online yet, but generally the problem is to describe a simple rule such as “the wheel will spin for twenty seconds and the ball will stop on a random number” using unit tests and design the wheel functionality with TDD. Solving the problem is a good way make people think what actually they need to describe in a test, where the context of that particular unit ends, to teach hexagonal architecture principles and using mocks for design.
• Refactoring Golf, an exercise suggested for Software Craftsmanship 2010, teaches refactoring techniques by giving participants the code “before” and “after” and they have to refactor it to get from one shape to another in a disciplined way, using IDE tools as much as possible. While discussing this exercise, someone said that it was particularly effective when done on real project code for in-house workshops. In-house coaches can use this exercise to teach team members why the code they wrote is suboptimal and how to fix it. “Bottling code smells” was mentioned as a really good idea to facilitate this: if a class needs to be refactored, team members should capture a class before the refactoring and use that as a starting point for later exercises.
• Pairing Roulette, where participants frequently switch pairs to work on the same problem over and over, such as in Corey Haines’ code retreat events, is a great way to learn how other people are doing TDD and to see the value of naming conventions and learn different techniques for that.

Analysing further these exercises, we tried to identify as what makes a good TDD exercise. These are the ideas we identified as common for all of them:

• All of them require participants to do something in a very constrained and disciplined way. This helps focus the exercise on a particular thing and make it possible to do a meaningful piece of work with lots of people in a short time. Some further ideas how this could be applied was to impose a constraint that setters or getters cannot be used (for example, to teach Tell Don’t Ask)
• All of them provide relatively instant feedback. Further ideas on how this can be applied to improve other exercises, mostly coming from Jason Gorman‘s work on pair-coaching TDD, are to work against checklists, get pairs to asses themselves (Gorman advises still having a principle coach review this to avoid the problem of blind leading blind and also to switch pairs to get more objective assessments) and playing the role of a teacher (which makes participants think harder about something because they have to explain it). Another great idea coming from the Software Craftsmanship 2010 submission process is to record yourself doing an exercise and watch the video – this technique has been used for a long time by public speakers to “step out of the box” and see their flaws more objectively.
• All of them start with a full context that is easy to understand. Simple tasks, more or less already known to the participants, such as the Convay’s game of life, work best because they allow participants to focus on the goal of the workshop and not waste time understanding the problem to be solved. For me, the Tic Tac Toe variant of TDD As If You Meant It worked much better than the original Go version for that same reason. Having a full context is also very important because it allows people to get on with the exercise without spending time setting up external dependencies. This is why the temporal constraints work better for a short exercise than trying to demonstrate mocks and hexagonal architecture by making people run into a problem of using an external context or a database or a queue.
• Longer exercises start out small and allow participants to build up a model
• They all focus on building habits, not giving participants a list of rules to follow in day to day work

See also David Harvey’s post about this session. Also, see other posts about the GoosGaggle event.

I’ve written against UI test automation several times so far, so I won’t repeat myself. However, many teams I interviewed seem to prefer UI level automation, or think that such level of testing is necessary to prove the required business functionality. Almost all of them have realised six to nine months after starting this effort that the cost of maintaining UI level tests is higher than the benefit they bring. Many have thrown away the tests at that point and effectively lost all the effort they put into them. If you have to do UI test automation (which I’d challenge in the first place), here is how do go about doing it so that the cost of maintenance doesn’t kill you later.

Three levels of UI test automation

A very good idea when designing UI level functional tests is to think about describing the test and the automation at these three levels:

• Business rule/functionality level: what is this test demonstrating or exercising. For example: Free delivery is offered to customers who order two or more books.
• User interface workflow level: what does a user have to do to exercise the functionality through the UI, on a higher activity level. For example, put two books in a shopping cart, enter address details, verify that delivery options include free delivery.
• Technical activity level: what are the technical steps required to exercise the functionality. For example, open the shop homepage, log in with “testuser” and “testpassword”, go to the “/book” page, click on the first image with the “book” CSS class, wait for page to load, click on the “Buy now” link… and so on.

At the point where they figured out that UI testing is not paying off, most teams I interviewed were describing tests at the technical level only (an extreme case of this are recorded test scripts, where even the third level isn’t human readable). Such tests are very brittle, and many of them tend to break with even the smallest change in the UI. The third level is quite verbose as well, so it is often hard to understand what is broken when a test fails. Some teams were describing tests at the workflow level, which was a bit more stable. These tests weren’t bound to a particular layout, but they were bound to user interface implementation. When the page workflow changes, or when the underlying technology changes, such tests break.

Before anyone starts writing an angry comment about the technical level being the only thing that works, I want to say: Yes, we do need the third level. It is where the automation really happens and where the test exercises our web site. But there are serious benefits to not having only the third level.

The stability in acceptance tests comes from the fact that business rules don’t change as much as technical implementations. Technology moves much faster than business. The closer your acceptance tests are to the business rules, the more stable they are. Note that this doesn’t necessarily mean that these tests won’t be executed through the user interface – just that they are defined in a way that is not bound to a particular user interface.

The idea of thinking about these different levels is good because it allows us to write UI-level tests that are easy to understand, efficient to write and relatively inexpensive to maintain. This is because there is a natural hierarchy of concepts on these three levels. Checking that delivery is available for two books involves putting a book in a shopping cart. Putting a book in a shopping cart involves a sequence of technical steps. Entering address details does as well. Breaking things down like that and combining lower level concepts into higher level concepts reduces the cognitive load and promotes reuse.

Easy to understand

From the bottom up, the clarity of the test increases. At the technical activity level, tests are very technical and full of clutter – it’s hard to see the forest for the trees. At the user interface workflow level, tests describe how something is done, which is easier to understand but still has too much detail to efficiently describe several possibilities. At the business rule level, the intention of the test is described in a relatively terse form. We can use that level to effectively communicate all different possibilities in important example cases. It is much more efficient to give another example as “Free delivery is not offered to customers who have one book” than to talk about logging in, putting only a single book in a cart, checking out etc. I’m not even going to mention how much cognitive overload a description of that same thing would require if we were to talk about clicking check-boxes and links.

Efficient to write

From the bottom up, the technical level of tests decreases. At the technical activity level, you need people who understand the design of a system, HTTP calls, DOM and such to write the test. To write tests at the user interface workflow level, you only need to understand the web site workflow. At the business rule level, you need to understand what the business rule is. Given a set of third-level components (eg login, adding a book), testers who are not automation specialists and business users can happily write the definition of second level steps. This allows them to engage more efficiently during development and reduce the automation load on developers.

More importantly, the business rule and the workflow level can be written before the UI is actually there. Tests at these levels can be written before the development starts, and be used as guidelines for development and as acceptance criteria to verify the output.

Relatively inexpensive to maintain

The business rule level isn’t tied to any particular web site design or activity flow, so it remains stable and unchanged during most web user interface changes, be it layout or workflow improvements. The user interface workflow level is tied to the activity workflow, so when the flow for a particular action changes we need to rewrite only that action. The technical level is tied to the layout of the pages, so when the layout changes we need to rewrite or re-record only the implementation of particular second-level steps affected by that (without changing the description of the test at the business or the workflow level).

To continue with the free delivery example from above, if the login form was suddenly changed not to have a button but an image, we only need to re-write the “login” action at the technical level. From my experience, it is the technical level where changes happen most frequently – layout, not the activity workflow. So by breaking up the implementation into this hierarchy, we’re creating several layers of insulation and limiting the propagation of changes. This reduces the cost of maintenance significantly.

Implementing this in practice

There are many good ways to implement this idea in practice. Most test automation tools provide one or two levels of indirection that can be used for this. In fact, this is why I think Cucumber found such a sweet spot for browser based user interface testing. With Cucumber, step definitions implemented in a programming language naturally sit with developers and this is where the technical activity level UI can be described. These step definition can then be reused to create scenarios (user interface workflow level), and scenario outlines can be used to efficiently describe tests at the business rule level.

New SLIM test runner for FitNesse provides similar levels of isolation. The bottom fixture layer sits naturally with the technical activity level. Scenario definitions can be used to describe workflows at the activity level. Scenario tables then present a nice, concise view at the business rule level.

Robot Framework uses “keywords” to describe tests, and allows us to define keywords either directly in code (which becomes the technical level) or by combining existing keywords (which becomes the workflow and business rule level).

The Page Object idea from Selenium and WebDriver is a good start, but stops short of finishing the job. It requires us to encapsulate the technical activity level into higher level “page” functionality. These can then be used to describe business workflows. It lacks the consolidation of workflows into the top business rule level — so make sure to do create this level yourself in the code. (Antony Marcano also raised a valid point that users think about business activities, not page functionality during CITCON Europe 09, so page objects might not be the best way to go anyway).

TextTest works with xUseCase recorders, an interesting twist on this concept that allows you to record the technical level of step definitions without having to program it manually. This might be interesting for thick-client UIs where automation scripts are not as developed as in the web browser space.

With Twist, you can record the technical level and it will create fixture definitions for you. Instead of using that directly in the test, you can use “abstract concepts” to combine steps into workflow activities and then use that for business level testing. Or you can add fixture methods to produce workflow activities in code.

Beware of programming in text

Looking at UI tests at these three levels is I think generally a good practice. Responsibility for automation at the user interface level is something that each team needs to decide depending on their circumstances.

Implementing the workflow level in plain text test scripts (Robot Framework higher level keywords, Twist abstract concepts, SLIM scenario tables) allows business people and testers who aren’t automation specialists to write and maintain them. For some teams, this is a nice benefit because developers can then focus on other things and testers can engage earlier. That does mean, however, that there is no automated refactoring, syntax checking or anything like that at the user interface automation level.

Implementing the workflow level in code enables better integration and reuse, also giving you the possibility of implementing things below the UI when that is easier, without disrupting the higher level descriptions. It does, however, require people with programming knowledge to automate that level.

An interesting approach that one team I interviewed had is to train testers to write code enough to be able to implement the user activity level in code as well. This doesn’t require advanced programming knowledge, and developers are there anyway to help if someone gets stuck.

Things to remember

To avoid shooting yourself in the foot with UI tests, remember these things:

• Think about UI test automation at three levels: business rules, user interface workflow and technical activity
• Even if the user interface workflow automation gets implemented in plain text, make sure to put one level of abstraction above it and describe business rules directly. Don’t describe rules as workflows (unless they genuinely deal with workflow decisions – and even then it’s often good to describe individual decisions as state machines).
• Even if the user interface workflow automation gets implemented in code, make sure to separate technical activities required to fulfil a step into a separate layer. Reuse these step definitions to get stability and easy maintenance later.
• Beware of programming in plain text.

According to Pryce, people often jump into unit testing the business domain core of their context, and with complicated technical infrastructures such as in heterogeneous financial systems the result is that the domain model often doesn’t really fit into how it needs to plug into the infrastructure. Sometimes the resulting design is impossible to integrate at the end.

Pryce recommended an alternative approach when working with complicated infrastructure: start with end-to-end system tests and a very rough model of the core (“only enough of the process that it needs”). Then “jump in and out of the domain level [implementing scenarios] to ensure easy integration”. In this approach, end-to-end technical tests are used to guide the development of whole feature sets, very similar to how high level acceptance tests are used when the risk is in the complexity of the business domain. They are used to guide development at a larger scale, pointing out which details need to be fleshed out by unit tests and helping to avoid the “TDD equivalent of analysis by paralysis”, where teams keep writing unit tests for all possible cases even if they don’t contribute to the design. Jim Shore, in his article Microsoft Gets TDD Completely Wrong, wrote that figuring out what test will best move your code towards completion is the hardest step for beginners. End-to-end system tests can certainly help with this.

Pryce gave several examples where system end-to-end tests pointed to infrastructural solutions and configuration changes as alternatives to developing custom software. With complicated technical infrastructures, some existing components might be able to provide parts of the required functionality. In one case 20000 lines of Java code was replaced with a few XSLT files that transformed messages in a message broker. The system tests ensured that the system still gave the same functionality. He proposed that starting with system end-to-end tests might help point out similar situations in the future where such unnecessary development won’t be done in the first place.

Isolation

This approach isn’t without challenges, though. “In a large system, your system fits into a set of byzantine systems controlled by someone else, chosen for political reasons and hard or impossible to test”, said Pryce. Major challenges to implementing an end-to-end system test efficiently are scaling to handle computing intensive tasks, reliably handling asynchronous operations, concurrency and distribution. Figuring out where exactly are the borders of your sub-system and isolating it from the rest of the world are key decisions for this approach to succeed. Pryce provided a rule of thumb: “If I can change it or replace it, it’s part of my system”.

He said that it is crucial to simplify byzantine APIs and protocols, in order to speed up development but also to provide natural isolation points. This approach is very similar to Anti-corruption layers from Domain Driven Design, with a difference that Pryce advises isolating at the level of the simpler protocol rather than providing an alternative implementation of the simpler API. Such isolation allowed him to keep more code under test (the real implementation of the simpler API) and also to choose whether to run tests against a fake test dependency or a real system. Isolating at the protocol level also allowed him to control the load that automated tests were putting on other real components.

Pryce advised running overnight tests again snapshot releases of external systems to get early warning of release candidate problems. Talking about a project for a financial institution where he applied this approach, he said that they got it to such a level that teams working on legacy components checked build monitors of his team before signing off a release of their systems.

Side-effect: systems easy to support

Because of a large number of moving parts, implementing end-to-end tests reliably requires us to be able to answer the following questions:

• Where are the system edges?
• What is the system doing?
• Has it stopped doing it yet?
• Has it failed?
• Why has it failed?

The same answers are important for supporting the system in production. Because of that, Pryce concluded that development driven by system tests led to a system that is very easy to support. Implementing end-to-end system tests reliably requires us to design and implement modules, provide integration and inspection points to answer these questions. The same mechanisms can be used in production to monitor and support the system. Similar to how a side effect of (unit) TDD is a system that is easy to modify, end-to-end system test driven development leads to a system easy to support.

see other articles about the GOOSGaggle event

Marick started by saying that he was one of the reviewers of the original mock paper, and that he at that point completely misunderstood the idea and thought that the authors were reinventing test stubs. It wasn’t until four months ago, when he read Growing Object Oriented Software, that he really understood what the concept is all about. Marick said that “the point of mocks is illustrated by a photo of surgeon working on a cow. [You, the surgeon, work with a team and when] you explain what you want to do, reach out your hand and magically the tool you want appears in your hand”.

Demonstrating how he uses now mocks, Marick went through the design and implementation of a recent project with an exercise involving attendees pretending to be objects and mocks in four iterations. After that, he pointed out that he really designed the code in the test, rather than in the implementation. “This is just stenography”, said Marick pointing to the implementation.

There isn’t much more that happens in the implementation apart from what was in the test, and mocks allow us to describe the design in tests with less friction (note: this refers to the implementation of the object under test, not its dependent objects). Objects with complex dependencies require us to set up all those complicated dependencies in the test with classic TDD. This increases the clutter in the test. Using mocks allows us to avoid setting all that up in a test and describing the test (and design) more efficiently. Marick said that this benefit surprised him once he started using mocks properly. “Instead of giving the right time slice, mocks allow you to just give back a token representing a time slice,” said Marick, pointing how mocks facilitate outside-in design, “If it looks as if there is going to be some complicated work, I push the work away into a new object”.

“Doing top-down makes it much more likely that I’ll hit my destination”, said Marick, pointing out that there is no rational reason to choose top-down over bottom-up design, but from his experience the style of working with mocks leads to better design faster. “Things are flowing smoothly without a lot of friction.”, said Marick. Analysing the resulting code, Marick said that the objects ended up not doing much. He stressed out that conventional TDD makes him pile more and more code into a class or a method until it gets so big that “shame forces me to notice that these five methods are doing the same thing and pull them out. Things get big, then small, then big again”. “Mocking really encourages me to push”, said Marick, pointing out that with mocks he gets to smaller objects quicker, allowing him to “make decisions at the last responsible moments”.

Image credits:Interplast @ Flickr

Introduction to BDD is an intensive one day workshop which introduces behaviour-driven development to developers, business analysts and testers. The optional programming module is offered in Java or .NET, with Cucumber to automate BDD scenarios.

Hands-on BDD with Cucumber is a three day workshop which immerses the participants into a project driven by Specification by Example and Behaviour-Driven Development. The workshop is adjusted to fit your business domain and particular needs, so that the participants get real-world experience and instant benefits. It is combines the Introduction to BDD, a day of working on a realistic domain example taken from your recent project or a future phase of a project, and a day of programming exercises for test automation developers. During the workshop, we use Cucumber to manage BDD Scenarios. The programming modules can be either in Java or .NET.

These two workshops are offered at a 20% discount for all bookings made before 31st January 2010. For more information on these and my other workshops, check out the training page.

Both workshops are offered as on-site training only (there are no public workshops scheduled at this time). Contact my company, Neuri Ltd for more information and to book a workshop.