I’ve written against UI test automation several times so far, so I won’t repeat myself. However, many teams I interviewed seem to prefer UI level automation, or think that such level of testing is necessary to prove the required business functionality. Almost all of them have realised six to nine months after starting this effort that the cost of maintaining UI level tests is higher than the benefit they bring. Many have thrown away the tests at that point and effectively lost all the effort they put into them. If you have to do UI test automation (which I’d challenge in the first place), here is how do go about doing it so that the cost of maintenance doesn’t kill you later.

Three levels of UI test automation

A very good idea when designing UI level functional tests is to think about describing the test and the automation at these three levels:

• Business rule/functionality level: what is this test demonstrating or exercising. For example: Free delivery is offered to customers who order two or more books.
• User interface workflow level: what does a user have to do to exercise the functionality through the UI, on a higher activity level. For example, put two books in a shopping cart, enter address details, verify that delivery options include free delivery.
• Technical activity level: what are the technical steps required to exercise the functionality. For example, open the shop homepage, log in with “testuser” and “testpassword”, go to the “/book” page, click on the first image with the “book” CSS class, wait for page to load, click on the “Buy now” link… and so on.

At the point where they figured out that UI testing is not paying off, most teams I interviewed were describing tests at the technical level only (an extreme case of this are recorded test scripts, where even the third level isn’t human readable). Such tests are very brittle, and many of them tend to break with even the smallest change in the UI. The third level is quite verbose as well, so it is often hard to understand what is broken when a test fails. Some teams were describing tests at the workflow level, which was a bit more stable. These tests weren’t bound to a particular layout, but they were bound to user interface implementation. When the page workflow changes, or when the underlying technology changes, such tests break.

Before anyone starts writing an angry comment about the technical level being the only thing that works, I want to say: Yes, we do need the third level. It is where the automation really happens and where the test exercises our web site. But there are serious benefits to not having only the third level.

The stability in acceptance tests comes from the fact that business rules don’t change as much as technical implementations. Technology moves much faster than business. The closer your acceptance tests are to the business rules, the more stable they are. Note that this doesn’t necessarily mean that these tests won’t be executed through the user interface – just that they are defined in a way that is not bound to a particular user interface.

The idea of thinking about these different levels is good because it allows us to write UI-level tests that are easy to understand, efficient to write and relatively inexpensive to maintain. This is because there is a natural hierarchy of concepts on these three levels. Checking that delivery is available for two books involves putting a book in a shopping cart. Putting a book in a shopping cart involves a sequence of technical steps. Entering address details does as well. Breaking things down like that and combining lower level concepts into higher level concepts reduces the cognitive load and promotes reuse.

Easy to understand

From the bottom up, the clarity of the test increases. At the technical activity level, tests are very technical and full of clutter – it’s hard to see the forest for the trees. At the user interface workflow level, tests describe how something is done, which is easier to understand but still has too much detail to efficiently describe several possibilities. At the business rule level, the intention of the test is described in a relatively terse form. We can use that level to effectively communicate all different possibilities in important example cases. It is much more efficient to give another example as “Free delivery is not offered to customers who have one book” than to talk about logging in, putting only a single book in a cart, checking out etc. I’m not even going to mention how much cognitive overload a description of that same thing would require if we were to talk about clicking check-boxes and links.

Efficient to write

From the bottom up, the technical level of tests decreases. At the technical activity level, you need people who understand the design of a system, HTTP calls, DOM and such to write the test. To write tests at the user interface workflow level, you only need to understand the web site workflow. At the business rule level, you need to understand what the business rule is. Given a set of third-level components (eg login, adding a book), testers who are not automation specialists and business users can happily write the definition of second level steps. This allows them to engage more efficiently during development and reduce the automation load on developers.

More importantly, the business rule and the workflow level can be written before the UI is actually there. Tests at these levels can be written before the development starts, and be used as guidelines for development and as acceptance criteria to verify the output.

Relatively inexpensive to maintain

The business rule level isn’t tied to any particular web site design or activity flow, so it remains stable and unchanged during most web user interface changes, be it layout or workflow improvements. The user interface workflow level is tied to the activity workflow, so when the flow for a particular action changes we need to rewrite only that action. The technical level is tied to the layout of the pages, so when the layout changes we need to rewrite or re-record only the implementation of particular second-level steps affected by that (without changing the description of the test at the business or the workflow level).

To continue with the free delivery example from above, if the login form was suddenly changed not to have a button but an image, we only need to re-write the “login” action at the technical level. From my experience, it is the technical level where changes happen most frequently – layout, not the activity workflow. So by breaking up the implementation into this hierarchy, we’re creating several layers of insulation and limiting the propagation of changes. This reduces the cost of maintenance significantly.

Implementing this in practice

There are many good ways to implement this idea in practice. Most test automation tools provide one or two levels of indirection that can be used for this. In fact, this is why I think Cucumber found such a sweet spot for browser based user interface testing. With Cucumber, step definitions implemented in a programming language naturally sit with developers and this is where the technical activity level UI can be described. These step definition can then be reused to create scenarios (user interface workflow level), and scenario outlines can be used to efficiently describe tests at the business rule level.

New SLIM test runner for FitNesse provides similar levels of isolation. The bottom fixture layer sits naturally with the technical activity level. Scenario definitions can be used to describe workflows at the activity level. Scenario tables then present a nice, concise view at the business rule level.

Robot Framework uses “keywords” to describe tests, and allows us to define keywords either directly in code (which becomes the technical level) or by combining existing keywords (which becomes the workflow and business rule level).

The Page Object idea from Selenium and WebDriver is a good start, but stops short of finishing the job. It requires us to encapsulate the technical activity level into higher level “page” functionality. These can then be used to describe business workflows. It lacks the consolidation of workflows into the top business rule level — so make sure to do create this level yourself in the code. (Antony Marcano also raised a valid point that users think about business activities, not page functionality during CITCON Europe 09, so page objects might not be the best way to go anyway).

TextTest works with xUseCase recorders, an interesting twist on this concept that allows you to record the technical level of step definitions without having to program it manually. This might be interesting for thick-client UIs where automation scripts are not as developed as in the web browser space.

With Twist, you can record the technical level and it will create fixture definitions for you. Instead of using that directly in the test, you can use “abstract concepts” to combine steps into workflow activities and then use that for business level testing. Or you can add fixture methods to produce workflow activities in code.

Beware of programming in text

Looking at UI tests at these three levels is I think generally a good practice. Responsibility for automation at the user interface level is something that each team needs to decide depending on their circumstances.

Implementing the workflow level in plain text test scripts (Robot Framework higher level keywords, Twist abstract concepts, SLIM scenario tables) allows business people and testers who aren’t automation specialists to write and maintain them. For some teams, this is a nice benefit because developers can then focus on other things and testers can engage earlier. That does mean, however, that there is no automated refactoring, syntax checking or anything like that at the user interface automation level.

Implementing the workflow level in code enables better integration and reuse, also giving you the possibility of implementing things below the UI when that is easier, without disrupting the higher level descriptions. It does, however, require people with programming knowledge to automate that level.

An interesting approach that one team I interviewed had is to train testers to write code enough to be able to implement the user activity level in code as well. This doesn’t require advanced programming knowledge, and developers are there anyway to help if someone gets stuck.

Things to remember

To avoid shooting yourself in the foot with UI tests, remember these things:

• Think about UI test automation at three levels: business rules, user interface workflow and technical activity
• Even if the user interface workflow automation gets implemented in plain text, make sure to put one level of abstraction above it and describe business rules directly. Don’t describe rules as workflows (unless they genuinely deal with workflow decisions – and even then it’s often good to describe individual decisions as state machines).
• Even if the user interface workflow automation gets implemented in code, make sure to separate technical activities required to fulfil a step into a separate layer. Reuse these step definitions to get stability and easy maintenance later.
• Beware of programming in plain text.

One of the biggest issues with UI tests described at the task level (such as click there, type this etc) is that they are very brittle and hard to understand. Too many details in technical steps make it hard to see the big picture, so tests work nicely as a regression check but once something fails it is hard to figure out what exactly went wrong. They are also bound to a particular UI design, so when the design changes, lots of tests break. Selenium works on the technical task level and tests written for Selenium frequently fall into this black whole. Once the design is changed, you might as well do everything again from scratch.

Instead of writing tests at the technical task level, the Page Object pattern suggests creating one more level of indirection and describing business functionality of pages with objects (eg search for customers, submit registration form) and then implementing individual methods of these objects with Selenium. Tests are then written with the page object methods, which make them much more understandable and easier to maintain. Once the UI changes, we can reimplement only the relevant methods of Page Objects making the effects of changes relatively contained. Tests themselves don’t need to change. Similar ideas are implemented by xUseCase recorders and Domain-Specific Testing Languages in Selenium (see Mickey Phoenix’s Agile 2008 presentation).

Marcano said that page objects focus on the wrong thing. Taking ideas from interaction design, he suggested that it is natural to think about functionality on three levels — goals, activities and tasks:

1. Goal: Buy a book
2. Activities: Add the following items to the cart,…
3. Tasks: click search, type in book name, click book image, click Add to cart…

When thinking about web site functionality, people don’t think about pages but about activities that span multiple pages. Focusing on goal oriented cross-page activities instead of page oriented functionality makes tests more natural, easier to understand and maintain. Instead of Page Objects, Marcano suggested implementing activities with Selenium and then composing tests from activities. One more significant benefit of this approach is that activities aren’t baked into the user interface, unlike objects representing web pages, so the same test definitions can be kept for testing below the UI with different task implementations.

Anand focused on the process, and key take-away ideas from his presentation for me are:

• Four-column scrum wall: they added a “done-done” step after “done”, making it very explicit that a task needs to get verified after it is thought to be finished. Testers owned this phase which gave them the responsibility to see it through, not necessarily perform it themselves. For tasks that are not easily verifiable through the user interface or things such as usability or design where testers were not qualified to assert completeness, they were still expected to chase someone else to verify that the task is done. This could involve a code review, another developer or a designer confirming that something is OK etc.
• Requirements meetings: Product owner, technical owner and a tester met before a story went into development to discuss what the acceptance criteria is. This made at least three people capable of answering questions from developers and it cut down the need to document stuff. It also ensured that testers knew what they should expect and test on the end. This idea is going along the same lines as specification workshops but seems to be focused more on fleshing out key ideas rather than getting to the full specifications.
• Two-step continuous integration: Their functional test suite took a few hours to run (as it was running through the UI), and running this as part of the build even in continuous integration would make the feedback too slow. On the other hand, these tests had to be executed. Anand suggested doing continuous integration in two steps. First build and run basic (quick) tests. When that succeeds, run the entire test suite. I would typically investigate whether these two hours could be cut down or automated at a lower level to make it quicker, but this requires development resources so Anand’s solution might be very interesting to people who don’t have the option or knowledge to automate below the UI.
• Using named UI elements in Selenium scripts: to make UI tests easier to maintain and understand, they used macros to encapsulate locators for UI elements and gave them meaningful names such as “register button”. When the UI changes, they only update a macro, without the need to change tests. This also enabled them to write tests before the web UI was actually developed, and put in values for locators into macros later. This is again an idea similar to page objects but implemented completely from a tester’s perspective without the need to involve developers.
• Web testing below the browser – they used a tool of which I haven’t heard before, called Twill. Twill automates web testing by executing HTTP requests but not through the browser, making it easier and faster to automate tests than with Selenium. It does not execute javascript so it cannot be applied to complicated browser-oriented tasks, but it is very useful for form submission, link parsing etc. With Python, they built a DSL for testing web pages using Twill. It seems like it could do a very good job for a large number of web tests that don’t involve JavaScript.

Nathan started with a great cartoon introduction to acceptance testing (he promised to upload the slides to nathanbain.com) and then went on to talk about what he would have liked to have known when he was starting with agile testing. Some of the key take-away points are:

• The rules of agile testing are still being written, and lots of information is only available online on blogs and mailing lists – testers need to read and research more. Nathan also recommended Agile Testing by Lisa Crispin and Janet Gregory and my Bridging the Communication Gap as good sources of information on the current state of agile testing.
• Although testers might find themselves in uncharted territories on an agile team, the team expects of them to be the testing experts so they have to find ways to help the team rather than waiting to be told what to do. Testers should be proactive and help define their role by looking at the ways they can improve process and product.
• Promote proactive testing and defining acceptance criteria as tests in the team to raise awareness of what testing is about and get support and help from developers and business. Nathan organised workshops and presentations on acceptance testing to help other members of his team understand how he can help them better.
• Getting more involved in the process is the key to success. Nathan suggested learning about the domain, working closely with developers, going to meetings even if you’re not invited and making sure that you are on all the mailing lists. Nathan warned that people who are outside the loop get left behind, and by being more involved testers can help the team more but also show the others their value better
• Instead of writing down bug reports, discuss them – sometimes the bug turns out not to be a bug at all and the discussion helps to make discovering a real bug constructive rather than an act that conflicts with development. This discussion helps the morale on the team.
• Recruiting casual testers elsewhere – try to get project managers, product owners, developers and anyone else involved with testing. Prepare test scripts as simple check-lists that they can follow. Describe manual tests in plain English so that you can get more people involved.
• Using acceptance tests as a good start for exploration – instead of specifying everything in detail, specify the core as acceptance tests and then use them as a start for exploratory testing, sort of a reminder of where to begin and what to look for.

Altogether a very enjoyable evening. I really liked this and I think that we should have more experience reports like these. The event was filmed and I strongly recommend watching the videos (they should be available in a few days on the Skills Matter agile testing page (subscribe to my rss feed to get notified when they become available).

The next agile testing evening at Skills Matter will be on the 18th of August. David Evans and Mike Scott from SQS-UK will talk about Testify, a project that helps kick-start Test-Driven Development in .NET by installing and setting up all the basic tools that people typically need and creating a basic project structure that makes TDD easier. The event is free as usual, but up-front registration is required for capacity planning. You can find more information on this event and register at skillsmatter.com.

Opensource .NET tools crash course

The first will be a three-day crash course on tools and practices aimed at .NET developers that want to learn about Alt.NET tools and Java developers that are migrating to .NET and looking for good equivalents to the tools that they are used to working with. The course gives an overview of the most popular opensource .net tools and introduces modern development practices that these tools promote, such as test driven development, continuous integration, dependency injection, object-relational mapping and web development using the model-view-controller pattern.

Learn how to:

• Implement TDD in .NET using NUnit, MBUnit, Rhino Mocks and FitNesse
• Utilise Aspect oriented programming and Dependency Injection using Castle Windsor
• Efficiently build Web applications using the MVC pattern in Monorail and utilising Monorail and Script# for Ajax and test them using Selenium Remote Console.
• Manage persistence easily using ORM tools such as ActiveRecord and NHibernate
• Introduce continuous integration in your projects using CruiseControl.NET and CI Factory

See the full programme.

Agile Web Development with the Castle Framework

The second one is a two-day course on Agile Web Development using the Castle project, teaching the basics of the Castle Framework and helping people develop a solid understanding of its benefits. Over the course of the two days, attendees will create a simple but complete web application using agile Web development practices such as Inversion of Control, Dependency Injection, Aspect Oriented Programming, Object/Relational Mapping and applying the Model-View-Controller pattern.

Learn how to

• Apply agile web development practices like MVC and dependency injection
• Use ActiveRecord to manage the object-relational mapping and the database layer
• Use the Monorail MVC engine to create web applications that are easy to maintain and test
• Explain the basics of Monorail views, layouts, rescues
• Use the NVelocity view engine to build web UIs for Monorail
• Apply Windsor Microkernel to configure and wire application components
• Unit test the data access layer with Castle
• Unit test web controllers
• Describe how Castle components come together to help us develop web applications easier
• Explain why this approach is much more effective than ASP.NET
• Apply best practices, common pitfalls, and tips and tricks for Castle Web development

See the full programme.