I am evaluating whether to use FitNesse with FitSharp for acceptance testing some web services.

In reading your book and searching online, I could not find a template or recommended approach to doing this. The following is a working idea on the approach. I would like to hear your feedback if there is a better means of doing this!

Template for writing web service acceptance tests:

1. Generate proxy class using the SVCUTIL / WSDL tools in Visual Studio .NET
2. Wrap the proxy in a helper class that exposes properties for the requests/responses along with setting up the endpoint. This class also includes an Execute() method.
3. Compose a fixture class containing an instance of the helper class
4. From FitNesse, initialize the setup of a fixture object with endpoint data and instance variables for web service communication.
5. Execute()
6. Check data in assertions

This very much depends on who the stakeholders are for the tests – who is concerned about them and who is involved in writing them.

FitNesse magic works best on automating shared expectations about what the software should do before you start writing code – examples that you can agree on with your business customers and analysts, because you can automate these examples and still keep them in a human readable form.

You can still get some benefits using fitnesse in a legacy environment if you decide to write tests for existing software but need to have them in a form that is easily readable by business analysts. But in this case just wrapping a web service with a technical autogenerated fixture won’t do the trick – try to create a table format that makes sense for the business instead of basing it on a technical wsdl.

If you don’t need this, and if the only stakeholders for these tests are developers, then you won’t get any benefits from FitNesse over MBUnit and it will give you higher maintenance costs.

If you are testing web services after the fact – after they were written – you will probably get much better return on your investment in test in just using a technical test tool such as NUnit or MBunit.

Related post: How to implement loops in FitNesse test fixtures

I’ve written against UI test automation several times so far, so I won’t repeat myself. However, many teams I interviewed seem to prefer UI level automation, or think that such level of testing is necessary to prove the required business functionality. Almost all of them have realised six to nine months after starting this effort that the cost of maintaining UI level tests is higher than the benefit they bring. Many have thrown away the tests at that point and effectively lost all the effort they put into them. If you have to do UI test automation (which I’d challenge in the first place), here is how do go about doing it so that the cost of maintenance doesn’t kill you later.

Three levels of UI test automation

A very good idea when designing UI level functional tests is to think about describing the test and the automation at these three levels:

• Business rule/functionality level: what is this test demonstrating or exercising. For example: Free delivery is offered to customers who order two or more books.
• User interface workflow level: what does a user have to do to exercise the functionality through the UI, on a higher activity level. For example, put two books in a shopping cart, enter address details, verify that delivery options include free delivery.
• Technical activity level: what are the technical steps required to exercise the functionality. For example, open the shop homepage, log in with “testuser” and “testpassword”, go to the “/book” page, click on the first image with the “book” CSS class, wait for page to load, click on the “Buy now” link… and so on.

At the point where they figured out that UI testing is not paying off, most teams I interviewed were describing tests at the technical level only (an extreme case of this are recorded test scripts, where even the third level isn’t human readable). Such tests are very brittle, and many of them tend to break with even the smallest change in the UI. The third level is quite verbose as well, so it is often hard to understand what is broken when a test fails. Some teams were describing tests at the workflow level, which was a bit more stable. These tests weren’t bound to a particular layout, but they were bound to user interface implementation. When the page workflow changes, or when the underlying technology changes, such tests break.

Before anyone starts writing an angry comment about the technical level being the only thing that works, I want to say: Yes, we do need the third level. It is where the automation really happens and where the test exercises our web site. But there are serious benefits to not having only the third level.

The stability in acceptance tests comes from the fact that business rules don’t change as much as technical implementations. Technology moves much faster than business. The closer your acceptance tests are to the business rules, the more stable they are. Note that this doesn’t necessarily mean that these tests won’t be executed through the user interface – just that they are defined in a way that is not bound to a particular user interface.

The idea of thinking about these different levels is good because it allows us to write UI-level tests that are easy to understand, efficient to write and relatively inexpensive to maintain. This is because there is a natural hierarchy of concepts on these three levels. Checking that delivery is available for two books involves putting a book in a shopping cart. Putting a book in a shopping cart involves a sequence of technical steps. Entering address details does as well. Breaking things down like that and combining lower level concepts into higher level concepts reduces the cognitive load and promotes reuse.

Easy to understand

From the bottom up, the clarity of the test increases. At the technical activity level, tests are very technical and full of clutter – it’s hard to see the forest for the trees. At the user interface workflow level, tests describe how something is done, which is easier to understand but still has too much detail to efficiently describe several possibilities. At the business rule level, the intention of the test is described in a relatively terse form. We can use that level to effectively communicate all different possibilities in important example cases. It is much more efficient to give another example as “Free delivery is not offered to customers who have one book” than to talk about logging in, putting only a single book in a cart, checking out etc. I’m not even going to mention how much cognitive overload a description of that same thing would require if we were to talk about clicking check-boxes and links.

Efficient to write

From the bottom up, the technical level of tests decreases. At the technical activity level, you need people who understand the design of a system, HTTP calls, DOM and such to write the test. To write tests at the user interface workflow level, you only need to understand the web site workflow. At the business rule level, you need to understand what the business rule is. Given a set of third-level components (eg login, adding a book), testers who are not automation specialists and business users can happily write the definition of second level steps. This allows them to engage more efficiently during development and reduce the automation load on developers.

More importantly, the business rule and the workflow level can be written before the UI is actually there. Tests at these levels can be written before the development starts, and be used as guidelines for development and as acceptance criteria to verify the output.

Relatively inexpensive to maintain

The business rule level isn’t tied to any particular web site design or activity flow, so it remains stable and unchanged during most web user interface changes, be it layout or workflow improvements. The user interface workflow level is tied to the activity workflow, so when the flow for a particular action changes we need to rewrite only that action. The technical level is tied to the layout of the pages, so when the layout changes we need to rewrite or re-record only the implementation of particular second-level steps affected by that (without changing the description of the test at the business or the workflow level).

To continue with the free delivery example from above, if the login form was suddenly changed not to have a button but an image, we only need to re-write the “login” action at the technical level. From my experience, it is the technical level where changes happen most frequently – layout, not the activity workflow. So by breaking up the implementation into this hierarchy, we’re creating several layers of insulation and limiting the propagation of changes. This reduces the cost of maintenance significantly.

Implementing this in practice

There are many good ways to implement this idea in practice. Most test automation tools provide one or two levels of indirection that can be used for this. In fact, this is why I think Cucumber found such a sweet spot for browser based user interface testing. With Cucumber, step definitions implemented in a programming language naturally sit with developers and this is where the technical activity level UI can be described. These step definition can then be reused to create scenarios (user interface workflow level), and scenario outlines can be used to efficiently describe tests at the business rule level.

New SLIM test runner for FitNesse provides similar levels of isolation. The bottom fixture layer sits naturally with the technical activity level. Scenario definitions can be used to describe workflows at the activity level. Scenario tables then present a nice, concise view at the business rule level.

Robot Framework uses “keywords” to describe tests, and allows us to define keywords either directly in code (which becomes the technical level) or by combining existing keywords (which becomes the workflow and business rule level).

The Page Object idea from Selenium and WebDriver is a good start, but stops short of finishing the job. It requires us to encapsulate the technical activity level into higher level “page” functionality. These can then be used to describe business workflows. It lacks the consolidation of workflows into the top business rule level — so make sure to do create this level yourself in the code. (Antony Marcano also raised a valid point that users think about business activities, not page functionality during CITCON Europe 09, so page objects might not be the best way to go anyway).

TextTest works with xUseCase recorders, an interesting twist on this concept that allows you to record the technical level of step definitions without having to program it manually. This might be interesting for thick-client UIs where automation scripts are not as developed as in the web browser space.

With Twist, you can record the technical level and it will create fixture definitions for you. Instead of using that directly in the test, you can use “abstract concepts” to combine steps into workflow activities and then use that for business level testing. Or you can add fixture methods to produce workflow activities in code.

Beware of programming in text

Looking at UI tests at these three levels is I think generally a good practice. Responsibility for automation at the user interface level is something that each team needs to decide depending on their circumstances.

Implementing the workflow level in plain text test scripts (Robot Framework higher level keywords, Twist abstract concepts, SLIM scenario tables) allows business people and testers who aren’t automation specialists to write and maintain them. For some teams, this is a nice benefit because developers can then focus on other things and testers can engage earlier. That does mean, however, that there is no automated refactoring, syntax checking or anything like that at the user interface automation level.

Implementing the workflow level in code enables better integration and reuse, also giving you the possibility of implementing things below the UI when that is easier, without disrupting the higher level descriptions. It does, however, require people with programming knowledge to automate that level.

An interesting approach that one team I interviewed had is to train testers to write code enough to be able to implement the user activity level in code as well. This doesn’t require advanced programming knowledge, and developers are there anyway to help if someone gets stuck.

Things to remember

To avoid shooting yourself in the foot with UI tests, remember these things:

• Think about UI test automation at three levels: business rules, user interface workflow and technical activity
• Even if the user interface workflow automation gets implemented in plain text, make sure to put one level of abstraction above it and describe business rules directly. Don’t describe rules as workflows (unless they genuinely deal with workflow decisions – and even then it’s often good to describe individual decisions as state machines).
• Even if the user interface workflow automation gets implemented in code, make sure to separate technical activities required to fulfil a step into a separate layer. Reuse these step definitions to get stability and easy maintenance later.
• Beware of programming in plain text.

Bach compared the situation in testing today to early days of software development where compiling and writing code were all called programming, then compiling got separated and renamed to reflect the fact that it is completely done with a machine. Checking can be automated to a great degree and delegated to the machine, said Bach, offering an example of classic TDD unit tests. Testing as a concious process needs to be done by a trained specialist. This allows us to put more emphasis on exploratory testing, which Elisabeth Hendrickson listed as one of the key seven practices for successful agile testing.

On a similar note, Mary Poppendieck talked about how testing is a learning activity at the Agile Testing Days conference in Berlin, contrasting it to checking the correctness using known expectations.

See other articles from the Oredev conference.

What was really interesting for me in this report is that the change was driven by the demand for better quality. Hemal joined uSwitch as a QTP automation specialist a year ago, when the process in place had quite a few agile elements. Developers were doing TDD, continuous integration was in place, they were doing three weeks sprints and had a Scrum wall. Developers worked in small teams with testers on the team and “never more than two or three seats away from the business”. Two days at the end of every iteration were left for cleanup and testing. But they still had lots of quality problems, which led to a company-wide effort to focus on quality. Out of that, little by little, they fundamentally changed the development process and how they think about software development.

Fat-er wall

Kuntawala said that although developers and testers were working closely together, there was still an invisible wall between them and developers. Developers would develop and testers would test – throwing features over that invisible wall to each-other. As there was one tester to three developers, testers quickly became a bottleneck. “QA and development were ping-ponging each-other. When the time is up, deadlines have to be met and the code gets shipped regardless of quality”, said Kuntawala. This phased approach, albeit done in short iterations, still did not allow them to get the quality they expected. So they started a company-wide initiative to focus on quality and get everyone to think about improving quality.

The first thing that they did is remove the testing bottleneck and get rid of the imaginary barrier between developers and testers (Kuntawala called it the Fat-er wall, anagram of waterfall). Developers were asked to start thinking about testing and they were educated on how to test and learned the theory behind testing. Likewise, testers got educated about development and started to develop and look at the system underneath the user interface. “You get a better perspective of what needs to be tested in the right place”, said Kuntawala. This all resulted in code that was more testable and much better tested. Instead of a massive number of QTP tests, which were poorly documented and used only by testers due to licensing costs, they switched to Selenium for UI testing.

The tester role was dropped completely. Instead of talking about testing, they just called everything development. Testers got their job roles changed to reflect this. One of the developers in the audience said that dropping the testing phase made people work much more carefully because they knew that “nobody is going to look at this after they commit and it will go to production”.

Sorting out deployment

At this point, developers were really releasing to the source code control system. Once the code was in the release branch, they considered their work done. The operations team would take the code from there and try to release it to production. This process was error-prone and it was their next area of focus. The operations team already had some tests to verify the deployment but they were unreliable (often timing out). After doing a few deployments together with the operations team, the developers started writing better tests to verify post-deployment status of the application. These tests are written with Selenium and operations team used them to quickly verify newly deployed code. Developers also make sure that they pass them before handing over to operations. Resolving deployment issues and timeouts on tests and working together to investigate these things resulted in much shorter time to run tests and being able to verify deployments better. It’s also worth noting that they deploy to half of the server farm, verify that it is working correctly, and then deploy to the other half. This allows them to quickly roll back and deploy without downtime.

Moving from tasks to stories

The next part of the process they optimised is being able to deliver smaller chunks of software. They previously used technical tasks to plan and divide work. Technical tasks made it hard to work out a specific acceptance criteria for each task. This also caused delays in deployment. “Tasks depend on each other so developers were reluctant to deploy it until everything is done”, said Kuntawala. So they moved to working on user stories instead of tasks. Stories reflect user value, they are individually deployable and it is easy to get a good definition of when a story is done. As part of the change, they started using acceptance tests as a guide for development and introduced specification workshops to get better involvement from the business side. In order to help people write better acceptance tests, they developed cheat-sheets, taking ideas from Jerry Weinberg and James Whittaker (Does it do what we want it to do? does it do what we don’t want it to do? think about where bugs can lurk: input, state, environment and data). Selenium tests were replaced with Cucumber and WatiR.

Avoiding mass sign-off and testing

Once the deliverables were broken down into stories and it was possible to ship them out sooner, the cycle at the end of the iteration where features get signed-off before a release became a bottleneck and was seen as an unnecessary delay. Instead of running the tests at the end, they started running tests immediately. Instead of one big demonstration at the end, they started demonstrating new features to the business and getting them signed off as soon as they were done. Finally, they started deploying these things on demand. To help with this, they started continuous monitoring on production. They chose Tealeaf for user experience monitoring, such as tracking error rates and usage of new features. This added visibility provides a safety-net against unnoticed deployment problems.

Restructuring the scrum wall

After all this, they restructured the scrum wall to reflect their new process, moving in the Kanban direction. It now has four columns: Ready, In Progress, Inventory and Done. A story becomes ready when it is added into the release plan. It is In Progress during development (including testing) and goes to Inventory when it is ready for deployment. It goes to Done once it is in production, making money. What’s really interesting about this is a limit of three stories in Inventory – in order to move things there, you first have to deploy what is already in the Inventory, reducing the difference between development and production systems. The limit on Done is 9, meaning that once 9 things are deployed, they do the retrospective. Instead of time-based iterations, they effectively have work-based iterations.

The result

The expected turnaround time for a feature a year ago was six to nine weeks. Today, the average turnaround time from requirement to production is just four days. And things are coming out without problems – apparently there hasn’t been a serious issue in production for over six months. One of the development managers from uSwitch was in the audience and said that “quality has increased substantially and our conversion rates have grown”. “We now see work hit production a lot sooner. It might seem we’re working faster but we’re doing things that are smaller and smaller”, commented one of Kuntawala’s colleagues during the discussion.

As some interesting quotes from the team, Kuntawala said that a colleague from the operations team started asking “is there a better way to test this”, that a business colleague asked “how are we going to test this” and that the business is telling them “we can’t keep up”. The company no longer thinks about development in terms of projects and iterations, but in terms of business value.

A very interesting observation during the discussion after Kuntawala’s presentation was that these changes were not driven by a big management decision. They were all implemented as small gradual improvements. Small changes, continuous problem solving and improvement made the transition smooth. “When we got comfortable, we started thinking about what’s the next thing we can do”, said Kuntawala, suggesting that the next things that they will look into will be fully automated deployments and better integration of development teams and the business. He concluded that in twelve months time the process will probably look a lot different. I look forward to hearing an update on this next year.

Testify is a wizard for creating new, portable projects. It sets up the project structure and installs the opensource testing toolset, a stack of “best of breed” tools needed for TDD. It also creates a trivial core project, a trivial UI and a set of trivial unit and acceptance tests that serve as examples for your tests, along with batch files to build the project and run all tests in the suite. The build files will do a full build, run all the unit and acceptance tests, executes static analysis, coverage report and so on.

David Evans talked about Testify earlier this year in London, prompting lots of questions about opensourcing it. As of today, testify is available under an opensource license from Google code repository.

see other news from the Agile Testing Days conference.