And by popular demand, here is the agile testing donut separately:

Julian spoke about alternative testing – or better put alternatives to testing – arguing that time to market is a key competitive advantage for Internet businesses and that exhaustive testing damages that. He cited examples of Facebook, Flickr and Google as companies that do not perform exhaustive regression testing before releases but deal with problems effectively when they appear. “Looking at production data is a much more effective way of discovering problems”, said Harty, adding that “We need to find ways to detect undesirable effects. Do root cause analysis and implement fast, robust mitigation”. He suggested using software equivalents of canaries in a coal mine, such as partial deployments, monitoring server logs and implementing automated checks that detect problems quickly.

Exhaustive regression testing mitigates the risk of functional problems but it isn’t the only way to do so. Several effective techniques for building quality into products from the start, combined with production monitoring and the ability to solve issues quickly, might cost a lot less and de-risk most expected problems cheaper than the time and effort required for regression testing, even if you write automated tests for other purposes.

This reminded me of one of the most puzzling findings from the survey of effective teams I conducted for Specification by Example. The team at uSwitch disable most tests after they pass initially. They use tests to guide development but (most of them) not for regression testing. It sounds counter-intuitive, but it works for them. They still maintain a great level of quality. When I spoke to them in May this year, they couldn’t remember when they caught the last serious issue was found in production. Instead of regresssion checks, they deliver small increments of functionality to increments of production environment and monitor user experience with automated tools. This derisks horrible problems significantly.

In Estimating Software Costs: Bringing Realism to Estimating, Capers Jones estimated that regression testing catches only 23% of the problems. Brian Marick wrote a long ago about a similar finding: 30% problems caught while running regression tests repeatedly. If that is true, slow exhaustive testing might actually cost some businesses more in lost opportunity and slower time to market than the problem that it would prevent. I don’t think that this is a general case, for example in extremely high risk or heavily regulated environments, but for many Internet businesses this might actually be viable.

This of course leaves the problem of Black Swans, completely unexpected and unpredictable issues. (There is a valid argument that exhaustive regression testing can’t prevent such things anyway.) How about de-risking these things as well?

We don’t know what a potential Black Swan could be (or otherwise it wouldn’t be a Black Swan) but we can train the organisation to deal with such problems effectively. Nathalie Roseboom de Vries van Delft presented at Belgium Testing Days about her experiences in (quite bizarre) simulations of disasters. She participates in simulations of floods and re-enactments of airplane and maritime disasters to check how rescue and emergency services deal with such events. Government agencies organise those simulations to maintain readiness and ensure that such events will be dealt with effectively if they occur. These events help to identify issues such as communication problems. For example, one conclusion from a flood simulation was to avoid using acronyms as rescuers from different countries understand them differently. Simulations also give people a chance to experiment with various approaches – for example mixing international rescue teams or using teams of people from the same country to solve particular tasks.

Government agencies aren’t the only ones who need to handle disasters effectively. Businesses could benefit from this too. In the January-February issue of the Harvard Business Review, Farzad Rastegar wrote about his experiences with a Black Swan – when the news about an upcoming recall of his company’s baby pushchairs was leaked to New York Daily News and published one day before the planned press release. His company, Maclaren USA, was preparing for the recall for months with resellers and regulators, but the news leak caught them by surprise. Their mail servers went down, communication likes broke and the damage was hard to control. It took a lot of time, effort and money to clean up the mess, although they spent months preparing. Rastegar concludes that they learned quite a lot about the company as a result of that and decided to change the reselling structure to be able to deal with such problems better in the future. After hearing Natalie Roseboom, I can’t help but think that periodic disaster simulations could help much better.

Maybe all these ideas can be combined. Organisations can deal with smaller issues through production monitoring and “canaries”, at the same time ensuring that they can handle Black Swans effectively through training and simulations. An organisation could simulate injecting a problem and see how the copes with it, both from a technical and a business perspective, identify mistakes and make the processes more effective. The cost of such simulations would be high, but it would significantly derisk issues that are unpredictable and lower the cost of handling such problems. By definition, they happen very rarely, this could be a good strategy to justify dropping regression testing completely and getting much faster and cheaper deliveries.

The workshop left me with the feeling that the sweet-spot for Robot is automation in heterogeneous environments, because of its support for different languages and the host of ready-made libraries for user interface, network, operating systems and API testing (eg ssh, swing, swt, web, selenium, database, mobile, file management, process control). These libraries in particular allow people without much programming experience to efficiently script automated tests that talk to different elements of a system and organise them so that they are relatively easy to manage.

Another area where Robot seems to be better than the competition is managing huge sets of tests/executable specifications. Pekka said that Texas Instruments have a set of 2 million tests executing through Robot. To help teams manage large sets of tests, Robot has taken the idea of test tags much further than any other tool I’ve seen. For example, Robot automatically collects execution statistics and sorts them by tags, presenting a very useful global summary. I haven’t seen this use of tags before but I love it and I hope that other tools will steal this idea soon.

The danger, as with all such systems, is that people will start programming in keywords and introducing overly complex structures that become unmanageable. That is why I hate the Scenario table feature in FitNesse. A nice feature of Robot, that I have not spotted before, is that there is no difference in how (script) keywords are used in higher level keywords from how library definitions (code keywords) are used. This allows developers to push an implementation of a keyword into code once it becomes complex enough, without changing any of the tests that use it. I haven’t done this myself on a commercial project so I can’t guarantee it, but my gut-feel is that this allows people to control the complexity of a keyword-script system effectively and keep it manageable.

The Robot IDE is where the growth will be in the near future. It already has some nice functions that make creation of Robot tests easier and the core team is now working mostly on adding features that will make searching, changing and maintaining easier. This includes extract-keyword refactoring. I don’t really expect this to ever become as fully featured as a programming IDE, but it should address the most common problems with managing layers of specifications built on layers of script keywords.

The test structure is getting a lot less technical. Pekka showed all examples in plain-text format which he now prefers over the HTML format. I always found the Robot HTML test format a bit too technical but the plain text one looks and feels much better. Robot now supports several ways of matching parameters, including a Gherkin-style language.

Whittaker started by saying that software development was like manufacturing 20 years ago, and that the cost of fixing problems after a release was much higher than before software was released. “We don’t do software like this any more”, said Whittaker, adding that “You could be using Google docs and it will update under you and you won’t even know it”. Whittaker concluded that the amount of time it takes to fix a bug before and after shipping for their software is now absolutely the same (note “amount of time”, not cost).

Whittaker suggested looking at a software system as a patient in hospital care. “Testing is like healthcare, an ongoing process”, said he, adding that effective hospital care requires physicians to quickly assess the state of a patient. For that, they use patient charts, which give them a history of disease and treatment, and monitors, which give them visibility over key vital signs such as heart rate.

“I don’t have a clipboard to tell me where are the problems and the life support monitor”, said Whittaker. As a test engineering director, he said that this information was available to him but in several SQL databases which he had to query manually, so he set his team the task of making him run less SQL queries.

As a result, the development teams at Google have introduced several tools which provide the functionality of the patient charts and health monitors. (But have luckily avoided the “get the most expensive machine, in case the administrator comes problem”).

Attributes-Capabilities-Components

The company policy is that anyone can look at any test cases regardless of whether they own the related product or not. They created a tool called Testify that makes statements about the health of their products based on test cases and execution results. It also assists them in providing a historical view of diagnostics.

In order to provide a statement about a software product, they realized that the information needs to be structured and organized. “Doctors have a huge advantage – all their patients look the same”, said Whittaker. To make a meaningful statement about software health, they started listing desired attributes of their products. For example, some of the desired attributes of Chrome OS is that it is stable and secure. “Any time a sales person uses an adverb or an adjective we create an attribute”, said Whittaker. These attributes are then compiled into a tag cloud to provide visibility of relative importance of different attributes.

For each attribute, they list relevant software components that can affect it. This took them about 20 minutes for the Chrome OS. After that, they listed capabilities, starting from verbs that people used to describe the functionality. For example, capabilities of Chrome OS are “connects to internet”, “renders a web page”. It took them about two hours to list 304 capabilities of Chrome OS. Being surprised that there are not more capabilities, they ran the list through a review process and ended up with 314 capabilities in total.

Capabilities, components and attributes “describe our patient”, said Whittaker. According to him, Google is very specific in that it has a very low ratio of testers to developers. “Rich” teams have one tester for four or six developers, the typical ratio is one to fourteen and some teams have one tester for twenty eight developers. He said: “If a developer wants something tested, they have to ask nicely and be prepared for No. The test team needs to understand what to test and you have to make sure that everything you code has to have a set of capabilities attached to a component attached to an attribute”. They assign risk on a capability/attribute matrix and the risk decides where test resources are involved. As a result, developers pay a lot of attention to the quality of their products.

Using “Tours” as test strategy patterns

Similar to the way doctors use classes of treatments (ie vitamin therapy or chemotherapy) to discuss a whole set of actions with their colleagues and patients, Whittaker advised using “tours” to describe test strategy patterns. He gave several examples of “tours” they use in Google, such as the Back-alley tour (visit places where very few people have been before), the All-night tour (run the software continuously overnight in a lab, without shutting down) and the Scottish pub run tour (take over the system completely for testing). These tours are a high level pattern language for both activities and diagnostics in a testing session, without going into the low level details. They allow the team to discuss meta issues and communicate more efficiently.

Heads-up displays

A monitor in a patient room tells visiting doctors everything they need to know about vital signs of a patient. They do not need to exit the room to find out the details. Whittaker said that video games are great examples of how heads-up displays show complex statuses on a screen in a very usable form. “They [players] are not writing SQL queries during a game”, said Whittaker, arguing that testers should not be writing SQL queries to get vital software signs as well. He then showed several examples of systems that they have developed at Google to show vital signs of their systems.

One visualisation screen shows the number of unit tests per module per developer, enabling him to see quickly who is writing unit tests and who is not. As a result of this visualisation, the testers started to reward good developers through a peer bonus system and stimulate them to be better in writing automated tests.

Another screen shows who fixes bugs and who doesn’t. They created a video-game like screen with developers’ avatars at the top half and bug-like creatures at the bottom screen, representing logged bugs. As soon as a defect is assigned to a particular developer, the related bug moves from the bottom part of the screen to the top half and starts attacking the avatar of the related developer. When the bug is fixed, the developer avatar kills the bug. This visualisation allowed them to quickly spot the developers that have lots of assigned open bugs. Putting this “game” on a screen where developers can see it caused the average time a bug stays open in their system to go down significantly.

So is testing like healthcare?

At the end, Whittaker concluded that testing is a “continuous ongoing activity that mirrors the way physicians tour hospital wards”.

My main objection to this talk is using a metaphor for software development that can only be stretched so far. I don’t know why software development always has to be like something else and why people can’t just take it for a practice in its own right. The attitude that software is always sick and in patient care doesn’t really fit well with my experience. I’ve seen quite a few sick systems, but lots of very healthy ones as well – and they both required testing. I was building software for fun, not for money, 20 years ago, so I cannot really talk about what it was like in the industry from experience. However, my understanding of the situation then was that software development was never like manufacturing.

I do however see a lot of value in learning from other fields, including healthcare, and applying relevant lessons learned there to software. Exposing system characteristics and providing more visibility is definitely a good idea.

Read a related post on Improving test practices at Google.

I am evaluating whether to use FitNesse with FitSharp for acceptance testing some web services.

In reading your book and searching online, I could not find a template or recommended approach to doing this. The following is a working idea on the approach. I would like to hear your feedback if there is a better means of doing this!

Template for writing web service acceptance tests:

1. Generate proxy class using the SVCUTIL / WSDL tools in Visual Studio .NET
2. Wrap the proxy in a helper class that exposes properties for the requests/responses along with setting up the endpoint. This class also includes an Execute() method.
3. Compose a fixture class containing an instance of the helper class
4. From FitNesse, initialize the setup of a fixture object with endpoint data and instance variables for web service communication.
5. Execute()
6. Check data in assertions

This very much depends on who the stakeholders are for the tests – who is concerned about them and who is involved in writing them.

FitNesse magic works best on automating shared expectations about what the software should do before you start writing code – examples that you can agree on with your business customers and analysts, because you can automate these examples and still keep them in a human readable form.

You can still get some benefits using fitnesse in a legacy environment if you decide to write tests for existing software but need to have them in a form that is easily readable by business analysts. But in this case just wrapping a web service with a technical autogenerated fixture won’t do the trick – try to create a table format that makes sense for the business instead of basing it on a technical wsdl.

If you don’t need this, and if the only stakeholders for these tests are developers, then you won’t get any benefits from FitNesse over MBUnit and it will give you higher maintenance costs.

If you are testing web services after the fact – after they were written – you will probably get much better return on your investment in test in just using a technical test tool such as NUnit or MBunit.

Related post: How to implement loops in FitNesse test fixtures