One of the biggest issues with UI tests described at the task level (such as click there, type this etc) is that they are very brittle and hard to understand. Too many details in technical steps make it hard to see the big picture, so tests work nicely as a regression check but once something fails it is hard to figure out what exactly went wrong. They are also bound to a particular UI design, so when the design changes, lots of tests break. Selenium works on the technical task level and tests written for Selenium frequently fall into this black whole. Once the design is changed, you might as well do everything again from scratch.

Instead of writing tests at the technical task level, the Page Object pattern suggests creating one more level of indirection and describing business functionality of pages with objects (eg search for customers, submit registration form) and then implementing individual methods of these objects with Selenium. Tests are then written with the page object methods, which make them much more understandable and easier to maintain. Once the UI changes, we can reimplement only the relevant methods of Page Objects making the effects of changes relatively contained. Tests themselves don’t need to change. Similar ideas are implemented by xUseCase recorders and Domain-Specific Testing Languages in Selenium (see Mickey Phoenix’s Agile 2008 presentation).

Marcano said that page objects focus on the wrong thing. Taking ideas from interaction design, he suggested that it is natural to think about functionality on three levels — goals, activities and tasks:

1. Goal: Buy a book
2. Activities: Add the following items to the cart,…
3. Tasks: click search, type in book name, click book image, click Add to cart…

When thinking about web site functionality, people don’t think about pages but about activities that span multiple pages. Focusing on goal oriented cross-page activities instead of page oriented functionality makes tests more natural, easier to understand and maintain. Instead of Page Objects, Marcano suggested implementing activities with Selenium and then composing tests from activities. One more significant benefit of this approach is that activities aren’t baked into the user interface, unlike objects representing web pages, so the same test definitions can be kept for testing below the UI with different task implementations.

Antony Marcano talked about this, not as part of the official programme but over beers which is another example of how informal chatting at conferences is often more valuable than the schedule. He mentioned how it is common for games to automatically save the complete state at checkpoints. This allows us to easily go back to the last saved position and continue playing if our character gets killed for whatever reason. So if something really bad happens after 10 hours of playing, you don’t have to do it all over again.

Functionality similar to that could help immensely with exploratory testing in software. When we find an issue, we could easily go back to the last checkpoint and reproduce it and also use that same saved game later to check if the problem was fixed. This could also be used if a customer encounters a problem in production to make bug reports a lot more complete and reliable.

Implementing this feature is by no means a trivial task and might be nearly impossible to retrofit into existing systems. The reason why gaming companies can do it is that they consider saving as an integral feature of their systems from start. I have yet to try this but my gut feel is that if a system is built with this in mind, the actual overhead of doing it is minimal. Modern games have states as complicated as lots of enterprise systems, with tons of actors, their states and event information at any time. Yet they manage to load it up in a few seconds and continue from the saved state. By designing for that up front, ensuring that all actors can be persisted and loaded and that all state is easily extractable, we should be able to do the same in enterprise software as well.

1. No collaboration — as a specification of a system that acts as the shared definition of done, acceptance tests need to be collaboratively produced and agreed upon. Business people, developers and testers should all provide input, each from their own area of expertise. The discussion that happens around the specification is actually more valuable than the tests themselves, as people build a shared understanding of the domain and the problem at hand. If this discussion doesn’t happen, then teams still feel the effects of the telephone game. Symptoms of this problem are developers writing acceptance tests themselves, testers being expected to handle everything about acceptance testing and business dictating tests without any feedback from developers or testers. The solution for this are collaborative specifications and specification workshops.
2. Focusing on how, not on what — business specifications should be clear and understandable and allow the team to implement the best possible solution without prescribing the design or the implementation. Too often tests describe how something should be implemented rather than what should be implemented. Tests such as these are too low level, hard to understand and always turn out to be a pain to maintain long-term. Instead of facilitating system change they inhibit it as people will be reluctant to introduce changes that break a lot of tests. Symptoms of this problem are action-oriented acceptance tests, lots of workflows or duplication in tests and tests that are very technical. The solutions for this are communicating intent and distilling the specification.
3. Tests unusable as live documentation — one of the greatest benefits of agile acceptance testing is that we are gradually building a human readable specification of the system. This specification is ideally automated to a great degree so that we can be confident that it is in sync with the code. This solves the problem of the running code being the only thing you can really trust about what the system does. Correct, easily understandable and accessible documentation is crucial for future change. If tests don’t serve as a live documentation that those benefits are lost. Symptoms of this problem are very technical tests, long tests, hard to understand tests, tests that are poorly organised so that you can’t quickly find relevant tests for a particular piece of functionality. Solutions for this is to ensure that tests can be used as a live documentation, reorganising them and updating when the ubiquitous language changes.
4. Expecting acceptance tests to be a full regression suite — acceptance tests are primarily a specification of what the system should do and should deal with exemplars that represents whole sets of cases. Specifying too much and covering all possible edge cases will make them hard to understand and might introduce an effect similar to paralysis by analysis. So acceptance tests cannot provide a full regression test suite and additional tests are required (including exploratory testing). Symptoms of this problem are tests that are very long and verify a large number of similar cases. Note that this doesn’t mean that regression tests can’t be written and automated using the same tools as acceptance tests.
5. Focusing on tools — some teams focus too much on tools and features of particular tools, disregarding things that do not naturally fit into their chosen toolset. This makes the specification vague in parts that aren’t covered by a particular tool. Instead of that, teams should be focusing on the business specifications and then choosing the right tool for the job. In particular, tools should not play an important part in the specification workshop. Symptoms of this problem are ignoring the UI (as UI tests are hard to automate) and wasting time on converting specifications in a format suitable for a particular tool during the specification workshops (which breaks the flow of the discussion and wastes valuable time).
6. Not considering acceptance testing as value added activity — agile acceptance testing is not about QA, but more about specifying and agreeing on what needs to be done. Teams that consider it part of QA often delegate the responsibility to it to junior programmers and testers, which effectively means that junior team members will be writing the specification (which is wrong on so many levels that I don’t know where to start). The solutions for this are specification workshops.
7. “Test code” not maintained with love — tests and code to automate them are often considered as less important than production code, so given to less capable developers and testers to implement and maintain. This leads to huge maintenance problems later and can cause teams to completely abandon full suites of tests. Acceptance tests are the specification of a system so they are crucial to the success of a project. They also need to be maintained and kept in sync with the live project language and design in order to serve as a live documentation.
8. Objectives of team members not aligned — if the job of business analysts is only to deliver the specifications, the job of developers only to ship the system and the job of testers to ensure quality, then people won’t pull in the same direction or invest time in producing good acceptance tests. This is a management issue but failures with acceptance testing might make it more visible.
9. No management buy-in — successful acceptance testing requires an active participation of the business and with no management buy-in the benefits will simply not be there (also related to #1).
10. Underestimating the skill required to do this well — introducing acceptance testing is a tall order and often means changing the way teams are organised and approach specifications. This requires a lot of time and investment in skill-up, mastering tools, facilitating workshops and dealing with resistance to change. Underestimating the effort required to do this properly might cause teams to think that they failed and get disappointed too early.

Xavier Warzee filmed the session so watch the videos if you are interested to see more of this discussion (scroll down on the page, it’s the last batch of videos). You can also check out two more write-ups by Nicolas Martignole and Xavier Bourguignon (both in French) and session notes from the CITCON wiki.

If applying acceptance testing and BDD effectively is something that you want to learn more about, you might want to register for the Writing Software that Matters workshop (Dec 4th, central London). I also run on-site workshops that help teams get started and implement agile acceptance testing.

Image credits: Cirilo Wortel

According to Freeman, the original Mock object idea came out of the Roles, Responsibilities and Collaborations object design school (Rebecca Wirfs-Brock). When applying TDD to design and develop such objects, we need to specify and test responsibilities of objects, and externally visible object state isn’t part of that. Verifying that an object matches its responsibilities requires that we “fire an event and see that the appropriate messages have been sent to its collaborators”. According to Freeman, interfaces aren’t enough to specify objects because “an interface tells you what methods you can call, not when and how to call them. it doesn’t specify the contract – this is where mocks come in.”

Freeman advises using mocks as a design tool, not primarily to isolate dependencies: “To prove that this works we need a thingy but we don’t have that thingy yet, so we define it with a mock.Work your way through the slice of a system until you get to something external”. Instead of mocking out external dependencies, Freeman suggests that “you should only mock what you own”. Answering a question about the practice of mocking out database access, he said that once the system boundary is reached, developers should define the external interface in terms of their domain, providing a concrete implementation of that interface that speaks to the external system. This way the code inside a system is isolated from external dependencies and the concrete implementation should be covered by integration tests. This idea comes from Alistair Cockburn’s Ports and Adapters architecture (also called Hexagonal Architecture), where the domain we work on is in the middle and ports help it communicate with the external world. Adapters for individual ports implement the responsibility for a particular external system integration. “Test adapters with the real thing”, suggested Freeman. In domain-driven-design this corresponds to specifying a repository for objects in the context in which we work, and then providing an implementation for that repository interface that talks to a specific database or a queueing system.

Speaking about different guidelines on when to use mocks, Freeman suggested a rule of thumb “we mock when the service changes the external world; we stub when it doesn’t change the external world – stub queries and mock actions”. However he said that teams should not be enforcing rigid rules.

As some of particularly bad ideas he’s seen and experienced with mocks, Freeman said that one of the biggest mistakes was to try to mock out standard java libraries. On a controversial topic of mocking concrete classes, Freeman said that it is generally a bad idea because that means that “you care about a dependency between objects enough to mock it but you haven’t made that relationship explicit in code”. If there is no interface defining the relationship between two classes then people won’t know which of the methods of the concrete class actually participate in the relationship. An interface suggests a role, and a class might have lots of other methods there. If the interface isn’t there then the relationship is just implicit, which will make it hard to understand and maintain these objects later. “Creating an interface also forces you to find a name for it so you might discover other things that weren’t obvious”, said Freeman. Freeman also suggested locking down all dependencies in unit tests (eg making them final), which forces developers to clear up where the dependencies are and how they are used.

Freeman’s upcoming book Growing Object Oriented Software, Guided by Tests covers this topic in more detail.

It was very encouraging to see the rising interest in agile acceptance testing, and we ended up almost running a full track on the topic with 4 presentations on lots of different aspects. I learned about Thoughtworks Twist from Andy Yates and narrative testing ideas from Antony Marcano and Andy Palmer. A key take-away from the conference for me was probably Marcano’s idea of “saving the game”, building systems so that you can easily store a snapshot of the current state and go back to that later. I can see how this can significantly aid in all kinds of testing, including acceptance and exploratory, so I have to look for ways to implement that in my future projects. Exchanging experiences of how people do acceptance testing was also very valuable and, judging by the comments at the closing session, it looks as if quite a few people went away with some new ideas around it.

A special treat for me was being able to pick Steve Freeman’s brains on mock objects, learn what the original idea behind that was and what he thought were his biggest mistakes in using mock objects.

I also got to speak with JetBrains developers on optimising build dependencies in TeamCity, and from that I went away with the idea to convert dependencies from build triggers to snapshots to avoid diamond build issues and optimise the cycle. I’m a bit unconvinced about that solving my problem but I’ll sure give it a try.

In retrospect, it was time well spent. The only negative remark is that classrooms didn’t really work for me as a way to organise openspace discussion. Lucky that we usurped the big room with chairs set in a circle for two acceptance testing sessions. Thanks very much to everyone who participated in organising the conference and I hope to see you next time.

I took lots of notes at the conference and will be cleaning that up and posting articles here next week, so subscribe to my RSS feed to get notified about that.