Gojko Adzic

David Evans and I facilitated a session on designing a killer agile test management tool last week at the UK Test Management Forum, with the goal of learning what are the biggest currently unsolved problems for agile teams in the area of testing at the moment. So for any tool vendors our there, here are the ideas.

We ran a variant of the Product Box game, described by Luke Hohmann in Innovation Games: Creating Breakthrough Products and Services, with five teams competing to design the best agile test management tool. They gave their products the following names: Silver Bullet, Agile Manager, Perfect 10, Nimble and FleXT.

Getting all kinds of different tools to talk to each other seems to be the biggest problem in the community at the moment. Instead of a monolithic all-in-one solution, all groups went for integration with other tools. Perfect 10 has “extensive integration with unit test tools, build tools, source code repositories”. Agile manager is able to “pull the requirements from whatever the requirements management tool you are using”. FleXT even integrates with third party defect management systems. The ability to pull the data out of a management tool also seemed very important, with FleXT offering REST and MS office interfaces and Nimble having a “unique universal adapter”. With such a wide range of integrated collaborators, the main feature of these tools seems to be rich reporting. Agile Manager automatically offers traceability to requirements and allows you to “dig in, find out the whole trend analysis, what the developers are doing, whether it’s working, whether it’s not working, who’s working hard and who is not”. FleXT also has coverage management – other teams didn’t seem to think that this was a key feature.

Improved collaboration was an interesting topic as well. Agile Manager offers continuous integration and analysis: “While developers are running the code we are telling them whether it is going to work or not. We’re going to put red squiggly lines below the code.” Nimble allows people to “block and tackle any problem”. It also performs continuous testing and has a “soft wall for code review”. FleXT has a virtual story board with virtual sticky notes.

A very surprising outcome of this workshop for me was that most tools seemed to provide a holistic view at a range of projects instead of focusing on managing tests. Many of them were more about story management than tests or defects. Perfect 10, for example, has a product catalogue and, given the priorities, works out automatically which sprint a feature will go into and automatically produces velocity and burndown charts. It has a current sprint view with a quick status overview of the requirements and tests for an on-going iteration. It also has a product catalogue dashboard, with a quick overview of several projects and the ability to drill down to identify problems. Nimble has “cross-project control, resource monitoring, defect tracking, task progress and graphical dashboards” and “shows you exactly where you are and where you want to be”. Agile Manager goes even further to produce a “a cost/benefit analysis model – how you’re moving against your budget” and predicts what the implications might be if someone comes in and changes one of the requirements, integrated with the cost benefit model. “if you put that feature in, that’s great but it’s going to set the project back this by this much time and it’s going to cost this amount of money”.

Coming back from the future into present time, many of these features already exist in a bunch of different tools, but the workshop suggested that we are lacking a universal aggregator for all the data that provides a holistic view at the whole product range. Sonar seems to be the closest thing to an universal integrator I’ve seen so far – it provides heuristics and analytics on code coverage, unit testing and code quality with a plug-in extension model. So we could easily extend it to pull in functional test statistics and defect management metrics as well.

I wouldn’t be surprised if many story management features mentioned exist in one of the popular agile project management tools, but what those tools are probably lacking at the moment is the holistic approach.

Photos: Nathan Bain

Get notified when I post something new - subscribe via RSS or Twitter!

Trackback URI | Comments RSS