As an industry, I think that we are now well beyond the perception that quality can be controlled for success. Lots of good thinkers and thought leaders over the last 20 years have moved our thinking from control to assurance, but for the majority of teams I work with the responsibility of QA folks isn’t really assuring quality. In order to deliver effectively with short iterations, quality assurance has to be the responsibility of the entire team. Many techniques and practices that assure quality are in the requirements, analysis, programming and operations space, so people with testing hats on can participate but not really drive most of that. Trying to define the responsibility of testers in an agile world at one of my clients, we got to the conclusion that testers in their team should have two key roles:
- supporting the team in deciding what needs to be tested, how to test it and performing a specialist part of testing
- visualising current state of quality for stakeholders to make more informed decisions
This struck a chord with something I’ve picked up from Elisabeth Hendrickson (it’s incredible how many gold nuggets of knowledge she drops everywhere she goes) at agile testing days 2009. she said that people asking for control really want visibility. so quality control is out of the door, assurance is following, what people really want is quality visibility. most of what we’ve been focusing on as a community over the recent years has been in the space of supporting the team (specification by example, exploratory testing, tdd) and the most efficient things in the visibility space are still horrible tables with incomprehensible amounts of data or static code analysis tools.
So I’d like to throw a new problem at the agile testing community: how do we effectively visualise quality? how to we do it in a way that is easy to maintain but provides people with enough information to make decisions on releasing and process improvement?
The answers to these questions will no doubt be very contextual but we can probably repurpose, reuse or invent some good tools that work in particular contexts. I’ve started collecting some tools and techniques from lots of different places. Not sure yet what to do with it, but it will most likely be either a wiki or a book or both.
If you have a good way of visualising quality or a story to tell, please get in touch. I’d like to talk to you about it.
mail: gojko@gojko.com
skype: gojkoadzic
twitter: gojkoadzic
|
|
Each person is responsible for the way they communicate in the same way we are responsible for our hair cuts. Good engineers have no problems visualising what they do so that others can see. Working software, suites of tests, executable requirements, the one-page-planner, test drivers, test consoles, video, documents, etc..
I think that the idea a simple graphic could point to ‘what needed testing’ is naive. The whole point of reams of data is that it needs a smart human to interpret it, the data doesn’t speak for itself. The data, in its raw forms, has to have more layers of abstraction added to it, and I would say that’s the job of the team. There are no short cuts to this. Maybe Elizabeth is right, we do need more visualisation, but I would say that lack of it means you have a poor team, and that’s where they search should be.
Could you be looking for a shortcut where one does not exist? The rubbish on my street is not a result of bad process or bin location, it’s to do with the people in the area, if you know what I mean.
I think you got the wrong message. I’m not looking for a simple graphic representation of what needs testing. i’m looking for techniques that people use to visualise the state of where they want to go and where they are now.
test drivers, consoles, executable requirements, suites of tests etc are good but very low level. higher level things are needed to support higher level informed decisions, that affect a whole set of parts of a large system across many different teams, for example.
examples I have so far are effect maps, QFD house of quality, ACC matrix from google…
I like where I think this is going. The solution for this ‘problem’ may be nearer than we think.
The issue is still more about data and less about the end point – visualisation. More importantly what information do you use to represent the solution state and what represents the current. Most importantly, how do you ‘measure’.
Ultimately the measure of anything produced for consumption are two-fold.
First, the measure of satisfaction of the product by its consumers
Second, and much less acknowledged is the cost of production (this encapsulates too many important thing – but you get the gist).
On the first – it really is down to consumer involvement – not just some dumbass questionaire after the fact, but close engagement before, during and after production. And repeatedly. A level of engagement that very very few organisations understand much less invest in.
So I think that the visualisation is a red herring, so is the depth of measuring. It is not that we do not measure enough. It is worse, we do not measure enough of the right things. So if we can reframe this ‘problem’ – what should we really be measuring the quality of software against? Then perhaps when we do that, the visualisation will be a doddle.
Mike
You’re right – a very important part of the “visualisation” problem is actually choosing what to visualise. lots of stuff that gets measured isn’t really providing a holistic view at what is important. so what do you measure and how do you present it in your context?
I look at the problem from a different angle. Quality or good enough quality is subjective. It depends on what trade offs are you willing to take and how they impact the user. You should follow good testing practices of course, but you will only be able to get a picture of your product’s quality when you put it in the user hands, so I do that as early as possible. Call it pilots or alpha/beta , but talk to your users and you will get a true picture of quality. People over processes.
Victor,
what you are saying is that the user decides on quality – but why should they have to wait until there is something put in their hands? How do we engage them better to help them define what quality is for them upfront? So that there is a clear target to aim for? How do we visualise that to show how we are doing?
What we also should take into account is that there are several tastes in quality to visualize. The two most common in any project are; the quality of the process and the quality of the product.
The first is probably the easiest to measure as we are already involved in this process. The trick here is to find out what measures our stakeholders (also) value. Then measure them in a meaning full way and find a visualization that makes sense to both the stakeholders and us.
The second is likely to be influenced by the first but is not necessarily a result of it. In more traditional development methods the process often works fine but the result not always is what was expected or desired. (One of the reasons people choose to use agile.) So if the process does not answer this question what does?
I think for a large part any answer to that question only emerges when the product is out into the real world and used. So we should have the people that actually use it come in and look at it as soon and as often as possible and tell us what they value. In this we can additionally learn from our marketing colleagues and use things like observations, panels and referential products.
Jean-Paul,
I agree with you that there are several aspects to visualise, but I don’t think we lack methods of visualising process: scrum boards, kanban boards, burn down charts etc visualise progress nicely and people have done a lot of work on that over the recent years. Let’s focus more on the second part, the vague one.
Good thoughts, nice challenge. It made me think about terms:
There are two different meanings of the term ‘control’. The one where you actively controlling something (e.g. as you do with a remote control for a toy), and the control where you are checking that things are in order.
For some reason we use the same word for both (even in Danish)!
I think you are talking about the active control, but the passive shouldn’t be discounted.
Another term which I think is important to consider is ‘quality’. There are many definitions of ‘quality’, but in this case I’d like to draw attention to the fact that is no single ‘quality’ of a product.
We can precisely describe ‘a quality’ when we are talking about a property or a collection of properties of a product, but we cannot precisely describe THE quality of a product. It’s holistic.
This may seem odd because quality is something that everyone can have (and has) an opinion about, but the thing is that this opinion is always only subjective.
Now my point is, that it is only possible to measure (and visualize) quality when it is fragmented into well defined test operations on a product. So we have to trade the qualitative (sic!) element of ‘quality’ for quantitative measurements. Deciding what to measure is where the context comes into play.
I’ll brain storm and submit my input to you.
/Anders
My initial thoughts –
One of the ideas I’ve used to tackle this problem is a heat map. This is a 2d map of the system areas (we get this map either by asking the team about the important areas of the system, or by using a reference such as the main menu items etc) where the size of the circles represent complexity of each area (if you like the number of tests we should have), and the colour of the circles the coverage we feel we have achieved (green==good to red == poor). Alternatively you can indicate the coverage by what portion of the circle is filled.
Because I am often brought in on legacy projects with severe test-technical debt, the tests completed/run statistics only tell a small part of the story. Often the most interesting piece of information to the business, is the level of RISK, which is obviously high in areas where we know our tests coverage is less than we’d like (or non existent in many cases..). That is why I like the above heat map approach as we can easily take this into account.
Therefore, I think if we are going to start really nailing this point we need to stop thinking in numbers of test cases executed and start thinking in RISK. Maybe, as part of this we can incorporating code coverage analysis/complexity analysis/ and code change frequency into the picture in determining the RISK.
I believe, with a cross functional team approach to quality (rather than just individual testers working on a finished build), creating such systems should now be in the realms of what is achievable.
Thanks for your post Gojko,
Clare
Hi Clare,
I definitely like the heat map idea as I have tried a similar approach suggested in this article http://www.stickyminds.com/sitewide.aspFunction=edetail&ObjectType=COL&ObjectId=14400&tth=DYN&tt=siteemail&iDyn=2 and it was a simple and light weight manner for the team to asses the quality of testing. The challenge starts I guess when trying to determine what is deemed to be enough and the trade offs the team is willing to make base on the stakeholders decisions as these discussions can be very subjective
Adrian