Tom Gilb spoke today at the Next Generation testing conference in London, arguing that testing is only part of the larger area of quality assurance. Gilb said that “real quality assurance” consists of seven ideas which are all clearly lean. “They are frugal, effective, non wasteful and early, which are all corner stones of lean”, said Gilb, adding that “‘Quality’ is far more than bugs. Bugs are an interesting subject but there are 50-odd other qualities your stakeholders are interested in”.

Starting the presentation, he stated that “quality assurance” is far more than testing, and that those other things can be far more cost effective, by a factor of 10 to 100. “You should be probably doing the other things if you want to ensure quality”, said Gilb. Citing Capers Jones’ research with 15000 businesses, he said the effectiveness of regression testing (as defined in catching problems that are out there) is only 15%-30%. Integration testing, according to the same source, is effective 25%-40%. The most effective tests are high volume beta tests, with more than 1000 clients, which measured 60%-80%. “We find all that we find, but we don’t know about the ones that we don’t find and we can have an illusion that those bugs are not there”, said Gilb.

Gilb quoted Capers Jones that between 8 to 10 defect removal stages applied together can lead up to 95% effectiveness. As practices to bundle together, Gilb offered designing quality in, defect prevention, inspections and applying combined testing best practices. The effectiveness of inspections goes from 25%-40% for informal design reviews to 45%-80% for formal code inspections. Although this does not solve the full problem, Gilb said that “the only saving grace for inspections was that you can spot things earlier and fix them more cost effective”. The point about inspections is not to increase quality, said Gilb. “Inspections do not impact the quality - quality is a constant and should be set in your requirements”, said he, “and the whole development process should aim to bring the system to the level of quality”. Inspections might help to get the system to the level of shippable quality earlier. “Although it’s fairly hard to remove defects once they are in there, it’s fairly easy to prevent them from going into the system in the first place”, said Gilb. Defect prevention is the most effective way of avoiding defects, with case studies of effectiveness over 90% over three years, said he.

“You don’t get quality by testing it in, you get it by designing it in”, said Gilb. Many business users never define what they actually want in a measurable way according to him, which is why projects do not meet the expected levels of quality by design. “To do quality assurance, you need to be able to quantify qualities important for your business”, said Gilb, adding that “you have to define robustness and intuitiveness, and to define it you need to define a scale of measure”. He gave an example of intuitiveness, which can be defined by how long it takes for new people to do a task. “Qualities are many and variable. There is not one dimension of measurement” said Gilb. He advised teams to look at system performance from the aspects of capacity (how much), resource saving (efficiency) and quality (how well). “The same applies to all the ‘ilities’”, said he, adding “Quantify the quality to ‘assure’ it. If you pretend to do quality assurance and neither anyone else can measure it your knowledge is of a meagre and unsatisfactory kind”. There will be dozens to hundreds of such qualities but Gilb said teams should focus on the top 10.

Seven key quality assurance techniques, according to Gilb, are:

  1. Establish who your stakeholders are, and the stakeholders should decide qualities. Stekeholders have values and needs and many of those needs translate into qualities.
  2. Quality quantification. “If you quantify quality, a lot of things follow. You can put it in a contract and pay when it is develop. You can give it to an architect and say deliver something to this level. You can give it to a tester and say check if it delivers it“, said Gilb.
  3. Assuring that designs give qualities. “When designing, you have to account for how effective you expect them to be with regards to your qualities. You have to be able to say: I think it's 50% to our goal. If you cannot say that, you're wasting time”, said Gilb.
  4. Measure quality levels in specifications with inspections. Many specifications are vague and ill defined and Gilb advised applying inspections to root out garbage from them.
  5. Establish numeric quality gateways. Define an exit process that tells you when you are done with a task. He gave an example of “no more than more one major defect per page” for inspections of requirements. Defining a numeric exit and publishing it causes people improve the quality of work. Gilb gave an example of a bank in London that went from 82 defects per page to no more than one in six months. “This is one of the most powerful ideas to change“, said he, explaining that numeric exit points give us good feedback
  6. Implement a defect prevention process. Gilb quoted a case study from Raytheon that improved quality 10 times by using a defect prevention process. At the start of the change, 43% of their time was spent on fixing bugs and rework. By fighting rework systematically, in 10 years they got to 5% of time spent on rework.
  7. Use frequent feedback for improvement and to assure quality.

The slides from this presentation are available at http://www.gilb.com. These concepts are also explored in more detail in his book Competitive Engineering