Nov 3, 2010
Published in: Testing articles
Tom Gilb spoke today at the Next Generation testing conference in London, arguing that testing is only part of the larger area of quality assurance. Gilb said that “real quality assurance” consists of seven ideas which are all clearly lean. “They are frugal, effective, non wasteful and early, which are all corner stones of lean”, said Gilb, adding that “‘Quality’ is far more than bugs. Bugs are an interesting subject but there are 50-odd other qualities your stakeholders are interested in”.
Starting the presentation, he stated that “quality assurance” is far more than testing, and that those other things can be far more cost effective, by a factor of 10 to 100. “You should be probably doing the other things if you want to ensure quality”, said Gilb. Citing Capers Jones’ research with 15000 businesses, he said the effectiveness of regression testing (as defined in catching problems that are out there) is only 15%-30%. Integration testing, according to the same source, is effective 25%-40%. The most effective tests are high volume beta tests, with more than 1000 clients, which measured 60%-80%. “We find all that we find, but we don’t know about the ones that we don’t find and we can have an illusion that those bugs are not there”, said Gilb.
Gilb quoted Capers Jones that between 8 to 10 defect removal stages applied together can lead up to 95% effectiveness. As practices to bundle together, Gilb offered designing quality in, defect prevention, inspections and applying combined testing best practices. The effectiveness of inspections goes from 25%-40% for informal design reviews to 45%-80% for formal code inspections. Although this does not solve the full problem, Gilb said that “the only saving grace for inspections was that you can spot things earlier and fix them more cost effective”. The point about inspections is not to increase quality, said Gilb. “Inspections do not impact the quality - quality is a constant and should be set in your requirements”, said he, “and the whole development process should aim to bring the system to the level of quality”. Inspections might help to get the system to the level of shippable quality earlier. “Although it’s fairly hard to remove defects once they are in there, it’s fairly easy to prevent them from going into the system in the first place”, said Gilb. Defect prevention is the most effective way of avoiding defects, with case studies of effectiveness over 90% over three years, said he.
“You don’t get quality by testing it in, you get it by designing it in”, said Gilb. Many business users never define what they actually want in a measurable way according to him, which is why projects do not meet the expected levels of quality by design. “To do quality assurance, you need to be able to quantify qualities important for your business”, said Gilb, adding that “you have to define robustness and intuitiveness, and to define it you need to define a scale of measure”. He gave an example of intuitiveness, which can be defined by how long it takes for new people to do a task. “Qualities are many and variable. There is not one dimension of measurement” said Gilb. He advised teams to look at system performance from the aspects of capacity (how much), resource saving (efficiency) and quality (how well). “The same applies to all the ‘ilities’”, said he, adding “Quantify the quality to ‘assure’ it. If you pretend to do quality assurance and neither anyone else can measure it your knowledge is of a meagre and unsatisfactory kind”. There will be dozens to hundreds of such qualities but Gilb said teams should focus on the top 10.
Seven key quality assurance techniques, according to Gilb, are:
Get practical knowledge and speed up your software delivery by participating in hands-on, interactive workshops:
Get future articles, book and conference discounts by e-mail.