Chris Ambler, European test manager for Microsoft games, spoke about how games companies are testing software during his keynote Testing games is not a game – it’s serious stuff at XPDay 2008 today in London. Ambler said that the game industry is from a technology perspective doing much more bleeding edge than traditional industries, so it faces bigger challenges. As examples, he cited instantaneous response times for players worldwide, worldwide concurrent releases in twenty or more languages and having to get it right in first release (“what goes on disk stays on disk”). Because of that, Ambler argued that games producers face much bigger technical challenges and their QA practices have to be more advanced than in other branches of the software industry. According to Ambler, the trends and practices that he sees in this field are applicable to more traditional software markets, such as finance, and can help people produce better software.

Ambler talked about his engagement as the European QA director for Electronic Arts, where he tried to get the business leaders define what quality is in order to be able to assure it. The effort ended up concluding that people really did not know how to define quality. Quality means different things to different people, Ambler concluded. He defined his meaning of it as:

  • Progression: people can get through the software from the beginning to the end
  • Stability: people should not have unexpected problems with the software
  • Experience: emotion that people get out of it

Taking the first letter of each word and inverting it, he said that it’s like ESP – very hard to define precisely. According to Ambler, and this is consistent with what I’ve seen in a number of places, lots of effort gets wasted because people involved in different roles do not talk to each other. Ambler said that designers, testers and developers each think that their part of the business is the most important and they do not communicate effectively together. The way to achieve quality is to get these people to work together. He concluded that as quality is individual requires collaboration, rather than automation. As some practices that have significantly improved the quality on his projects, he cited getting test leaders to participate in developer scrum meetings, getting developers and testers involved in the specification process and focusing on collaboration, not automation.

Ambler said that historically, from mainframe systems to modern web and online games systems, the amount of development on projects is reducing and the amount of testing is increasing. This is due to the fact that we have better and better tools available for development. In the games industry, Ambler sees blending of these processes with design going throughout the project from start till shortly before the end. Development starts a bit after design and ends a bit after, testing as well. These processes no longer have separate phases and there is a blend between them on the projects. This is how the games companies keep up the quality of their products in spite of the technical challenges that they face, and Ambler sees this trend as more and more important for the traditional software industry as well.