James Surowiecki, author of The Wisdom of Crowds, gave a keynote speech today at the first day of Agile 2008 conference in Toronto. His talk was about how to harness the collective intelligence better and what conditions have to be met for teams to be more intelligent than any single individual in the team.

Surowiecki started with a story about Francis Galton, a prominent English Victorian-age statistician infamous as the founder of eugenics, who organised a contest at a country fair in west England. There was an ox on display and people guessing the weight of the ox. Galton took the average of all guesses and expected a deeply flawed estimate, but the average was in fact spot on. Surowiecki also mentioned jelly bean counting experiments that are apparently a modern example of the same idea. The researcher holds a jar filled with jelly beans and asks the audience to estimate the number of beans in the jar. According to Surowiecki, the average of all answers is always very close to the true number of beans, often closer than any individual guess. The organisers of Agile 2008 conducted a similar experiment, asking conference attendees to estimate the number of lines of code in Visual Studio. The average value of all guesses was 47 million and the actual number is 43.3 million of code. The interesting thing as well is that only two people guessed better than the group average.

According to Surowiecki, Microsoft now uses a similar technique to estimate how long projects are going to take. He talked about an early experiment with implementation team members effectively betting on when the project is going to complete. Although the project was scheduled to end in November, most people were “buying shares” for February, sending a strong signal that the project is not on track. Bestbuy ran an experiment with polling 100 people from the company how much Christmas card sales are going to jump and that estimate was more precise than the estimate of their expert team. Experiments like the ones in Microsoft and Bestbuy also show that tapping into group intelligence allows a more efficient flow of information in organisations. People closer to the problem often have a better sense of what is going on than those higher in the chain, but hierarchical organisations often impede an efficient flow of information. Surowiecki quoted Lewis Platt, a former CEO of HP, “If HP knew what HP knows, we would be three times more profitable”.

Surowiecki’s comment about those experiments was that “under the right conditions, groups of people can be remarkably intelligent, can be smarter than even the most intelligent person in the group”. But that leaves us with the question of discovering the criteria you need to make your group a wise one. The answer to that question, Surowiecki argued, can be broken down into three categories.

  1. aggregation: there has to be an efficient way to aggregate lots of different judgements into a single collective judgement. The final product has to represent the conclusion of the group.
  2. diversity: “the more diverse the team is, the more likely it is going to be smart and produce good work”. In this example, the focus is on cognitive diversity, getting input from people who are thinking about the problem from different perspectives and use different approaches and heuristics.
  3. Independence: people have to offer their own judgment and knowledge rather than spend time looking at what other people want. Surowiecki said that “groups are smartest when people in them work as individuals”.

Diversity on the team

Although it works best for larger groups, from 100 people onwards, Surowiecki said that there is a lot of research that proves great effects even with groups of 6 or 7 people. The challenge with smaller groups is to work harder to foster diversity and individuality. Small groups also often have a problem with flushing out the knowledge that is not shared, as the discussion tends to focus on topics that everyone knows. So one of the key practices to create a smart small group is to focus on putting individual insights and knowledge into the open as well. During the Storytelling Skills for Agile Teams workshop in the evening, Rachel Davies and Rebecca Wirfs-Brock suggested writing designer stories that reflect concerns and ideas about the project from individual viewpoints and then making people read them out to the group, which might be an interesting solution to the problem of voicing individual knowledge.

Diversity is especially valuable in small groups because it expands the range of information. Bring people together who are not going to look at the problem from the same angle, and they will not make the same mistakes. You want people that make mistakes at worst not correlated, at best negatively correlated so that they cancel each other out. Without diversity, groupthink becomes a problem. Surowiecki said that “the more homogeneous groups talk to each other, the dumber they become”. Because of the same opinions, there is sort of an echo effect, and people become convinced that their ideas are correct. At that point, it becomes really hard to see blind spots. A possible solution for this is to appoint the Devil’s advocate, someone who should intentionally try to find flaws in common opinions. Surowiecki said that a single different opinion can often make a big difference but that a pitfall with this approach is that, if the same person is always the Devil’s advocate, other team members will start discarding that opinion. Having some less experienced people on the team can also help because they will ask questions that experts might overlook.

Group diversity is also important to eliminate the effects of peer pressure. Surowiecki quoted an experiment where eight people were asked to identify which of the three lines shown to them are of the same length, with seven people actually being told to intentionally give a wrong answer. The eight person (the one that was actually the subject of the experiment) often selected the same wrong answer to get along with the group. When the same experiment was repeated with six people giving the wrong answer and the seventh giving the right one, then the subject chose the right answer as well.

Surowiecki argued that the best group decisions do not emerge out of immediate consensus, but out of conflict, and that companies should promote a culture that encourages a “good fight”. Diversity is very good for that, but it can also pose a challenge, as it takes some work to make a diverse group function well.

Group intelligence is not a war on expertise — you need people who have a good idea of the problem in the first place. Polling random people on the street about the number of lines of code in visual studio would not produce a high-quality estimate. The underlying idea is how to expand the power of experts and get something better than would be produced by any particular person in the group.

How can we apply this to software projects?

An immediate thing that comes to mind is to build teams out of people with different opinions. Have people on the team that think differently, use different tools and approach the problems from a different angle. This will help the team spot blind spots easier and avoid the echo effect, increasing the team collective intelligence.

Getting a diverse group of people involved with flushing out specifications is one of the key practices I advocate to effectively implement agile acceptance testing. Business domain experts, developers and testers all have to be involved in discussing examples because they will all come with different concerns and ideas. By discussing all those ideas upfront we get a more detailed picture of the system and identify potential pitfalls much earlier. When the specifications are written only by a single person or a homogeneous group, lots of things pass unnoticed until much later in development.