Bulding smart teams

James Surowiecki, author of The Wisdom of Crowds, gave a keynote speech today at the first day of Agile 2008 conference in Toronto. His talk was about how to harness the collective intelligence better and what conditions have to be met for teams to be more intelligent than any single individual in the team.

Surowiecki started with a story about Francis Galton, a prominent English Victorian-age statistician infamous as the founder of eugenics, who organised a contest at a country fair in west England. There was an ox on display and people guessing the weight of the ox. Galton took the average of all guesses and expected a deeply flawed estimate, but the average was in fact spot on. Surowiecki also mentioned jelly bean counting experiments that are apparently a modern example of the same idea. The researcher holds a jar filled with jelly beans and asks the audience to estimate the number of beans in the jar. According to Surowiecki, the average of all answers is always very close to the true number of beans, often closer than any individual guess. The organisers of Agile 2008 conducted a similar experiment, asking conference attendees to estimate the number of lines of code in Visual Studio. The average value of all guesses was 47 million and the actual number is 43.3 million of code. The interesting thing as well is that only two people guessed better than the group average.

According to Surowiecki, Microsoft now uses a similar technique to estimate how long projects are going to take. He talked about an early experiment with implementation team members effectively betting on when the project is going to complete. Although the project was scheduled to end in November, most people were “buying shares” for February, sending a strong signal that the project is not on track. Bestbuy ran an experiment with polling 100 people from the company how much Christmas card sales are going to jump and that estimate was more precise than the estimate of their expert team. Experiments like the ones in Microsoft and Bestbuy also show that tapping into group intelligence allows a more efficient flow of information in organisations. People closer to the problem often have a better sense of what is going on than those higher in the chain, but hierarchical organisations often impede an efficient flow of information. Surowiecki quoted Lewis Platt, a former CEO of HP, “If HP knew what HP knows, we would be three times more profitable”.

Surowiecki’s comment about those experiments was that “under the right conditions, groups of people can be remarkably intelligent, can be smarter than even the most intelligent person in the group”. But that leaves us with the question of discovering the criteria you need to make your group a wise one. The answer to that question, Surowiecki argued, can be broken down into three categories.

  1. aggregation: there has to be an efficient way to aggregate lots of different judgments into a single collective judgment. The final product has to represent the conclusion of the group.
  2. diversity: “the more diverse the team is, the more likely it is going to be smart and produce good work”. In this example, the focus is on cognitive diversity, getting input from people who are thinking about the problem from different perspectives and use different approaches and heuristics.
  3. Independence: people have to offer their own judgment and knowledge rather than spend time looking at what other people want. Surowiecki said that “groups are smartest when people in them work as individuals”.

Diversity on the team

Although it works best for larger groups, from 100 people onwards, Surowiecki said that there is a lot of research that proves great effects even with groups of 6 or 7 people. The challenge with smaller groups is to work harder to foster diversity and individuality. Small groups also often have a problem with flushing out the knowledge that is not shared, as the discussion tends to focus on topics that everyone knows. So one of the key practices to create a smart small group is to focus on putting individual insights and knowledge into the open as well. During the Storytelling Skills for Agile Teams workshop in the evening, Rachel Davies and Rebecca Wirfs-Brock suggested writing designer stories that reflect concerns and ideas about the project from individual viewpoints and then making people read them out to the group, which might be an interesting solution to the problem of voicing individual knowledge.

Diversity is especially valuable in small groups because it expands the range of information. Bring people together who are not going to look at the problem from the same angle, and they will not make the same mistakes. You want people that make mistakes at worst not correlated, at best negatively correlated so that they cancel each other out. Without diversity, groupthink becomes a problem. Surowiecki said that “the more homogeneous groups talk to each other, the dumber they become”. Because of the same opinions, there is sort of an echo effect, and people become convinced that their ideas are correct. At that point, it becomes really hard to see blind spots. A possible solution for this is to appoint the Devil’s advocate, someone who should intentionally try to find flaws in common opinions. Surowiecki said that a single different opinion can often make a big difference but that a pitfall with this approach is that, if the same person is always the Devil’s advocate, other team members will start discarding that opinion. Having some less experienced people on the team can also help because they will ask questions that experts might overlook.

Group diversity is also important to eliminate the effects of peer pressure. Surowiecki quoted an experiment where eight people were asked to identify which of the three lines shown to them are of the same length, with seven people actually being told to intentionally give a wrong answer. The eight person (the one that was actually the subject of the experiment) often selected the same wrong answer to get along with the group. When the same experiment was repeated with six people giving the wrong answer and the seventh giving the right one, then the subject chose the right answer as well.

Surowiecki argued that the best group decisions do not emerge out of immediate consensus, but out of conflict, and that companies should promote a culture that encourages a “good fight”. Diversity is very good for that, but it can also pose a challenge, as it takes some work to make a diverse group function well.

Group intelligence is not a war on expertise — you need people who have a good idea of the problem in the first place. Polling random people on the street about the number of lines of code in visual studio would not produce a high-quality estimate. The underlying idea is how to expand the power of experts and get something better than would be produced by any particular person in the group.

How can we apply this to software projects?

An immediate thing that comes to mind is to build teams out of people with different opinions. Have people on the team that think differently, use different tools and approach the problems from a different angle. This will help the team spot blind spots easier and avoid the echo effect, increasing the team collective intelligence.

Getting a diverse group of people involved with flushing out specifications is one of the key practices I advocate to effectively implement agile acceptance testing. Business domain experts, developers and testers all have to be involved in discussing examples because they will all come with different concerns and ideas. By discussing all those ideas upfront we get a more detailed picture of the system and identify potential pitfalls much earlier. When the specifications are written only by a single person or a homogeneous group, lots of things pass unnoticed until much later in development.

I'm Gojko Adzic, author of Impact Mapping and Specification by Example. My latest book is Fifty Quick Ideas to Improve Your Tests. To learn about discounts on my books, conferences and workshops, sign up for Impact or follow me on Twitter. Join me at these conferences and workshops:

Specification by Example Workshops

How to get more value out of user stories

Impact Mapping

4 thoughts on “Bulding smart teams

  1. Gokjo,

    I didn’t know you were at Agile 2008. I hope we get the chance to meet while here. I’ve admired your writing and thinking for some time now.

    I missed the keynote this morning, so your explanation was very helpful to me.

    One question that occurs to me is this: How does the wisdom of crowds vary depending on whether the thing to be determined is a scalable or non-scalable quantity?

    From “The Black Swan”, “non-scalable” quantities are those that follow normal distributions. Like human height or weight, is becomes enormously unlikely to observe non-scalable quantities far from their mean. That means that the bulk of the probability distribution is relatively near the average value.

    Scalable quantities are those like annual income, ones where the probability of finding a value 10, 100, or 1000 times the average follows a power-law distribution.

    When observing a scalable quantity, people tend to treat it as intuitively similar to a non-scalable one. That is, they will dramatically underestimate the likelihood of finding a value far away from the average.

    It seems to me that the wisdom of crowds should work best with non-scalable quantities. On the other hand, “lines of code in Visual Studio” is a highly scalable value. In fact, I spoke to people tonight who had guessed everything from 85,000 lines to 150,000,000. (I guessed 5,500,000, myself.)

    To have the average of all guesses come so close to the real value is, frankly, almost chilling.

    -Michael Nygard

  2. Hi Michael,

    If you see me in the hall, just stop me and it would be great to meet you. Btw, Marisa and I are doing a talk on Thursday (at 16h at the Tools stage) so that might be a nice time to meet as well if we don’t catch up between sessions.

    Regrading the scalability question, I remember someone asking about different distributions after the talk but I don’t remember what the answer was. I guess that most of the keynote material, apart from the software development references, was simply taken from the presenter’s book, so that might be a good place to look for an answer.

  3. While defining software requirements we often depend on the product manager or business analyst. In fact these subject matter experts are one of the busiest people in the software product development supply chain. They tend to become the bottleneck. One way of putting wisdom of crowds to work is to take opinion of the end users and aggregate it using a web 2.0 like mechanism to facilitate posting reviews and ratings for the proposed features of the product. This will not only relieve the pressure on the BAs and Product Managers – but hopefully give more accurate requirements.

  4. Pingback: The Wisdom of Crowds and groupthink in Agile Software Development at Mark Needham

Leave a Reply

Your email address will not be published. Required fields are marked *