Quality over speed

Although most agile literature focuses on fast feedback as the primary idea to align software with customers’ needs, it seems to me from the discussion that the quality of feedback is more important than the speed. A good idea to improve quality of feedback was to invite passionate users to join in, giving them early access and allowing them to influence the product. Another good idea was to observe how the software is actually used (I proposed that as the new year’s resolution in 2008 on this blog, and considering that the new year is coming soon again, maybe it’s time to repeat that proposal).

Release and then measure

Of course, this doesn’t mean that quick feedback is not important. Finding whether we’re going in the right direction quickly is key to successful software development. One idea to improve this is to focus early deliveries on the smallest chunks of software that could provoke and provide meaningful feedback. Expanding on that idea, we discussed rapid prototyping instead of analysis. I must admit, I’m not completely in favour of that idea (and I recently wrote against releasing unpolished software), but this might work better for mass-market than in-house software which I mostly do.

This led to a discussion on the idea of releasing a bunch of features quickly and then asking the users to choose which ones they like and which ones should be binned. 37 Signals and Twitter were mentioned as examples for this, releasing and then dropping many features that didn’t work. The key to do this successfully for mass-market applications such as web sites is to have good monitoring on feature usage. Tealeaf customer experience monitoring was mentioned as a tool to support this.

Another idea on how to improve the quality of feedback is to build feature partitioning into the system, enabling features to be turned on and off on demand. This supports measuring effects of individual features. It also allows the system to gracefully degrade services if it starts running out of capacity. Mixcloud was mentioned as a good example of this in practice.

The right sample

As an idea how to improve the amount of feedback we can get from end-users, we discussed giving something in exchange for feedback. This is probably most applicable to web sites where you can give users small gifts or virtual goods in exchange for filling in a form.

We also discussed how choosing the right sample was crucial to get good feedback. An idea that I especially liked was to align the sample with user personae chosen for the project. Clearly defining the target market before a project starts and basing the feedback sample on that is one of those things that seem common sense but are yet to be adopted by software community at large. Another key idea for mass-market software was gradually growing the sample. An example of what can happen if you only listen to early feedback of a small group is the ISnack 2.0 blunder. Kraft foods chose ISnack 2.0 as a new name of their Cheesybite product based on positive feedback from focus groups, which provoked “Facebook hate groups, blogs and angry Tweets [..] and T-shirts trashing the name” when it went public and finally causing the company to admit defeat and bin the name.

Here’s the updated mind map with all the ideas from both workshops:

Some more food for thought

Adewale Oshineye spoke about one of his earlier projects (from what I remember, for Dixons) where an uncaught security bug turned out to be a very important feature allowing stores to share information. This reminded me of one of my projects where a feature essentially developed for easier testing of a network flow algorithm proved to be a key selling point for a system (I wrote about this already, see Blinded by the user interface). Jeff Patton said that both these stories are examples of value produced by accident, and then raised probably the most provoking question of the session: How do we intentionally discover features that users have not even thought of but that could bring a lot of value? This question was, unfortunately, left without an appropriate answer. Round 3 of this discussion is scheduled for the SPA conference next year in May, and this will surely be one of the major themes of that workshop.

See other articles from XP Day 09

The effort to introduce agile development was provoked by lengthy releases, often with half-finished features. With random work assignment, no clear prioritisation or plan, Rough described that previous development culture of the company was as “JFDI” (just f* do it).

The change started with a “shock therapy” scrum, introducing one week iterations with scrum by the book. Bowley said that the initial gain from this was very quick feedback. They implemented strict XP development practices, enforcing unit tests, integration tests and as much as possible acceptance tests for any new work. They also clearly defined team principles and communicated them. All work was done with a single stream without work-in-progress limits. In addition to scrum by the book, they implemented analysis workshops with the whole team around a whiteboard, converting to BDD scenarios after the meeting.

Development team was unable to do any longer term planning at this stage, explaining it by not having enough data. (However, they started collecting data at that point, and their average cycle time was 45 days). Doing very rough projections, they realised that “if we did it the way we were doing it, we wouldn’t do it in time”, said Bowley. The work was split between two teams to achieve the goal faster.

Ben Drury, the CEO of 7Digital, says that he was scared by the transition, especially as Bowley and Rough announced that it will get worse before it gets better. However, he said that in retrospect doing it was the right decision. “When you’re a smaller company competing with giant, one of the things that’s really important is speed”, said Drury, adding that the change in the process has improved their speed of innovation. They were able to build and release an open B2B API, which is now their competitive advantage. They now have the ability to for different teams to build products in parallel, where they only had the capability to work only on one thing at a time a year ago.

Stephen Somerville, the sales director of 7Digital, said that it was a steep learning curve to understand this new approach. “There was a period of great uncertainty and a real reluctance to give any estimates, which was a difficult thing to sell to our customers. Now we built up data [...], we’re in a really good place”, said Somerville, adding that the visibility and the statistics are really useful for him now. He also said that the new development approach is beneficial because “there are so many places where you can adjust the direction, so you don’t end up going too far [in the wrong direction]“.

Drury said that his initial concern was that they were having far too many meetings, which was later adjusted by the team. Instead of weekly retrospectives, they do a retrospective every two weeks, although the iterations still run weekly. The analysis meeting is merged with the prioritisation session, and is now run on demand. The team is now more proactive and also proposes features, and they implemented a fully automated production deployment.

Their average cycle time today is 20.4 days. They now have a dedicated tester, who leads testing and assists developers, partially leads analysis sessions and is responsible for pre-approval before a feature goes to the stakeholder for sign-off. Compared to six months ago, they spend more time on acceptance tests. They now run three streams with work-in-progress limits.

As one of the downsides, Drury said that there is a lot of jargon around the process and terms that they don’t understand so having more clarity in that area would be beneficial.

See other articles from XPDay 09. I’ll be posting more articles about this conference, subscribe to my RSS feed to get notified about that.

Making things more visible

The first thing Mitchell did is reorganising the task board from a classic scrum board to something that gives stakeholders much more visibility. The new board had a much more detailed breakdown of work phases on the horisontal axis and people on the vertical axis and work tasks placed on the grid with post-it notes. Mitchell took a picture of the board to the eXtreme Tuesday club to get feedback, which pointed to several problems. Developers were hoarding tasks with one particular developer completely overloaded with tasks he started. Things were also being batched up in the testing column, which pointed to a problem that most testing was done at the end. The redesigned board also made it visible that starting a new task is easy but that finishing a task was hard.

Moving away from scheduled meetings

Mitchell said that “in an environment where politically you can’t say No, the idea you have to stop and re-plan didn’t work”, so the typical scrum concept wasn’t really working for them. Wanting to “have some benefits of Scrum without running scrums”, they set up several rules for a task to move from one column on the board to another. Once a piece of work is ready for development, a developer was supposed to talk to at least one other developer about the intended changes. The team was responsible for deciding whether to organise a design meeting if they needed it. Before finishing a task, developers were required to write a how-to-test document. The testers were asked to spend one hour a day doing acceptance testing for tasks in development even if they are doing release regression testing that day. This improved the feedback after development and made things move quicker across the board.

Limiting work in progress

With the improved visibility they realised that “the more things you try to do at once, the slower it goes”, but it was hard to find the exact point where to cut it. In order to improve visibility even more, they reorganised the task board again. Instead of swim-laning by people, they started using swim-lanes for tasks in progress. They also had a date stamp to mark the start and the end of development on a task, helping to track how long things took to get through the flow. They extended the task board more upstream, adding analysis. Perhaps the most important change at this point was using stickers to represent developers so that a developer could work on only one task at a time and tasks weren’t assigned to developers until they are ready. This resulted in establishing a task queue for things that were analysed but not yet being developed.

Moving to a pull system

The next step was to sort out things piling up in the analysis queue. “What you really want to do as a project manager is not let the stuff start”, said Mitchell, adding that “specifications are like bananas, you can warehouse them but not for long”. Analysis was moved closer to development in the process and developers were allowed to pull tasks to analysis. By this time, it seems that the trust was restored and that the team was delivering things quicker. According to Mitchell, his team was no longer the bottleneck so they shifted the responsibility for slow deliveries to other teams. Seeing the analysis queue and understanding what goes on, traders started to realise that there is a limit to how much work can be done. “[They] stopped telling us about low order stuff”, said Mitchell. The board was changed again, and a limit of four swim-lanes for tasks in progress was imposed. The entire team started swarming on a feature. This helped reduce the queueing even more.

Giving up on time-boxed iterations

After a release with critical bugs, they stopped doing anything that wasn’t directly aimed at solving the problems. This resulted in six weeks without scrums or estimations or any other process overheads. After this, the team realised that the development works just fine with the flow and they kept it like that. Instead of time based iterations, they now had work-based cycles. Two week releases were still kept as a heart-beat.

Kanban to reduce political dangers?

This experience report was very interesting for me because all the changes were largely driven by political dangers. The need to increase visibility to mitigate political risks led to interesting discoveries about process flaws and improvements which help rebuild trust and improve the team’s capacity for delivering. It is also interesting that a quality crisis made them move from time based to work based iterations, which reminded me of the story of migration to lean and kanban at uSwitch.com.

I’ll be posting more articles about XPDay 2009 sessions soon. Subscribe to my RSS feed to get notified when I post something new

Google had a team of Test Mercenaries, who joined different teams for a short period of time to help them with testing. In most cases, they could see what was wrong after a few days and started helping the teams, but the effort wasn’t a success. When they left, teams would not improve significantly. Striebeck said that testing “wasn’t effective to teach”. Knowing what makes a good test often relied on personal opinion and gut-feel. Doing what they often do in similar situations, Striebeck said that they decided to collect data. The things that they were interested in were figuring out the characteristics of good tests and testable code and how to know in the first place that a test is effective. They decided to use a return-on-investment criteria: low investment (easy to write, easy to maintain), high return (alert to problems when it fails). According to Striebeck, Google spends $100M per year on test automation, and wanted an answer whether they are actually getting a good return on that investment. They estimated that a bug found during TDD costs $5 to fix, which surges to $50 for tests during a full build and $500 during an integration test. It goes to $5000 during a system test. Fixing bugs earlier would save them an estimated $160M per year.

To collect data, they set up a code-metrics storage to put all test execution analytics in a single place. Striebeck pointed out that Google has a single code repository, which is completely open to all of the 10000 developers. Although all systems are released independently (with release release cycles randing from a week to a month), everything is built from HEAD without any binary releases, and the repository receives several thousand changes per day with spikes of 20+ changes per minute. This resulted in 40+ millions of tests executed per day from a continuous integration service plus IDE and command line runs, they collected test results, coverage, buld time, binary size, static analysis and complexity analysis. Instead of anyone deciding whether a test is good or not, the system observed what people do with tests to rank them. They looked into what a developer does after a test fails. If the code was changed or added, the test was marked as good. If people changes the test code when it fails, it was marked as a bad test (especially if everyone is changing it). This means that the test was brittle and has a high maintenance cost. They also measured which tests are ignored in releases and which tests often failed inthe continuous build and weren’t executed during development.

The first step was to provide developers reactive feedback on tests. For example, the system suggested deleting tests that teams spent loads of time maintaining. They then collected metrics on whether the people actually acted on suggestions or not. The system also provided metrics to tech leads and managers to show how teams are doing with tests.

The second step, which is in progress at the moment, is to find patterns and indicators. As they now have identified lots of good and bad tests, the system is looking for common characteristics among them. Once these patterns are collected, algorithms will be designed to identify good and bad tests, and manually calibrated by experts.

The third step will be to provide constructive feedback to developers, telling developers how to improve tests, what tests to write an dhow to make the code more testable.

The fourth step in this effort will be to provide prognostic feedback, analysing code evolution patterns and warn developers that their change might result in a particular problem later on.

I will be covering XpDay 2009 on this blog in detail. Subscribe to my RSS feed to get notified when I post new articles.

Based on whether the IT is doing the right thing (aligned with business goals) and whether it is doing it right (effective), the researchers divided the companies in four groups:

The figures in the picture are comparing to the average and over three years – so, for example, the group that has IT aligned with the business but not effective has 14% less sales growth than the average of all surveyed companies, but they spend 13% more than the average on the IT. On another hand, the group that has a mismatch in what IT does and what the business wants, but with effective IT, spends 15% less on IT then average but sees 11% more sales growth than the average over three years.

A very important note from Allan’s talk is that the transition from the top-left corner into the top-right corner is virtually impossible, unlike the transition from the bottom right corner into the top-right corner. Doing the wrong thing but doing it right turns out to be better than doing the right thing poorly.

For more information, see Allan Kelly’s post on this topic.

See other articles about Xp Day 08