Here is a situation I have seen a few times more than I would have wanted: the team and the clients split the whole project into several phases that will be shipped every few weeks; early deliverables mostly provide the plumbing for the later ones, and have only enough user-interfacing functions for the plumbing to be tested functionally. The clients set up and play with each new release. They dutifully give their feedback and participate in testing each time. When the whole thing comes together on the end, it turns out that the system still requires a lot of changes to solve the problem that it was intended for — the iterations and customer feedback failed to provide the directions.

The problem hides in way that the project was split: early deliverables mostly provide the plumbing for the later ones and the whole thing comes together on the end. So, until very close to the end of the project, the deliverables could not really be used in production. Unless the software is really being used, we cannot expect authoritative feedback. The customers may be able to give us their general gut-feel and comment on functional correctness, but usability and feasibility can only be measured once the software is really actually used by people in the course of their daily business. Technical integration issues as well as usability (human-integration) issues are expected and typically not problematic to solve, but we must not let them pile up. Early feedback should allow us to solve some of those problems during the whole project, instead of finding them out on the end.

One more reason why deliverables should be used in production is that the customer’s business will rarely stand still during the course of the project, making the end result effectively a moving target. If the deliveries are not really used until the end of the project, then we have no idea whether they are in touch with the present business requirements. On the other hand, if the deliverables are used every day, we will know that straight away – and most likely the clients will ask for adjustments early.

So how do we avoid this problem?

Here are three ideas to avoid the problem:

• Focus on user stories, not on technical use cases
• Plan deliveries so that they are complete in the sense that they can really be used in production
• Divide and conquer: Split a big project into several mini-projects, and then focus on delivering those in a sequence

Focus on user stories

User stories have emerged over the last several years as the preferred way of planning scope for agile projects. Focusing on user stories is a great way to split the project into parts that actually bring value to the customer, since every user story should have a clearly defined customer benefit. So planning deliverables in terms of user stories instead of technical use cases is a great first step.

A pitfall that we must avoid is to then batch stories based purely on technical dependencies. Although it makes perfect sense to have early deliverables provide the technical infrastructure for latter ones, and that type of dependency must be taken into consideration when planning deliverables, ordering purely based on that leads to the exact problem that we are trying to avoid. This is effectively incremental, not iterative development.

Have production-worthy deliveries

Instead of focusing on technical order, it is better to try plan deliveries so that every shipped version of the software can really be used in production. Although it is not always possible to adhere perfectly to this idea, it is a good general rule and we should try to keep the deliveries complete in that sense. While working on the delivery plan, just ask the clients this simple question: “pretend it’s magic and in a few months we give you these stories — would that piece of software be complete in the sense that it can actually be used in production?”. If the answer is no, go back to the drawing board.

Doing production-worthy deliveries several times during the project might require writing some throw-away code, but that is a small price to pay for early feedback (to programmers) and getting business benefits early (to customers). For example, several years ago I worked on a trading system that was supposed to replace excel-based trading and distribution management. One of the major problems that we were supposed to solve was optimising transport routes and providing transport plans to local centres. So our first delivery was a piece of software that allowed traders to input their trading plans (from the excel sheets), produced an optimised transport plan, and allowed them to export it back to excel. The first and third part of that delivery were thrown away later, but we got good feedback on a fairly complex part of the system and solved tons of edge cases that were not identified during initial planning. Traders were happy because this saved them a few hours of work every day, and they got that several months before the whole system was in place.

Break a big project into mini-projects

Reg Braithwaite wrote that his Golden Hammer for building software iteratively is to divide it into separate products. This makes excellent sense for building larger systems where the software will perform a lot of functions for a lot of people on the end. By dividing one single large product into smaller products we force ourselves to have cleaner interfaces and looser coupling between software components. This can allow us to develop them independently, and will also reduce the risk of the whole thing missing the target on the end. Each of those mini-products would be a deliverable, and it should be usable on its own.

I recently worked on a highly scalable e-commerce web site. We split the project into three separate products: an e-commerce integration platform, high-performance caching system and the actual of web site. The integration platform was delivered first, with a low-fi card processing web that was used for some 3rd party integration. It went live a year before the rest of the system, helping us flush out any problems with transaction processing. We had new requirements about this part of the system every few months, driven by the fact that people were actually using it. So when the project was close to the end, this integration platform was in touch with reality. The caching system went live next, and it was used to support a mobile content distribution application. Both the low-fi card processing site and the mobile app were not part of the initial plan, but we were able to slot them in nicely. The web site front end then built upon the foundations of those two systems that were in production for months. Thinking about the whole project in terms of separate products both helped us deliver it quicker (because we had real production feedback for parts of the system early on) and also increased the value of what we were doing (since the same components were used to support different applications and smaller projects that started after that big one). Divide and conquer approach paid off big.

Deliveries and iterations

I know that some of you, reading this, will now start to complain how iterations should be two to four weeks long and that the business problems are often larger than what can be solved in such a short period of time. I intentionally avoided using the word “iteration” in this discussion, because I do not think that every iteration needs to be delivered.

If an iteration result is not complete in the sense that it can really be used every day, then the iteration delivery will just sleep on the UAT servers after testers play with it for a while. To get quick feedback on such iterations, I propose giving clients access to the local integration environment and letting them play with the software there. You can avoid packaging and polishing and they do not have to update their local installations.

If an iteration produces software that can really be used every day, then ship it and make the clients use it. It is definitely useful to get feedback early, but having something go live, even if it causes rework or throwing away parts of software later, is much much more valuable than software that is only tested. It has more value for the customer, since they are getting a part of their problem solved earlier. It also has more value for the development of the whole project, since it brings real, authoritative feedback from everyday use.

Image credits: Emin Ozkan and Sachin Ghodke/SXC

Incremental is not iterative

Using a very interesting combination of pop music and rock star images, Jeff Patton told a story of a failed agile project in his XpDay keynote “Embrace Uncertainty”. The project started off nicely, almost by the book, with customer involvement and stories split into iterations, based on what functionality is to be delivered in what release. After they got something delivered to play with, customers changed their minds (as they so often do) and new stories and features were introduced into the plan. After a few deliveries, the scope kept growing and growing instead of reducing. From the developer perspective everything worked as planned – customer was expanding the scope and developers are there to oblige, because that is the essence of agile practices. Spice Girl Mel B was used for the role of a developer writing user stories and losing all sight of the big picture (while “So tell me what you want, what you really really want” was playing in the background). For the customer, the thing simply did not work – iteration after iteration, they were not any closer to having the project done.

The problem was that the concept of iterating had been lost and iterations were replaced by increments – making the project fall into the same trap as waterfall projects do. Iterations were aimed to deliver blocks of functionality. Jeff demonstrated that visually using the Mona Lisa painting (powerpoint slides from the keynote have still not been released, so the pictures below are not the same ones as Jeff used, but they show the point):

	Delivery 1	Delivery 2	Delivery 3
Incremental plan
Iterative plan

Although both these plans aim for the same thing, the incremental plan does not really reduce the risk of delivering something unsuitable to the client. The big picture appears only at the very end. Because increments are done in detail, a lot of effort is wasted when a piece needs rework (and the initial releases are almost certain to fall into this category). Iterative development offers a chance to see the picture from the start, and guide the development towards the full picture in steps. Not carving stuff in stone from the start allows us to change them easier later on, and we know that we’ll need to do that. Jeff gave the following rule of thumb to check quickly if your plan is iterative or incremental: “it’s not iterating if you do it only once”.

Dealing with uncertainty

However, iterative planning is harder to sell. Stakeholders want the project delivered by a fixed date on a fixed budget, and an uncertain iterative plan does not help with that. The XpDay keynote also offered three strategies for dealing with that:

• Follow the money: focus the project on business goals, not on features nor user stories. Prioritise goals before prioritising stories – that will make it easier to decide what features need to be implemented completely by when. I wrote about this subject last year (see The magic of goals), discussing how focusing on goals makes everybody happier.
• Don’t choose your solution too early: defer writing user stories till last possible moment. Focus on what users need to accomplish at that moment. This is intended to allow easier rework and reduce wasted effort. Lean software development has a similar principle “Defer Commitment”, which calls to Maintain Options (Think of code as an experiment – make it change-tolerant) and Schedule Irreversible Decisions at the Last Responsible Moment (Learn as much as possible before making irreversible decisions). See Lean Software Development: An Agile Toolkit and Implementing Lean Software Development: From Concept to Cash.
• Build up feature quality iteration by iteration: this helps to stay within budget, deliver all features in each iteration but not necessarily all on the same level of quality. This technique is especially effective with customers that cannot prioritise features and insist on having everything delivered. Jeff discussed an example of a bus – it’s pointless to ask the client whether brakes or transmission are more important. But the good thing with software is that you don’t have to deliver everything at top quality from the start. We can first build a bus with low quality transmission and high quality brakes, then improve transmission in the next iteration.

Image credits: Malene M./SXC