Iterative delivery is bringing software production very close to monthly magazine publishing. Although being on time and without problems cannot be attributed to most software releases, good magazines maintain quality and produce great issues month after month. What can we take from magazine publishing to improve software releases?

I spent two years as editor-in-chief of Mikro (PC World edition for Serbia).Relja Jovic (then executive editor, now editor-in-chief) and I significantly improved the production process, leading the magazine out of constant fire-fighting and staff burnout into a stable production rhythm. While most of our competitors were sinking in the recession, we improved - in a year, our magazine became the market leader, nearly doubling the circulation. We achieved this by applying quite a few tricks and principles from agile software development, because magazine publishing looks strangely similar to a software shop. If those two industries are based on similar principles, and I think that they are becoming more and more similar, how come that it is normal for magazines to be on time, but at the same time nobody is really surprised when software is late? I think that the missing link between iterative software projects and publishing is mostly in the perception of the delivery - the publishing approach can be used to make software deliveries much more reliable.

How can magazine publishing relate to software?

I found a lot of similarities between publishing and software development - both software and magazines are, on the end, combined products of a diverse group of contributors working in parallel, on several threads, with similar resource and ‘integration’ issues.

Writers were for the magazine what programmers are in a typical software company - they produced the articles (source) which then had to be cleaned up (unit-testing), put into a nice layout (front-end GUI), assembled into a magazine (integration, packaging), cleaned up again because of layout and spelling errors (integration testing) and published. Pre-press and technical editors were our front-end developers and integrators, proof-readers were our testers.

Each magazine issue was a mix of big features and less important articles. Some had to be published due to contractual obligations or timing - they were like high priority software updates. Other articles could be postponed - they were like low-priority software features. Some topics were small and could be finished in a single monthly cycle. Others were quite big and took a few months to plan and prepare. It’s similar in the software world - some features are implemented over a few iterations, and some can be developed, packaged and delivered in the same iteration.

The money came in from two sides. Readers are the equivalent of small individual software users, and advertisers are the equivalent of big corporate software users that pay a lot of money, but usually come with a list of demands. We had to keep good relations with advertisers, so sometimes we had to cover hardware that no normal reader would be interested in, or events and conferences which our advertisers sponsored. That is like a big customer demanding some obscure feature, or stopping you from throwing out a piece of legacy code which nobody else uses.

We experienced a lot of problems that are common in software production - articles taking too long to to finish, people getting sick or failing to complete assigned tasks on time, unavailable resources. Our plans constantly changed to include last-minute wishes of sales and marketing. Yet, every month we published a new issue (delivered a new release, or iteration), never missing a deadline, with only minor problems (production bugs) getting through. Though software is arguably faced with more complex challenges, reaching every milestone, always on time, without staff burnout, and with only minor production issues, is not an easy task. I wish that I could achieve the same with my software projects.

Achieving predictability

Looking back, there were two major components to our success which I cannot find in most of my software experience: predictability and consistency.

Successful products are aligned with customers’ expectations - when people get what they expected from a product, they are going to be satisfied, and will probably buy it again. Mikro was published on 164 pages, on the first of each month. Each issue had four to five big features, about twenty pages of news, thirty pages of hardware reviews, ten pages of software, ten pages of multimedia-related articles, and so on. Note that this general structure does not imply what the big features were, nor which software and hardware products were reviewed - but the general structure was there. No matter how hard I try, I cannot describe common features of my software releases in just two sentences with such consistency.

Readers and advertisers knew exactly what to expect, because we actively helped them to form those expectations. Then, we met the expectations month after month - always trying to exceed them a bit, but not too much. The predictability of our ‘releases’ allowed us to make realistic plans and had an excellent effect on customer relations - we never let our customers down.

Predictability could be reached easier then with software because there were less ‘unknowns’. Software projects are mostly a compromise between four factors: quality, cost, time and scope. For the magazine, time and scope were fixed: it was published on specified dates, and had a fixed number of pages.

We had a very strict ‘delivery’ slot. It was not just marketing, and living up to expectations that the magazine will be in the mailbox on the first of the month. I think that our readers and advertisers could forgive us for being late a day or two. The biggest problem was printing - there were just two companies in the country which could meet our printing requirements, and their services were booked several months in advance. If we missed our printing slot by more than a few hours, the issue simply could not be published - and all our efforts would be pointless. Imagine that as a deadline for your software project.

Though this constraint might seem like a big obstacle, it really helped the process a lot. As we knew that we had strict ‘release cycles’, we had realistic plans - much more realistic than those I see in software. This does not mean that marketing did not push us to the limit with last minute changes, but everyone in the organisation knew that we could not afford to be late - so there were no unrealistic promises.

Although such hard time constraints are rare with software, I think that artificially imposing them could help a lot. It would remove one variable from the project equation, and significantly reduce the complexity of the problem. The customers would surely appreciate having predictable release dates, since they could make long term plans for upgrades. I am sure that marketing and sales would love to have more predictability. Most programmers would object at first, probably, but that is where the other constraint comes into the play.

In addition to knowing ‘when’, we also knew ‘how much’. Each month, we had to publish 164 pages. If there was no time to complete some articles, we moved them to the next issue, and threw in a few items that were easier to do. But on the end, we had 164 pages. The marketing could not demand that we publish 170 pages one month (or 500, which would be more likely if our magazine was software). The programming equivalent of this constraint would be a fixed release scope. This ‘fixed scope’ does not imply specific features, just the quantitative customer perception. We had to deliver a fixed number of pages, in a generally defined structure, and that is what our readers and advertisers expected. In order to reach the deadline, we chose a mix of hard and easy features for each issue. Some were very important, some were there just to fill in the space. It was crucial to make each issue interesting to both readers and advertisers, so we spread important features throughout the issues.

Iterative delivery in publishing works because customer expectations are consistently met on a quantitative level. Hard repetitive deadlines and fixed scope reinforce each other: features are replaced to achieve deadlines, but rescheduling features is not a problem since everyone knows that the next delivery will also will be on time.

Quantitative scope can be defined in a similar way for software projects - for example, each release could include two important upgrades and twenty less important new features. If the releases were frequent enough, and reliable, I am sure that most clients would agree to have features delivered through that pipeline.

The scope limit also works in the other direction - if we had more than 164 pages prepared for a single issue, some features would be thrown out. They were stored in the famous ‘first-aid kit’ box, waiting to be published in the next issue. The first-aid kit saved us from unexpected problems. In programming terms, we had a write-buffer between the production and the release. The software equivalent would be to defer deploying features, keeping some for the rainy days. A certain pack of features would be delivered each month, and ‘first-aid kit’ items could be used to fill the gaps if the team was under-staffed or if some unexpected problems caused delays.

The key is in reaching reliability and consistency - giving the customers, marketing, sales and management a feeling of what they can expect, and making that promise come true. From that viewpoint, it was much better to publish 164 pages two months in a row than to publish 200 pages one month, and only 128 in the next issue. Our customers, sales or marketing did not know when we exceeded the quota, nor when we borrowed from the first-aid kit. We delivered what we promised, and that was important.

Publishing software

All this can also be achieved in software - by moving away from software production into software publishing. Each iteration would include a mix of important and less-important features, and would be delivered on time, giving customers the perception of predictability and consistency.

Applying the publishing model turns the typical software production upside down - in most software processes, features determine the release schedule, and release plans are known up-front. Software publishing would start with the release schedule and quantitative scope - list of features would be derived from that. We did not even have a release plan until half-way through the iteration in the magazine. About two weeks before the deadline, we created the first version of the release plan by selecting features that were complete or could be completed in time. The plan would then serve to focus the efforts - features that were not immediately required would get postponed, and people working on those features would help to finish articles that were included in the plan. While most software plans are based on inter-dependencies of modules, publishing software introduces one additional goal - making each release interesting. Important features should be spread so that each customer could see every release as something that adds value.

Key principles for turning iterative development into software publishing are:

  • Establish a reliable delivery cycle, with fixed dates (it seems that one month or three-month cycles are most realistic). Think about skipping one release during the summer, because of holidays.
  • Establish customer expectations about the structure and volume of releases (example: three big items, 10 small enhancements, performance improvements).
  • Spread important features throughout iterations. Note that this does not imply order of production - just the order of delivery.
  • Adjust the work threads to reach the goals - select a mix of hard and easy features for each iteration.
  • Half-way through each iteration, make a target ‘release plan’ for that iteration, listing all the features that should be included. Then focus on finishing, polishing and integrating those features.
  • If features get thrown out of the release plan, or new features have to be added to it, do not expand the quantitative scope. Replace big items with bit items, small features with small features, but do not change the total ‘volume’ of the release.
  • Form a first-aid kit of half-finished and complete features that were thrown out of the release plan or implemented above the quota, and include them in the next release.

The publishing model does not imply any underlying way of working, since it is mostly concerned with deployment and releases. However, it would naturally fit iterative processes much better. Longer projects are more suitable for publishing than shorter ones, since longer project can be divided into a lot of sequential iterations. Dynamically scheduling features would require a relatively modular structure, but could also be applied to maintenance phases of a stable product.

Challenges of software publishing

The biggest challenge to the publishing model is cycle planning. As with any change, it will take some time for the new way or working to stabilise - in XP terms, to measure the velocity. After a few iterations, a good release cycle should emerge - and it will become clear how much work can be achieved in a cycle, when should the the release plan be created, what is the cut-off point for features, and how much time is required for polishing. It took us four months to estimate this properly in the magazine, and I guess that it will take at least that much for software.

The second big challenge will probably be enforcing fixed date and scope, since we are not used to thinking about projects in those terms. Typical software iterations are declared in terms of features and plans, with target date just as an estimate that could be easily changed. In the publishing model, target date and scope define an iteration - the exact list of features is less important. Hard deadline and hard scope limit are the focus points for the iteration - so it is very important to enforce them.

The third big challenge would be to find the appropriate driving force - the equivalent of magazine editors. The responsibilities of a typical editor are divided in the software world among various senior positions. This article is already too big, so I will leave the topic of editors and their responsibilities for some other time.

Great Expectations

Instead of specifying the list of features and then struggling to hit the date, software publishing would work by specifying the date and quantity and then trying to reach the features. Predictability in terms of when and how much will be released would be built into the release process. I expect that moving closer to the publishing model could significantly improve the way we deliver software on longer projects. Achieving predictability and consistency should allow us to make more reliable long term plans. Having fixed scope and delivery dates should serve as a good negotiation platform between programmers, project managers, sales and management. Spreading key features throughout iterations will give the customers a perception that they are getting something important with each release. Combined with a reliable delivery sequence, that should make our customers more satisfied and make negotiations about last-minute features much easier, since they will know that the next delivery can be reliably expected. It should also allow easier release planning, and internal resource scheduling.

I have used the ‘write-buffer’ on several longer projects over the last two years, with very good results, but have never thought of the whole release process in publishing terms. Of course, as Snepscheut put it, ‘In theory, there is no difference between theory and practice. But, in practice, there is’. This publishing paradigm may sound nice on paper (or on the screen), but it will take a few published projects to actually prove that it works. The idea sounds very promising, so I plan to test this model and see if it will live up to the promises.