Does Agile imply no design? Can the architecture evolve from doing the simplest thing possible? Is up front design good or bad? These questions, it seems, are still very controversial in the agile community. At the Architect Insight Conference in Newport yesterday, several presentations touched upon the topic of design and architecture in agile development.

While introducing agile templates for Visual Studio 2005, Nick Hines from ThoughWorks talked at length about ThoughtWorks Agile, a methodology which they use to develop software. ThoughtWorks is one of the best known supporters of agile methodologies, and their process is ‘not radical as XP, but incorporating XP practices such as TDD and Continuous integration’. According to the presentation, it is a mix between XP and Scrum, revolving around user stories. A user story life cycle in TW Agile has a few more stages than in similar methodologies, starting from New to Defined, Analysis Complete (also contains acceptance criteria for TDD), Development Complete, QA Complete and concluding with Signed Off (after business review). There was no mention of up-front design – the Story-finding workshop precedes planning, but it’s outputs are low-fi UI prototypes, state change diagrams and process maps, not system architecture or design. The presentation continued by explaining four key roles in TW agile: Analysts, Developers, QA and Project managers.

Introducing the Architect

So far, everything looked almost by the book, with a notable addition of the QA phase for stories. But the next slide was really controversial. It explained the responsibilities of an architect in an agile team:

  • 'Cutting' code - there are no Ivory towers, architects do not write code 24/7, but have to get involved with the code to understand the challenged.
  • Maintaining overall system architectural vision
  • Maintaining conformity with organisational standards
  • Having the final word on disputed design decisions
  • Overall responsibility for successful delivery of the project
  • Interacting with the customer and customer's IT staff as appropriate

Now this was very interesting. First, the architect has the ultimate responsibility for successful delivery, but architects were not mentioned in any part of the story-driven development. Is an agile architect then an outsider? It seems not, as the architect also has to cut code, to be involved. Next, the architect has to maintain overall system architectural vision, but there was no mention of design, visions or anything like that in the story life-cycle. Where does this vision come from? How does one develop it if stories are not discussed at length until it’s their turn for development? Is the agile architect a modern form of Deus ex machina that can somehow resolve problems with a clear vision out of the blue sky? Responsibility for the project delivery, and interfacing with customers, have traditionally been the roles of a project manager, so is the architect then the same person as a PM?

I talked to Nick Hines after his presentation about those concerns. He argued that there is an implied sence of some architectural vision (‘detailed enough’) throughout the project, and after the talk we pinpointed the Story-finding workshop as the time when that vision is formed, although that is not codified in the process. That is when all the technical people are in the room, and that is when the brainstorming happens.

The issues of design, story inter-relations and architecture have always been controversial in the agile community, and still are according to questions during several presentations on Architect Insight, and an on-going debate on the XP mailing list. Should design be done up front, how much, and where does it go in the classic story life-cycle? How does it get justified to the customer in a story-driven approach? When does it turn into BDUF, the big bad wolf of agile programming?

Even in XP, the System Metaphor allows for some architectural vision and up-front design. Yet some people seem very convinced that there should be no design, citing principles like ‘You Arent Gonna Need It’ and ‘Do The Simplest Thing That Could Possibly Work’. YAGNI page on C2 Wiki does not seem to justify such extreme views: ‘This doesn’t mean you should avoid building flexibility into your code. It means you shouldn’t over-engineer something based on what you think you might need later on.’, leading us back to the overall architectural vision - as such a vision is necessary to know where to be flexible.

My personal opinion is that some design is a good thing, and is definitely required for all projects except most trivial ones. That initial design is used to develop the vision which Nick Hines talked about, and identify main risks. It seems just irresponsible to ignore such issues, since they will effect the team, project, customers and users. It looks like J.R.R. Tolkien must have had similar problems when he wrote: ‘It will not do to leave a live dragon out of your plans if you live near one’.

Continuous design

If it is obvious that something will have to be refactored soon, does it pay off to do the simplest thing possible anyway? Or is it better to do the right thing straight away but keep it simple? Most developers I know will agree that if something will need to be refactored this week, we might as well do it properly straight away. If something will need to be refactored in two years, then to hell with it. But where do we draw the line? Is it one iteration, or two? What distance from the dragon is considered near?

This question leads us back to Architect Insight Conference, and a panel hosted by Ron Jacobs, with Ivar Jacobson, Jack Greenfield and Colin Bird. On the topic of evaluating software architects, the final argument was that a ‘good architect needs to spot refactoring risks and to know how much it will cost later to refactor’. Again, this architect figure somehow appears in an agile context, now effectively possessing the power to decide whether to do the simplest thing possible or to build more structure into the solution. Answering questions from the audience about up-front design, panellists somehow came to the conclusion that the architect should provide ‘enough design to keep the development going smooth, identifying the risks along the way’. To perform this continuous design, an architect should be one step ahead of the team, always providing other developers with enough insight so that they can implement the next set of stories.

Missing steps

It seems to me that there are a few more steps in the story life-cycle, hiding among typical descriptions of Agile methods. We always do a brainstorming session on the beginning of the project, and it seems that other organisations do a similar thing, but that step is somehow invisible, implied and never explicitly defined. Speaking about how to scale Scrum to multiple teams and large projects, Colin Bird mentioned a preparation phase (‘Sprint 0’) - maybe that is the time when the architectural vision is born.

It is, no doubt, very hard to describe exactly what goes on during that initial brainstorming and continuous design, and I guess not being able to describe it on paper is the reason why most of the agile literature just plainly ignores any notion of ‘architecture’. During his opening speech, Matt Deacon, Chief Architectural Advisor in Microsoft UK, urged conference delegates to participate in focus groups to define the types, capabilities, roles and responsibilities of architects in software development. If one of the biggest software companies in the world needs help on defining what an architect does, it’s no surprise that the agile community has trouble doing the same.

Ignoring architecture in agile literature, however, has a confusing effect - as all the heated debates show clearly that ‘people over process’ has been understood as ‘no process’ by a large group of people. Perhaps the reason for those typical agile/design debates is thinking about architecture and design in the waterfall sense - not really considering a continuous architecture and design, which obviously exists on larger agile projects.

XP and Scrum avoid having traditional PM or architect roles, yet someone has to take care of the responsibilities identified by Nick Hines. Someone needs to cater for that continuous design, as suggested by Colin Bird. Someone needs to decide when to cut corners and when to refactor. To perform the continuous design, an ‘Architect’ should be one step ahead of the rest, always providing other developers with enough insight so that they can implement the next set of stories.

So, it seems, there definitely is an Agile Architect role, responsible for the initial architectural vision and continuous design. Although this person might actually officially be a senior developer, the mix of skills and responsibilities would indicate that it is a full time job for one person, maybe even for two people if all the responsibilities identified by Nick Hines are considered. On the end, the titles are not really important – the only thing important is that the job gets done.