Without taking a position on who’s right or wrong in this case, I’m very interested in how this whole thing is going to play out. A recent major rewrite of their flagship web site is one of the most publicised apparently successful IT projects in the UK.

Amazon uses them as a case study for cloud computing.

Phil Willis spoke at several conferences, including Qcon London last year, on how applying domain driven design on this project helped domain experts to get involved in software development, and how they maintained a deep, malleable domain model, whilst meeting deadlines. This is one of the key case studies used by the Domain Driven Design community to prove that DDD works.

ThoughtWorks use them as a reference of how they help big companies implement agile processes. On the back of that project, they got to do the same with AutoTrader, ran by a Guardian subsidiary Trader Media. Their web site quotes Tom Turcan, General Manager for Digital: “A multi-million pound project running on time, and absorbing growth in scope – remarkable”.

A common thread here is that DDD, clouds, agile apparently gave better service more efficiently and produced more business value for the same investment. If the Guardian News Media board is now thinking of pulling the plug on all that, then that is seriously casting a shadow on those claims. Cloud computing aside, both Domain Driven Design and Agile development rely heavily on on-site close collaboration of business users and IT development teams. I guess a multi-million pound project running on time, and absorbing growth in scope, is less remarkable than the money they expect to save by sending the jobs abroad. Or maybe this is a confirmation that their new web site runs on its own and doesn’t need that many people to maintain it.

Update (2:30 PM) The Register picked up the story this morning as well

“We need to start capturing the intent of the user. Our domain is focused on behaviours. Users should send back commands, not DTOs. The server needs to be told to do something.”, said Young, advising developers not to use editable data grids in user interface, because they are data oriented. Users should specify the intent, not edit data, said he.

“Actually figuring out what your users do and why they do it will cost money. But once you have done it, it will give you a better architecture and keep you at the same cost – or much much lower. Even at the same cost, we can get a lot of benefit.”, said Young, and suggested CQRS and Event Sourcing as the key patterns to support that.

CQRS

CQRS is based on command-query separation by Bertrand Meyer but evolved. It fundamentally requires splitting apart queries (read operations) and commands (transactions). “Treating them as separate concepts forces us to think that there are different needs for different sides”, said Young.

When we build up DTOs, we have a structural view of our data. When we process a transaction, we have a much more behavioural view. The conceptual structures are actually using completely different boundaries in these situations. So teams often come up with a half-way answer, which leads to a system that is neither behaviourally or structurally optimal. Young said that CQRS enables us not to make these trade-off s any more.

Instead of having a domain layer, we can have a thin read layer for DTOs for queries. This makes it a lot easier to optimise queries compared to using O/R mapping systems. The only knowledge people need to have to write efficient queries is a data model – not the O/R mapping technology, domain models and so on.

Separating out the reads also cleans up a lot of things in the domain layer. The repositories no longer have lots of different read methods, but only those required for particular domain. We don’t have setters on domain objects – they become purely behavioural. “Domain objects are not property buckets, they expose behaviour. We can specialise our domain layer to process transactions. The code will be clearer and the aggregate boundaries will be a lot stronger”, said Young.

Event Sourcing

Once commands and queries are separated, we can look at whether we need to use the same data source at all. When the same data source is used for transaction processing and querying, the third relational normal form is often used for both things. This requires teams to create complex queries. Splitting data sources would enable us to specialise the query data source for reading, having a first normal form model. The two data sources can then be synchronised.

To avoid inconsistency caused by this synchronisation, we need to build from one source of truth. A good option is to use domain events – notifications about what happened as the result of commands being processed.

The data model for processing transactions can also be write-only. “This is how the business thinks about changes”, said Young, “You can’t refuse to record something that happened”. We can take this even further and not have the current state of the system in the transactional storage. Understanding this enables us to restructure the transactional processing data model to just accept results of command processing. If we unify the models for transaction processing and storage, this cuts down the cost of development and maintenance.

Recording events that resulted from transactions ensures that we don’t lose any information but we can still recreate the current state when needed. This makes auditing trivial and enables a number of other things, such as replay the state of system at any time to troubleshoot or test and extracting analytics in the future that we haven’t thought about while developing the system.

Having a system structured like this requires the domain model to only to know about events. This simplifies testing – both checking for expected results and unintended consequences. Testing events would check both at the same time – any unintended change would generate more events. Integration with other systems also becomes a lot easier, according to Young, because this naturally flows around events and commands.

The event storage can also act as a queue to feed the query storage, and essentially require only a single transactional commit from the system while processing commands. This makes the system even more performant.

See other news from the DDD Exchange 2010

Related post: Two data streams for a happy web site

Tiers aren’t layers

When applying DDD, many people have in mind a classic three layer architecture, said Dahan, and there is a common assumption that the architecture layers are the same as tiers of deployment. For example, anything that’s in the business domain layer is not going to be deployed to the client tier. According to Dahan, this is the biggest mistake teams make when applying domain driven design.

A layered model is useful to tell us not to mix user interface logic and business logic or business logic and databases, said he, but the model says nothing about deployment tiers. “Starting with an assumption that layers are equal to tiers straight-jackets our implementation and colours our impression of the model thinking that we have to have data transfer objects going across tiers. If they weren’t in the different tiers, we would not need data transfer objects”, said Dahan, continuing that “almost any project I’ve seen that said that they are doing DDD made this assumption. This constrains choices and causes too many problems that aren’t related to DDD. “

Expecting to have a single architectural model for everything, teams don’t understand that they can deploy the same components to multiple tiers. Things like validation are a common cause of confusion. This causes a lot of architectural problems, that teams work around by putting in solutions such as caches. But according to Dahan, caches are often a sign that teams assume that there is a single linear layered architecture, which then causes performance problems. Many of our misunderstandings and problems we get ourselves into are related to this mix-up – layers, tiers, and what is a business rule.

Logical and physical connections are different

Another big issue with implementing DDD today is that there is an overemphasis on reuse in code. “Reuse is drilled into us in the universities and work. Even though this doesn’t reflect what we see in practice this drives us to centralise the code. Preferably in the domain model and then reuse it anywhere. Too much reuse creates an unmaintainable mess, because the system becomes hard to understand and has lots of dependencies.”, said Dahan. “When you think about domain models, business logic, validation – be very careful about how you treat it and not projecting your ideas of reuse”, advised he.

When talking about tiers, understand that you can get the exact same component, put it on another tier, and not create a dependency. The problem is when a single logical element of reuse became equal to a single physical place. This leads to lots of service calls – if it’s logically in one place it has to be physically in one place, so lots of remote calls have to happen. Equalling logical and physical locations causes too much complexity. Dahan suggested an alternative: “We want to remove these original blinders. You are allowed to deploy business logic to the client tier – that doesn’t mean you mixed business logic and user interfaces. It’s OK to take the same validation component you use on the server side and deploy on the client side. It doesn’t have to be in the same place to be logically cohesive”.

Teams also need to understand that there are different models, said Dahan, continuing “We need to understand what needs to be immediate, what means stale, and when the information can be stale, then addressing these situations with different models.”

All rules aren’t created equal

Assuming that all rules are equally likely to change is another common mistake that teams make, said Dahan. This leads to all the rules being put in the same place and the same kind of reuse applied to the same rules. Validations, calculations and workflows are not necessarily the same. For example, the requirement “the username must be unique” is common but never changes. Workflow rules (eg when a customer cancels an order, the system checks if it has been shipped and cancels if not) change substantially more. The reason for this is, according to Dahan, because these rules are actually related to a particular domain, what makes a business unique. “By virtue of the fact that all systems have the exact same requirements around username uniqueness and field length, that they are not unique for a particular business so they are not unique”, said he. These are just technical constraints, according to Dahan: “The business doesn’t care that username fields are 8 characters. Usernames have to be unique for technical reasons to be able to select customers to check details”. Rules that are not part of genuine domain logic do not have to be implemented in the domain model, suggested he, because they do not model the domain. They may be deployed to a separate tier.

Dahan quoted Martin Fowler, who defined the domain model pattern in Patterns of Enterprise Application Architecture (The Addison-Wesley Signature Series): “Use the domain model pattern if you have complicated and ever-changing rules…. If you have simple not null checks and a couple of sums to calculate, a Transaction Script is a better bet”.

Race conditions hide business rules

Another big problem when implementing DDD, according to Dahan, is that assuming the race conditions affect the domain. Giving an example of a possible race condition between the user cancelling an order at the same time as an operator asking the system to ship the same order. “As developers, we think that race conditions are business logic, so we need to write code for that”, said he, but businesses don’t understand race conditions. “We made them up. They didn’t exist in a business before we [software] got involved.” From a business perspective, a millisecond should not substantially change the domain. Businesses have existed long before computers, on paper, with concurrency problem windows not in milliseconds but in days and enormous chances for race conditions.

“But businesses still functioned. We ignore that and think that race conditions matter. “, said Dahan. Instead of solving race conditions with code, he suggested factoring time into design and talking to the business users about how the process was implemented before computers. For example, if the user cancels an order that was already shipped, charging the customer money unless they are of a particular status. The command does not fail because of a race condition, it extends to handle edge cases.

“If you think you have a race condition, you don’t understand the domain well enough. These rules didn’t exist in the age of paper, there is no reason for them to exist in the age of computers. When you have race conditions, go back to the business and find out actual rules”, advised Dahan

See other news from the DDD Exchange 2010

“Anything can be called agile, anything can be called SOA – when anything can be called that it’s no longer a useful term”, said Evans. So he wanted to offer a new, precise definition of DDD, which defies “what it is, what it isn’t, while still leaving enough space for innovation.”

Evans said that one way to define DDD is as a set of driving principles:

• Focus on the core domain — “people get distracted by technology and we want to bring that attention back to the business domain”, said Evans. Even that whole business domain is too much to focus, according to him, and DDD requires us to focus on the core, the critical, most valuable part.
• Explore models in a creative collaboration with domain practitioners and software practitioners – “we have to collaborate, not just quiz [business experts] for requirements”.
• Speak a Ubiquitous Language in an explicitly Bounded Context

Speaking about the focus on core domains, Evans said that that has to be something very specific.”We want to focus with collaborative cooperation on our core domain – we don’t want to give the same kind of attention to invoicing in most companies”, said Evans. Giving an example of eBay, he said that it is easy to assume that online auctions are the core domain but that would be wrong. He compared Amazon and eBay, and said that you can buy books in both places and they have similar features, but their core domains are very different. For eBay, it’s the rating of the seller – this is what makes eBay effective. “A star rating tells me that lots of people did business with a seller and I trust that. Developing trust between a buyer and a seller is a subdomain of online auctions, and their approach to building trust is part of the core domain of eBay. eBay would not have been today if they didn’t get that right”, said Evans.

Another way to define DDD, Evans said, is that it is a pattern language. It is a set of interrelated problem/solution pairs that have helped teams realise the principles. It also gives us a language, a vocabulary, that allows us to discuss domain modelling and design clearly. According to Evans, this is the key to enable innovation and keep the definition of DDD precise. There is a lot of innovation going on at the moment in the community, especially around architectural practices, and one of the key things for DDD in the future will be embracing those innovations. “I don’t want to say what the best way to do domain driven design is – others will present genuine advancements but none of us will be even able to understand it if we don’t speak the same language”, said Evans, continuing that he wants to hold on to the position of the final arbiter of the terminology used by the community.

The building blocks (entities, value objects…) aren’t really key to DDD, although most people initially thought that this is what DDD is about, said Evans. Having said that, he suggested that there is a lot of innovation in the community especially around aggregates, services and domain events, and that these concepts will be important for applying DDD in the future.

Related post: QCon London 2009: Eric Evans – What I’ve learned about DDD since the book

See other news from DDD Exchange 2010

To keep this practical, I’ll use a classic cargo example from the blue book – cargo is an entity here, itinerary and legs are value objects.

All three need to be persisted. By default, if you apply the rules of relational normal forms and classic ORM mapping of one class to one table, you get three different tables.

Option 1: three separate tables, normalised

And this is where the problem lies. Efficient database access often requires us to have a primary key so what could be the primary keys for itineraries and legs? It can’t be a group of attributes as they are not guaranteed to be unique. It shouldn’t be all the attributes together as well because database rows are inherently mutable. Value object pattern explicitly allows you to replace a leg in an itinerary without fearing that you’ll unknowingly change the itinerary of a completely unrelated cargo. So the remaining solution is to have an auto-generated surrogate primary key, effectively making this value object an entity with an ID.

But the fact that this has a database ID does not necessarily mean that we have to expose it in a Java or C# class or make the object mutable. The storage ID can be regarded as a purely technical address locator, very similar to the object ID in Java/C#, which effectively represents the memory location. The technical framework (JRE or CLR) uses this technical identifier to manipulate objects but class users typically don’t see or use this. You can encapsulate a technical ID that helps you manipulate the object in the database behind a public class interface so that class users don’t see it. Hibernate can also map a private field to a database identifier for you, and even work without you declaring an identifier at all (although that is not recommended).

This gives you a perception of a value object in object code, but it requires very careful management and makes the system quite error-prone. Using copy-construction through a public interface will disconnect the object from its technical ID so you might get more than one database row for the same thing. Also, not using a value any more doesn’t guarantee that it will be cleaned up from the database, so this approach might lead to some stale data in the database. With simple components (eg itinerary) you’ll get an error back if two rows exist where you expected one, but for collections (eg legs) it is going to be a lot more problematic. We expect the collection to come out of the database so if Hibernate doesn’t clean things up properly we’ll get both the old and the new elements next time the collection is loaded. As Hibernate manages collections with its specific implementations, adding or removing an element is fine but replacing the whole collection can lead to serious problems.

It probably isn’t the most efficient way to store value objects either.

Relational normal forms are introduced so that we can easily query, merge and update data, but that is not necessarily what we want here. Especially the global update.

Option 2: denormalised, relevant hierarchy

Itinerary doesn’t make sense without a cargo, at least the persistent one does not. And there is only one active itinerary for a cargo, so instead of having a separate table of itinerary properties we could just merge them into the cargo table. Legs could be kept separate, as there many legs per itinerary, but we could use the cargo identifier and the leg index as a composite primary key.

This can still be managed easily by an ORM. Hibernate has a concept of an embeddable component that doesn’t have an identity separate from the enclosing entity. Itinerary in this case is an example of that. As for the legs, you can easily map that into a collection of embeddable items and use exactly the indexing strategy that we chose here. See Hibernate documentation on components for more information on how to map this exactly. With Hibernate annotations you can use one-to-many join table without the entity on the other end to actually map this. In any case, it is easily doable and keeps your Java or .NET objects clean, although the database model gets a bit dirty.

This approach can be very tricky to implement properly with distributed systems. Hibernate will automatically keep track of objects added to and removed from collections and can cascade updates to the database, but this is done using Hibernate-specific subclasses for sets and lists. Copy the elements to a normal list and most of the functionality will be lost. So this approach requires less care and overhead than the first because the simple components (eg itinerary) are true value objects, but collections can still cause stale data and problems.

Option 3: clob/blob/string representation next to entities

Another option is to throw away the relational model altogether and go for an alternative approach, storing the entire object graph of objects in a column in the cargo. This approach has a distinct advantage of much more efficient database loading, as it avoids any hierarchical queries to retrieve legs and significantly reduces row locking when updating legs (this is very important for MySQL or SQL Server users where the entire leg table might get locked although you don’t expect it). There are many ways this can be accomplished – binary serialization is the simplest but least flexible. Binary format isn’t the most readable so the data in the database will be effectively useless until it is loaded in memory, so any other tools using the same data store such as business intelligence or data warehousing will be lost. There are generic XML serialization libraries such as Thoughtworks Xstream which can pick up a class and dump it into XML but I’ve burnt my fingers with this approach so I wouldn’t recommend it. As with the binary form, Xstream serialization is bound to a particular class version so if the class changes significantly in the future (eg renamed or removed fields), you simply won’t be able to load any older versions. Generic reflection based serialization also tends to be a real performance hog.

A good option here to store the value in a human readable string that allows you to convert both ways. One recent example I had was a spreadsheet-like price management system where traders would define variables and formulas for fields. The hierarchy was like this:

Formula is a typical value here. Storing things in a relational normal form would overcomplicate things so much that I don’t know where to start. A formula can consist of other formulas so we have a hierarchical recursive dependency with symbols, operators etc all participating so elements could span and be connected with different tables. In short, loading and updating a formula like this would cause a world of pain for a database, even the ones that can do hierarchical queries. Yet when you look at it from the outside, traders enter formulas by typing text into a cell, which then gets parsed and transformed in a complicated object graph. Why not put in just a bit more effort and make this conversion two-way, and then store a text representation of the formula in the database? Sometimes there is a natural string representation of a whole value object graph, sometimes you have to be a bit inventive, but it pays off big on the end.

For example, we could store the itinerary as a list of legs, one per line, with tabs between leg attributes. Or there might be a genuine business string representation of a leg, such as LGB-SEA. Watch out for these in your meetings with business domain experts.

A conversion such as this one could run pretty quickly and gives you probably more optimised storage and much more readable data, not mentioning avoiding class loading problems. And, as there is only one row to load, data loading works much quicker and we don’t have to worry about the dreaded N+1 problem of object traversal.

So which of these two options is better? Well that depends on the context, as always. If you don’t need to query efficiently for individual items in the list, then you don’t really get any benefits of data normalisation, so the last option is probably the best. If you do need to search efficiently for cargoes with a particular leg or moving through a particular point, then the second option is much better. With the third approach we also lose the chance for database to enforce relational integrity. Again, this might not be always required and even if it is, we can implement it in the application layer.

Another interesting solution is a hybrid approach where data is stored in both forms. When loading objects, we load it only from the primary table and construct the object graph from a string. When searching, we query the normalised table. This hybrid approach gives you great database performance for reading and searching while complicating updates just a bit. It introduces data redundancy, but the benefits outweigh the costs in most cases. You can even optimise normalised data for the particular queries that you’ll run. For example, I worked on a i18n project for a web site where we used phrases as domain concepts (as it makes no sense to translate words without a context). But we also broke down phrases into individual words for querying and updated both tables when saving a phrase. This hybrid approach can work really well and you can take it even further and retrieve value objects from a cache or internalise them for memory or performance optimisation. Hibernate Search uses the same idea and maintains a full-text search index automatically for you.

As a summary, here’s a few things to remember when persisting value objects

• Not everything that has a database ID is an entity
• Not everything has to come from the database
• Not everything has to be persisted and restored in the same form
• Not everything has to be persisted and restored

Image credits: http://www.flickr.com/photos/andresrueda/3027534098/ and http://www.flickr.com/photos/stephee/3492684318/.