To keep this practical, I’ll use a classic cargo example from the blue book – cargo is an entity here, itinerary and legs are value objects.

All three need to be persisted. By default, if you apply the rules of relational normal forms and classic ORM mapping of one class to one table, you get three different tables.

Option 1: three separate tables, normalised

And this is where the problem lies. Efficient database access often requires us to have a primary key so what could be the primary keys for itineraries and legs? It can’t be a group of attributes as they are not guaranteed to be unique. It shouldn’t be all the attributes together as well because database rows are inherently mutable. Value object pattern explicitly allows you to replace a leg in an itinerary without fearing that you’ll unknowingly change the itinerary of a completely unrelated cargo. So the remaining solution is to have an auto-generated surrogate primary key, effectively making this value object an entity with an ID.

But the fact that this has a database ID does not necessarily mean that we have to expose it in a Java or C# class or make the object mutable. The storage ID can be regarded as a purely technical address locator, very similar to the object ID in Java/C#, which effectively represents the memory location. The technical framework (JRE or CLR) uses this technical identifier to manipulate objects but class users typically don’t see or use this. You can encapsulate a technical ID that helps you manipulate the object in the database behind a public class interface so that class users don’t see it. Hibernate can also map a private field to a database identifier for you, and even work without you declaring an identifier at all (although that is not recommended).

This gives you a perception of a value object in object code, but it requires very careful management and makes the system quite error-prone. Using copy-construction through a public interface will disconnect the object from its technical ID so you might get more than one database row for the same thing. Also, not using a value any more doesn’t guarantee that it will be cleaned up from the database, so this approach might lead to some stale data in the database. With simple components (eg itinerary) you’ll get an error back if two rows exist where you expected one, but for collections (eg legs) it is going to be a lot more problematic. We expect the collection to come out of the database so if Hibernate doesn’t clean things up properly we’ll get both the old and the new elements next time the collection is loaded. As Hibernate manages collections with its specific implementations, adding or removing an element is fine but replacing the whole collection can lead to serious problems.

It probably isn’t the most efficient way to store value objects either.

Relational normal forms are introduced so that we can easily query, merge and update data, but that is not necessarily what we want here. Especially the global update.

Option 2: denormalised, relevant hierarchy

Itinerary doesn’t make sense without a cargo, at least the persistent one does not. And there is only one active itinerary for a cargo, so instead of having a separate table of itinerary properties we could just merge them into the cargo table. Legs could be kept separate, as there many legs per itinerary, but we could use the cargo identifier and the leg index as a composite primary key.

This can still be managed easily by an ORM. Hibernate has a concept of an embeddable component that doesn’t have an identity separate from the enclosing entity. Itinerary in this case is an example of that. As for the legs, you can easily map that into a collection of embeddable items and use exactly the indexing strategy that we chose here. See Hibernate documentation on components for more information on how to map this exactly. With Hibernate annotations you can use one-to-many join table without the entity on the other end to actually map this. In any case, it is easily doable and keeps your Java or .NET objects clean, although the database model gets a bit dirty.

This approach can be very tricky to implement properly with distributed systems. Hibernate will automatically keep track of objects added to and removed from collections and can cascade updates to the database, but this is done using Hibernate-specific subclasses for sets and lists. Copy the elements to a normal list and most of the functionality will be lost. So this approach requires less care and overhead than the first because the simple components (eg itinerary) are true value objects, but collections can still cause stale data and problems.

Option 3: clob/blob/string representation next to entities

Another option is to throw away the relational model altogether and go for an alternative approach, storing the entire object graph of objects in a column in the cargo. This approach has a distinct advantage of much more efficient database loading, as it avoids any hierarchical queries to retrieve legs and significantly reduces row locking when updating legs (this is very important for MySQL or SQL Server users where the entire leg table might get locked although you don’t expect it). There are many ways this can be accomplished – binary serialization is the simplest but least flexible. Binary format isn’t the most readable so the data in the database will be effectively useless until it is loaded in memory, so any other tools using the same data store such as business intelligence or data warehousing will be lost. There are generic XML serialization libraries such as Thoughtworks Xstream which can pick up a class and dump it into XML but I’ve burnt my fingers with this approach so I wouldn’t recommend it. As with the binary form, Xstream serialization is bound to a particular class version so if the class changes significantly in the future (eg renamed or removed fields), you simply won’t be able to load any older versions. Generic reflection based serialization also tends to be a real performance hog.

A good option here to store the value in a human readable string that allows you to convert both ways. One recent example I had was a spreadsheet-like price management system where traders would define variables and formulas for fields. The hierarchy was like this:

Formula is a typical value here. Storing things in a relational normal form would overcomplicate things so much that I don’t know where to start. A formula can consist of other formulas so we have a hierarchical recursive dependency with symbols, operators etc all participating so elements could span and be connected with different tables. In short, loading and updating a formula like this would cause a world of pain for a database, even the ones that can do hierarchical queries. Yet when you look at it from the outside, traders enter formulas by typing text into a cell, which then gets parsed and transformed in a complicated object graph. Why not put in just a bit more effort and make this conversion two-way, and then store a text representation of the formula in the database? Sometimes there is a natural string representation of a whole value object graph, sometimes you have to be a bit inventive, but it pays off big on the end.

For example, we could store the itinerary as a list of legs, one per line, with tabs between leg attributes. Or there might be a genuine business string representation of a leg, such as LGB-SEA. Watch out for these in your meetings with business domain experts.

A conversion such as this one could run pretty quickly and gives you probably more optimised storage and much more readable data, not mentioning avoiding class loading problems. And, as there is only one row to load, data loading works much quicker and we don’t have to worry about the dreaded N+1 problem of object traversal.

So which of these two options is better? Well that depends on the context, as always. If you don’t need to query efficiently for individual items in the list, then you don’t really get any benefits of data normalisation, so the last option is probably the best. If you do need to search efficiently for cargoes with a particular leg or moving through a particular point, then the second option is much better. With the third approach we also lose the chance for database to enforce relational integrity. Again, this might not be always required and even if it is, we can implement it in the application layer.

Another interesting solution is a hybrid approach where data is stored in both forms. When loading objects, we load it only from the primary table and construct the object graph from a string. When searching, we query the normalised table. This hybrid approach gives you great database performance for reading and searching while complicating updates just a bit. It introduces data redundancy, but the benefits outweigh the costs in most cases. You can even optimise normalised data for the particular queries that you’ll run. For example, I worked on a i18n project for a web site where we used phrases as domain concepts (as it makes no sense to translate words without a context). But we also broke down phrases into individual words for querying and updated both tables when saving a phrase. This hybrid approach can work really well and you can take it even further and retrieve value objects from a cache or internalise them for memory or performance optimisation. Hibernate Search uses the same idea and maintains a full-text search index automatically for you.

As a summary, here’s a few things to remember when persisting value objects

• Not everything that has a database ID is an entity
• Not everything has to come from the database
• Not everything has to be persisted and restored in the same form
• Not everything has to be persisted and restored

Image credits: http://www.flickr.com/photos/andresrueda/3027534098/ and http://www.flickr.com/photos/stephee/3492684318/.

A few of us met up two weeks ago to continue the discussion from the AltNetUK 2009 conference on domain driven design beyond basic patterns and building blocks, or “the second half of the blue book” as Ian Cooper called it. Stuart Campbell raised the question of merging – should two similar projects be refactored and similar parts extracted into a shared kernel. The discussion that followed made it clear to me how dangerous is to consider this question without looking at the big picture from a business side (or again using the terms from the second half of the blue book, considering what really is the core domain and what makes the system worth writing).

This is again where the question ‘Why?’ should be raised before we discuss ‘How?’ and ‘What?’, something that should in my opinion be raised much more often on software projects. Merging common parts of similar projects and extracting a shared component makes a lot of sense, as it can reduce management costs and make changes to both projects easier to synchronize in the future. But that might actually be counter-productive.

If the end client or the company that is developing software for its own operations is in a fairly stable, low margin market where everyone delivers the same old service but their competitive advantage is being able to deliver it cheaper, then consolidating and merging makes sense. Losing a few bits of functionality or making it slower to deliver individual projects doesn’t really matter much, as the market and products should be fairly stable.

On the other hand, if the company is in an emerging innovative market, with high margins and competing on functionality, the overhead of maintenance for two separate projects might be negligible compared to potential gains of delivering faster and being more flexible with features. For a shared kernel to work, it has to be accompanied by serious change management procedures and a lot more integration testing. This introduces a significant overhead on implementing new features in that shared part. It also makes two previously independent projects reliant on the quality and delivery timelines of that shared part. Also, in a dynamic market, opportunities change and lots of code gets deprecated quickly – it is easier to throw it away if it does not belong to a “core” library. Very often, these “shared”, “util”, “api” or “core” libraries are essentially some kind of limbo where undead code lurks because nobody has the time or power to drop it.

The point I’m trying to make is that there is no right or wrong answer to the question of merging in general. In particular, cheaper maintenance is not always the competitive advantage of the business and the main business goal, although it seems as a good idea. Sometimes more expensive maintenance with more flexible delivery makes a lot more sense from a business perspective, especially for companies that claim to be innovators. Unfortunately, if this question is considered in isolation by the IT, it might actually turn out to shoot the business in the foot with the best of intentions.

I’ll explore several approaches to persisting value objects, present best practices and talk about common pitfalls for value object persistence.

For more information and to register see
http://skillsmatter.com/event/design-architecture/ddd-and-relational-databases-the-value-object-dilemma

Increasing visibility

A common topic in the discussion was increasing visibility, both the visibility of business value to developers and visibility of developer progress to business. We discussed different metrics to increase visibility of the progress, but this turned out to be quite a tricky subject. One interesting metric mentioned was cycle time, how much it takes to get a story from the backlog into production. This would require that stories have a fairly normalised size though. The conclusion for me is that metrics make it very easy to slide towards efficiency but not effectiveness (with the example of a hamster in a cage which might spin the wheel very efficiently, but going nowhere), which also lead us to the fact that teams should be aware where does the value come from and define what success looks like before a project starts. Ensuring that everyone knows the answers to these questions would significantly increase the value visibility and help people prioritise better.

Some interesting examples of what companies do to increase visibility were estimating technical debt with monetary value (how much will it cost to fix it) and assigning monetary value to stories (how much benefit will it bring) and implementation estimates (how much will it cost to make it). Although everyone agreed that getting these figures right would be a challenge, even rough estimates can help immensely to set the priorities and ensure that we’re building the thing of greatest value.

Another challenge mentioned is actually measuring how much value features bring when they are in production and do they justify the cost of maintenance. We all agreed that it is very hard to measure effects of individual features, but A-B testing was suggested as a way to to measure this. In A-B testing, a part of the system runs with a feature and a part of the system runs without it, and then the results are compared to determine the effectiveness of that particular feature. This approach has additional management overheads and risks involved though.

Waste

Unsurprisingly, the discussion often touched upon lean software development principles and looking at what causes waste in the process. Two good books mentioned about this were Implementing Lean Software Development: From Concept to Cash by Mary and Tom Poppendieck (ISBN 0321437381 Addison-Wesley 2006) and Agile Management For Software Engineering by David Anderson ( ISBN 0131424602, Prentice-Hall 2003). Anderson is also apparently working on a new book related to this subject. Additional suggested resources for Lean Software development included Lean Conference 2009 videos, David Anderson’s blog , Karl Scotland’s blog , David Joyce’s blog and the Agile in action Blog.

A common theme in the discussion on waste was building too much software, where people did not know whether they really needed it. As a possible solution for the problem we talked about the DDD core domain idea, where companies should focus on identifying a very small core of the system that makes it worth building, something that brings direct competitive advantage, and source the rest from external providers or buy off-the-shelf solutions. Core domain is explained more in Domain-driven Design: Tackling Complexity in the Heart of Software by Eric Evans (ISBN 0321125215, Addison-Wesley 2003).

Another source of waste was building “nice to have” features suggested by the business but not really bringing any value. As a possible solution for this, we talked about the Goals-Features-Requirements breakdown (a variant of which is Goals-Stories-Acceptance tests) where each feature introduced into the system has to be connected directly to a stated project goal, and each requirement has to be directly connected to a feature. If the person suggesting a requirement cannot name a feature (or likewise someone suggesting a feature cannot justify it by naming a goal), this gives us ground to push back and reject scope creep. This idea comes from The Art of Project Management by Scott Berkun (ISBN 0596007868 O’Reilly 2005).

The development counterpart of this is just-in-case code, something that developers decided to put in because they think it is needed. This was called the biggest source of waste by Mary Poppendieck in the Lean Software Development book. As possible solutions for this, we discussed specification workshops and agile acceptance testing – explained in my book Bridging the Communication Gap Specification By Example and Agile Acceptance Testing (ISBN 0955683610, Neuri 2009). As the answer to the problem ‘we know that we’ll need it soon’, we discussed a fine line between making the system flexible to accept a future change and actually building in flexibility. The former would involve isolating a piece of code that we know is going to change, the latter would involve actually building in a very flexible framework to start with. The conclusion was that very often these complex flexible frameworks don’t pay off as requirements might change and the requested flexibility never actually makes it into the product.

The fourth related cause of waste was working on a problem that people don’t know enough about and identifying that too late. Specification workshops are also a solution for that.

UI design and redesign was also perceived as a common source of waste, perhaps because all the people in the room were developers. Someone suggested Yahoo’s UI design pattern library as a good source of information to make this process quicker and easier.

Prioritisation

The last major theme of this discussion was prioritisation – how do we actually decide what to build first and how to deliver value quickest. Value visibility certainly helps, and assigning monetary value to features and costs of production might help prioritise. There was a concern though that only low-risk features would ever get implemented by this, but we concluded that this would probably be pushed back to the business and they would be able to decide how much risk they want to take.

We also discussed the difference between iterative and incremental development and how that helps with prioritisation – rather than asking customers to prioritise between a car door and an engine, for example, we could ask them to prioritise between a car with a poor door system but a great engine and a car with great doors and a poor engine, delivering a car that they could use in any case. This is explained in a lot more detail by Jeff Patton (also see Alistair Cockburn’s response and my write-up of Patton’s keynote at XPDay 07).

Having a map of goals to stories and tests makes it easier to prioritise as well, because business should be able to prioritise goals and we can derive priorities of lower levels based on that.

Here’s the video of my presentation titled ‘DDD in a distributed world’ from the DDD Exchange 09 conference in London last month. In the presentation, I discuss how domain driven design aggregates help build distributed systems that work better and solve latency, serialization and distribution problems. You can also download the slides and read a shorter version of this presentation.