Although there was no central theme of the conference, web as an enterprise platform and scaling out systems seemed to be the topics that attracted most people. Apart from that, the conference had really strong DDD and agile development line-ups as well. Keynotes were a lot weaker, although the speakers were impressive and I probably expected too much from them. Sir C.A.R. Hoare opened the conference but his talk was a bit too academic for my taste, discussing how computer science and software engineering can help each-other. Martin Fowler talked about the US presidential campaign and how software helped Obama’s supporters reach out into the communities and help them organise better. This talk would probably be much more interesting for a US conference than a UK one.

I did two talks. One on Slim and the future of FitNesse (thanks a lot to Skills Matter for giving me a slot they sponsored to talk about something I’m passionate about) and one on using messaging to improve web architectures. With such a focus on Web, I did not really know how this talk would be received, since it was partially based on challenging the idea that web apps are a golden hammer that can be used to solve any problem. On the end, I got the feeling that people enjoyed it and got a few laughs as well, so I’m glad that I chose that for the subject.

Curiously, QCon in the UK attracts lots of US based speakers so this gave me a chance both to listen to some people I don’t usually meet and to catch up with lots of folks that I only knew online or met at Agile 2008. I decided not to go to Agile 2009, so I’m really grateful for the chance to meet colleagues from the other side of the Atlantic at this conference.

I really liked the venue, primarily because there were lots of power outlets everywhere, so I did not have to worry about recharging my laptop once the battery dies. Because of that, I was able to take notes in an electronic form and review the talks on my blog. I reviewed almost all the talks that I attended in the first two days. On the third day I mostly prepared the slides for my talk in the evening so I only reviewed one session, and published that a few days later. But in general, I published reviews within minutes after the talks ended (and once even before the question time finished). In addition to causing me to be called a monster, quite a few people thanked me for covering the conference so efficiently (1 2 3 4). I’m glad that you liked it and I hope you’ll keep coming back to this blog for more interesting stuff in the future. If you missed any of the reviews, you can find them here.

There were lots of WiFi hotspots with free access, but because people moved around a lot sometimes it was hard to find a hotspot with available DHCP leases, but in general this worked OK as well (as confirmed by the huge amount of tweets marked with #qcon). On another note, although there was a cloakroom on the ground floor, I spent the first day carrying my coat with me all the time, and I noticed lots of people doing the same during all three days, so if there’s one thing I’d suggest to the organisers for next year, it would be to either organise a cloakroom on the conference reception floor or clearly mark where the cloakroom is downstairs.

To my great surprise, Dan North (who started the whole BDD initiative) and one of the people I respect the most in the industry urged people to read my book Bridging the communication gap which was absolutely amazing. Thanks a lot for this, I really appreciate it!

Altogether, I really enjoyed the conference. Trifork & InfoQ – you did a fantastic job. Thanks a lot for inviting me to talk as well and I’m looking forward to the conference next year.

I covered QCon extensively on this blog. Click here to read the reviews and news from the conference.

A very interesting observation during the talk was that Twitter started up with a CMS model and that they gradually moved towards a messaging model. I’ve seen this in a few applications so far, including a casino system, where the messaging model seems to fit best an application intended to power massive community of online users, it seems regardless of what the application actually does business wise. Applications start out completely different, but then more and more functionality gets bolted on top of user messaging capabilities that the whole architecture on the end gets refactored to utilise the messaging channels as the core information transport. With Twitter, I’d expect this to be more obvious from the start as it was intended to help people notify each other.

For every incoming tweet, the messaging system gets notifications for all the folowers, which are then processed asynchronously. One of the most important changes they introduced to improve performance in the last nine months is moving from a Ruby messaging middleware to a custom build JVM-based messaging middleware written in Scala.

The shift from the CMS to the messaging model allowed them to rethink caching policies and shared work. Before the upgrades, web sites were going directly to the database and API services had a simple page cache. Unsurprisingly, the upgrade ended by the whole Twitter running from memory with a backing database used just as a data store to recover cache on demand. The interesting thing, however, was that all the upgrades were done live, without shutting down the system. The changes were always introduced to one node, then regression issues were sorted out, and then the software would be rolled out to the whole cluster. They went as far as building a whole messaging system based on memcached APIs in order to be able to slot in such changes.

The upgrades included adding three more caching layers: a write-through vector cache of primary tweet keys with 99% hit ratio, a write-through row cache for tweets and users with 95% hit rate and a read-through fragment cache with rendered versions of different tweets for different clients with 95% hit rate. All these caches are based on memcached. Some other interesting optimisations introduced are that cache vectors have limited size (800 tweets back) and that rebuilding the home page for a user gets done by reading through the caches of users that he follows. As lots of people are rebuilding timelines based on the same users, this works very good according to Weaver. Another important upgrade was rewriting the queue middleware from scratch (moving from Ruby to Scala/JVM) and moving from a Ruby memcached client to a C client with optimised hashing. All together, this change allowed them to increase the web server performance from 3.32 requests per second without caching to 139.03 requests per second. Weaver said that API services work about four times faster than the web, which means that the API performance is roughly 550 requests/s [my calculation, not given during the talk].

The frond-end is completely written in Ruby/Rails, which remained the same before and after the upgrades. The middleware is now written in a mix of C and Scala, where the most important components are memcached, varnish (cache), kestrel (messaging, written in Scala) and a Scala comet server. They use MySQL for the data storage, which has not changed during the upgrades.

Some things that surprised me from this talk are:

• Web is only 10-20% of the traffic, the rest is through API services
• Web servers are still 50% of the cluster.
• Regular incoming traffic peaks are around 80 tweets per second. I expected this to be a lot more.
• before the upgrades their web servers shipped only 3.32 requests per second!
• For each tweet, message gets inserted for each user which follows a tweet. In average, a user has 120 followers so this comes to about 9600 messages/s at peak times
• Message servers run on three nodes. They decided to write their own messaging software in order to make the protocol memcached-like, and did not evaluate other available solutions.
• During Obama’s inauguration, they peaked at about 350 tweets per second for around five minutes.
• They had a ton of problems with garbage collection but strangelly haven’t looked into JRockit RT or something similar that has predictable GC. Twitter JVM middleware runs on the SUN JVM

Evan published the slides for the talk on his blog, including the cache diagrams.

I covered QCon in detail on this blog. See other reviews and news from the conference.

I covered QCon in detail on this blog. See other reviews and news from the conference

I’m covering QCon in detail on this blog. See other news and reviews from the conference

Probably the most important infrastructural change is that from version 3.0, Spring will work only on Java 5+ platforms. The project layout will change slightly, moving to Maven style finer grained system of projects, which will be built using the the new Spring build system (“as known from Spring web flow 2.0″) which is OSGi based.

In terms of new features, Johnson pointed out the following innovations:

• Expression language for configuration
• Comprefensive REsT support
• support for Portlet 2.0
• Declarative model validation
• Early support for Java EE 6

Spring expression language

Spring expression parser is the most important new addition in version 3.0, and will be shipped as part of the org.springframrwork.expression package. The Spring expression language is compatible with Unified EL, but according to Johnson does a lot more. You can use it anywhere where you can write a bean definition, for example in the configuration files:

<property name="prop1" value="#{systemProperties.databaseName}" />
<property name="prop2" value="#{strategyBean.databaseName}" />

Expressions can reference global system properties or properties of other beans in the same context. It can also be used in annotations, for example:

class Bean {
@Value ("#systemProperties.favouriteColor}")
private String favouriteColor;
}

This also applies to method parameter annotations. In addition to beans and system properties, expressions can access various web context properties such as contextProperties, contextAttributes, request, session.

Rest support and MVC changes

Building on annotations in Spring MVC introduced in version 2.5, Spring MVC will get strong REST support. An example of exposing controller methods with REST is:

@RequestMapping(value=/show/{id}", method=GET)
public Reward show(@PathVariable("id") long id){
//...
}

This will work across different protocols (JSON, ATOM, XML). There will also be new options for handler method parameters, including @ReqestHeader and @CookieValue. People will also be able to use their own annotations and include that into the @RequestMapping support.

Release schedule

Johnson announced that milestone 3 will be released till the end of March, with a release candidate coming out in early May and the general availability release in June.

I’m covering the Qcon London 2009 conference in detail on this blog. Click here for other news and reviews from the conference