Before I start, I’d like to point out that judging from this post it might seem that I have a negative view of cloud deployments, but nothing could be further from the truth. I have many nice things to say about the cloud, but lots of other presenters at CloudCamp do that all the time. I wanted to play the devil’s advocate a bit and expose some of the things that you won’t necessarily find in marketing materials.

First fundamental rule of cloud deployment: No single machine on the cloud is going to be any more reliable than any other machine there

Before the cloud, I was used to investing more in machines which were more important. Database boxes would have better power supplies than web servers, ideally redundant. Content servers got better disks and lots of them. A nice Cisco appliance would balance requests to web servers, and was infinitely more reliable than them. Web servers, for all I cared, could crash and burn at any time, as long as they did not all decide to do it at the same time. With the cloud, this isn’t possible. No matter how many virtual cores or memory you rent, all the boxes are running on very similar hardware. Or, putting it in another way:

Cynical version of the first rule: All your cloud boxes are equally unreliable

A very healthy way to look at this is that all your cloud applications will run on a bunch of cheap web servers. It’s healthy because planning for that in advance will help you keep your mental health when glitches occur, and it will also force you to design for machine failure upfront making the system more resilient.

A few months ago, one of the load balancing appliance providers tried to pitch their software for the cloud. Choose a box to act as a load balancer, install their software, and hey presto you have the equivalent of a balancer appliance in the cloud. Yeah, right. That machine is as probable to go down as any web server, and when it does your entire server cluster will be cut off from the Internet.

Second fundamental rule of cloud deployments: All machines will be affected by the same networking and IO constraints

Amazon lets people get pretty big virtual boxes in terms of processor or memory power. However, IO and networking are a completely different issue. For in-house data grid deployments, getting a separate set of network cards and putting them on a dedicated VLAN or even their own switch is a really good idea, because of the broadcast traffic between the nodes. You can’t do that on the cloud. Putting a card with hardware TCP offloading is not an option (and broadcast is also not an option at least on Amazon, but that’s another story). So the architecture has to work around this. Bottlenecks can’t be just solved by getting better hardware. Beware of this while designing an architecture that depends on all traffic going through a single file server, database machine or load balancer. If all the traffic goes through a single point, the entire capacity of the cluster will be limited to that machine’s IO or network constraints (which is probably shared with who knows how many other virtual machines on the same physical box).

Third fundamental rule of cloud deployment: networking is unreliable

When I started using the cloud, it took me a while to fully grasp the fact that it’s a really really big data centre. Even if you get only two machines there, they still run in a huge data centre, can be separated with lots of cable and switches and might not even be in the same building. That introduces a new level of complexity that I simply did not have to deal with earlier. It’s not often that I had problems with the network in the cloud, but strange things do happen. I watched my grid control machine disappear from the network and come back in ten minutes without any idea what happened. A few months ago I was completely puzzled by the fact that a web server can’t talk to an application server, although I was connected to both of them at that same time. They were working fine, but they simply did not see each other for a while.

Murphy’s law guarantees that these issues, although rare, will occur at the most inconvenient moment. So choose infrastructure and clustering software wisely. I used an opensource messaging system that worked perfectly inhouse, but the cluster simply couldn’t recover from network glitches in the cloud. Put in lots of monitoring scripts to check if machines can see each other, and plan for parts of the cluster being inaccessible for short periods of time.

There is no fast shared storage

All these things together contribute to the last big challenge I had to get my head around. There simply is no equivalent of a nice network attached shared storage on Amazon. If you’re planning to use a hot-standby database by sharing data files, think again. Elastic Block Clouds can only be mounted to a single machine. You can get a machine and expose an elastic block using NFS, but that falls under the effects of all three rules explained above. SimpleDB is slow. PersistentFS allows you to mount S3 as a file system to multiple machines, but read-only. There is no good solution for this, apart from designing around it.

How to keep your sanity

It took me a while to understand that just deploying the same old applications in the way I was used to isn’t going to work that well on the cloud. To get the most out of cloud deployments, applications have to be designed up-front for massive networks and running on cheap unstable web boxes. But I think that is actually a good thing. Designing to work around those constraints makes applications much better – faster, easier to scale, cheaper to operate. Asynchronous persistence can significantly improve performance but I never thought about that before deploying to the cloud and running into IO issues. Data partitioning and replication make applications scale better and work faster. Sections of the system that can work even if they can’t see other sections help provide a better service to customers. This also makes the systems easier to deploy, because you can do one section at a time.

To conclude, there are three key ideas to keep in mind:

• Partition, partition, partition: avoid funnels or single points of failure. Remember that all you have is a bunch of cheap web servers with poor IO. This will prevent bottlenecks and scoring an own-goal by designing a denial of service attack in the system yourself.
• Plan on resources not being there for short periods of time. Break the system apart into pieces that work together, but can keep working in isolation at least for several minutes. This will help make the system resilient to networking issues and help with deployment.
• Plan on any machine going down at any time. Build in mechanisms for automated recovery and reconfiguration of the cluster. We accept failure in hardware as a fact of life – that’s why people buy database servers with redundant disks and power supplies, and buy them in pairs. Designing applications for cloud deployment simply makes us accept this as a fact with software as well.

Mark Cusack from Clearspace Software suggested clouds as a very good solution for data retirement, storing massive amounts of data that is almost at the end of its business life cycle. This kind of data is read-only and not queried as often as the one the business is working at the moment, but it still needs to be kept for analytical or regulatory purposes. Instead of keeping it in their data centres and wasting hardware and effort to maintain and retrieve it, companies can ship this off to the cloud and reduce costs. Key ideas to make this work are, according to Cusack, compressing the data before moving (and saving on bandwidth transfer costs) and querying the data on the cloud (not retrieving it back for analytics). As the data is compressed, we can make multiple copies and distribute it to multiple clouds for availability. Clouds have an advantage here as well as by compression we convert a typical IO-bound problem (storage access) to a CPU-bound problem (decompression) which allows us to use cloud resources efficiently. To be able to query this on the cloud and still keep it safe, Cusack suggested encrypting network pathways and data rest-points for queries, employing tamper-proofing and auditing with message digests. Although not ideal, Cusack said that sending the encryption key to the cloud only to execute a query and not persisting it anywhere might be a viable solution for a lot of companies, as it reduces the risk window to only the time when a query is executed.

Miranda Mowbray from HP presented an alternative solution for data security in the cloud – obfuscating the data for privacy. Her solution is in a prototype stage and involves encrypting the data so that it preserves the structure, and applications can still work with data directly. For example, dates can be shifted by an amount, customer names can be replaced with encrypted versions etc. The goal is not to have clear-text sensitive information on the cloud at all. In order to query the data, the query can be obfuscated as well and work directly on the data, with obfuscation software decrypting it on the fly as it comes back from the cloud. Mowbray said that this approach might not be secure against all attacks and practical for all applications, but that all but 2.5% of all reported privacy incidents happen on non-obfuscated data so obfuscation significantly reduces the risk.

Anish Mohammed from CapGemini then talked about clouds and security from an evolutionary perspective. As key challenges on the clouds today, he pointed out trust and managing multi-tenancy. Also, according to Mohammed, what matters most for survival from an evolutionary perspective is adaptability, flexibility and cost effectiveness/efficiency. Comparing huge in-house data centres as dinosaurs that are strong but inflexible and cloud deployments with mammals which are not that strong or secure but very flexible, Mohammed said that mammals are coming out of the woods and that there is a trade-off between security and usability, as security has computational costs, and that the ecosystem would on the end define security restrictions.

Phil Wainewright from ZDNet attacked the idea of private or hybrid clouds (a very popular topic at CloudCamp London 2). Arguing that private clouds are a bad idea, Wainewright said that “Connected clouds are the stuff of the future – captive clouds live behind the firewall and should not be allowed”. According to Wainewright, three key principles of cloud computing are abstracting horizontal elements to an API, cloud-scaling and being connected. In such a scenario, any improvement to abstracted services is instantly available to everyone. Custom private cloud architecture is going to start to miss out on improvements at some point. Connected clouds also benefit from community aggregation, as they have more users which means better tested infrastructure with more relevant benchmarks. Multi-tenant connections force it to share share connections as well as code, providing open APIS and mediated integration. Wainewright said that the future belongs to the “fitmost” (explained as most connected, not fittest individually), because openness to connectivity allows agile adoption of new resources and better mobility, concluding that “the more open you are to the cloud, the more easily you can connect to what’s out there.”

The meeting was again organised as a set of lightning talks followed by a few open space discussions, with Alexis Richardson working as a host and keeping track of time. The length of lightning talks was halved to five minutes, but Simon Wardley again achieved an amazing feat – going through seventy slides in just five minutes. (At the first CloudCamp in London, he ran through a hundred slides in ten minutes). His talk was again a pleasure to watch, although it was mostly centred around the same issues as the last time: standards and avoiding a vendor lock-in. Trying to describe the risks involved in a vendor lock-in, he talked about a scenario when one company buys a huge number of machines, another provides a cloud on top of that, the third provides virtualisation to split the cloud into a number of “small” machines, fourth builds applications on top of that, fifth provides services based on those applications and so on. If the company that owns the hardware goes bust, the whole pyramid will collapse, affecting thousands of small companies that actually run applications on the fifth and sixth levels. He could not resist making a joke about sub-prime mortgage market, comparing this situation to one bank having a lot of houses, another packaging that into a financial instrument, third splitting that into lots of small instruments and so on. As a way to avoid system failure, Wardley again asked for more standardisation between vendors.

Joe Baguley from Quest discussed energy efficiency in a talk titled Virtualisation isn’t always green. He challenged the presumed model of virtualisation on blade servers as a way to save energy, showing research data which pointed out that an average server spends about 200 Watts when idle and about 300 Watts under full load (I did not get the exact figures from his slides, this is my interpretation). He also presented a comparison of several options to run an application. One hundred servers running a single application consumed 1MW, virtualised system built on top of 15 hosts consumed 260KW and 10 HPC grid servers consumed 150KW, making a grid much more energy efficient than a virualised cluster. There was not enough time to go into the details of this comparison, so I could not catch the actual source of the research and I am not really sure about the underlying measurements or whether this was really a like-for-like comparison. In any case, Baguley suggested that blade servers are not the answer for energy efficiency and called on companies to consolidate existing infrastructures and reuse them.

Paul Watson from Newcastle University then presented on the idea of Federated clouds. Most of the interest in cloud computing today centres around public clouds, but big companies have huge IT departments which can be used to gain similar benefits as well. Watson suggested an example of two departments in the same company sharing their infrastructure on demand based on cloud computing technologies, optionally scaling out to public clouds if there is no more internal capacity available. Watson’s vision of the future is that there is no single cloud but a multitude of private and public clouds and that the key challenge will be how to federate them together. He suggested service-level agreements as the key to a successful cloud federation, including scaling out to the infrastructure of other internal departments and scaling out to public providers. Watson then offered Arjuna Agility as a software solution for that challenge.

Duncan Johnson Watt from Enigmatec talked about Cloud-cover, the idea of using a public cloud for disaster recovery. He likened this idea to insurance policies, where companies would pay for reserving the capacity of a cloud for disaster recovery. Cloudcover would provide management of that, activating the cloud capacity or rolling back to normal infrastructure when the main data centre is online again. It seems as an interesting product, although the presentation was fairly ineffective. Watt used a pre-recorded video which did not really work as expected and did not fit into the rest of the talks.

Phil Dean, architecture manager at Cisco, presented a summary of discussions his group had with CIOs of various companies in a talk titled What CIOs are telling Cisco about the cloud. According to Dean, CIOs like the potential of providing a consistent service across the globe, service orientation and the potential to simplify services. On the other hand, CIOs were mostly concerned about the loss of control, business continuity and security strategies, migration and hybrid operation and the fact that clouds are today focused on technology, not on the business. As a key thing which clouds need to be more attractive to Cisco’s customers, Dean suggested a better a business focus: “put a business layer around the cloud to give it a meaning for your CIO “. According to Dean, CIOs envisage having a concept of a business broker which can select one of a number of available services in the cloud to get the best result. Other suggestions included strategies to cope with legacy system integrations, private services and upcoming private clouds. He gave an example of Gartner, which according to Dean have a private cloud growing fast.

Philipp Huber from Zimory then talked about Pervasive cloud computing. He focused on a common comparison between energy utility and cloud services, trying to answer why does the energy market run so smoothly and comparing that to today’s cloud market. As the reasons why the energy market functions well, he listed multi-tier supply (different companies are energy producers, distributors and resellers), having more than a century of development to get where we are today, demand, customers that are well educated about the offering and well established standards. In today’s cloud computing, Huber argues that there is a lot of supply with new cloud providers emerging every day, but the technology is still new and everything happens in a frenzy. The demand is, according to Huber, constantly increasing although the technology is still predominantly used by early adopters, while the wider public still has major concerns about security and stability. Huber pointed out that standardisation is probably the weakest point in today’s cloud market and that full interoperability is “still in the clouds”, as it works against the interests of individual cloud providers. As ways forward, he suggested improving security and compliance, making the billing models more unified and transparent and introducing standardised service level agreements. As some of the things that cloud users want, he suggested hybrid cloud models which would allow people to get the best of both worlds and seamlessly moving between public and private clouds.

Rhys Johnes from RBS then presented a talk titled Clouds are cool, so why aren’t we using them yet? focusing on opinions of banking IT sectors about cloud and especially why banks have still not started adopting clouds. He said that cultural change is the biggest issue today, with the requirement to give up control as a major hurdle to cloud adoption. Unlike a lot of other presenters who were enthusiastic about the idea of companies building private clouds, Jones thinks that internal clouds are unlikely as they don’t provide the same flexibility as public clouds. As a key advantage of clouds over private infrastructure, Jones pointed out that you can easily scale down with a cloud.

Wayne Horkan, CTO for UK and Ireland at Sun Microsystems, discussed the surge in data centre infrastructure capacity. In a talk titled The global cloud infrastructure build out, he said that datacentre buildouts are getting larger and larger and envisaged a massive adoption of clouds in the near future. Horkan compared the process to the Internet adoption: large corporations were forced to go online because of small competitors who were more flexible, moved online first and started stealing their business. Talking about the current cloud landscape, Horkan cited a research by IDG stating that the biggest benefits of cloud/on demand infrastructure are still easy test and deployment of applications. Citing Highscalability.com, he said that most companies run some form of opensource software wired together. Sun’s big for the cloud will focus on offering software which enables customers to run “compatible services”, and Horkan gave examples of MySQL and OpenSolaris which have versions adopted for the cloud. He then started advertising Network.com, Sun’s cloud service, but was interrupted as his five minutes were up.

Neil Hutson, senior director at Microsoft briefly talked about Microsoft’s cloud offering announced earlier this month at the PDC conference. In a talk titled Head in the clouds, feet on the ground, he said that Microsoft already has lots of infrastructure to run global applications such as Hotmail and Live search, but did not allow other companies to utilise that infrastructure until now. Windows Azure is Microsoft’s “operating system in the cloud”, offering virtualised computating environment based on windows server, durable storage and a range of application services. He again suggested that people might want to combine cloud and private infrastructures, giving an example of using the cloud as just data storage.

The meeting continued with open space sessions. One was focused on persistence in the cloud, one was focused on Microsoft’s cloud offering and one was focused on standardisation and interoperability. In my opinion, this was not organised as well as it should have been. The slot for sessions was short and we did not have enough time to start a meaningful discussion on persistence. I’m not sure how the other sessions got along. I could not help overhearing quite a lot of the discussion in the other session held in the same room. For the next time, I would really suggest separating these into different rooms and giving people more time for the discussions.

All in all, it was a very enjoyable evening and I think that events like these are crucial to help the cloud/grid industry get a wider adoption, so I am looking forward to the next cloud camp. The event was filmed, so all the videos will probably appear online soon on the Skills Matter web site.

Simon Wardley’s presentation on the trends in cloud computing was very interesting, both in terms of content and presentation style. He ran through a bit more than 100 slides in 10 minutes, but all those slides were photos or movie screen grabs that added a bit of humour to the talk and made it even more interesting to watch. Simon compared IT infrastructure to the invention and adoption of mass-distributed electricity, claiming that the recent events in the IT industry signaled a shift from a product based to service based economy.

He then pointed out that vendor lock-in, competitive pricing and being left with no choice to migrate are the challenges that we’ll need to face in the future. What happens if the infrastructure provider goes bust? In-house APIs and standards more or less force us to develop applications for individual provider environments, so migration today could host a lot of time and money. Simon suggested that the users should form some sort of a syndicate and demand that vendors start offering a uniform standardised service. His idea is to mitigate risk with compatibility and interoperability between providers, using use opensource and open standards, and making providers compete on price and service rather than on products. He also mentioned Eucalyptus as a possible common standard for the future. I never heard of that product before, so I definitely plan to look into that now.

There were three or four other talks by commercial vendors trying to plug their service which I did not find especially interesting. The only important fact I took from that part was that Amazon is building an infrastructure in Europe, which might make it much more interesting for the things that I’m involved with as US companies generally will not touch anything related to betting or gambling.

The last talk by Alan Williamson was the most interesting for me. He was from a company that, if I understood correctly, provides web site caching for big media companies. Alan’s talk was focused on problems that they experienced when moving to a cloud infrastructure, and how things are not as nice and clean as the providers would want us to think. From his experiences, it looks to me that the clouds are still only for early adopters, not ready for mass production. His main message was that with cloud infrastructures problems don’t magically go away, they just shift. You don’t have scalability or storage problems any more, but you need constantly monitor the cloud and your application in it. Alan pointed out examples when Amazon’s cloud failed and their applications got cut off from the Internet. As a solution, he proposed deploying the application on more than one cloud so that you have resilience. This requires writing the application in a way that can be easily ported to different providers, which in itself might be a challenge. One idea that was really striking was their analysis of getting off the cloud to a dedicated infrastructure again — apparently it would take them about three weeks of full-bandwidth transfer to download the data that they have in the cloud, making it virtually impossible to go back.

Adil Mohammed from Entrip pointed out an interesting example of Animoto, which grew from 25000 users to 250000 users in three days, scaling from 50 to 4000 servers in that time and growing at peak 20000 users per hour. The cloud deployment made it possible to do that, since growing that fast on a dedicated infrastructure would simply be impossible even if already purchased the hardware.

However, from my point of view the clouds are still not for production. Most of the companies I work with have to keep their data in-house for legal reasons, sometimes even to process it in-house. But clouds and on-demand infrastructure may be very interesting for development and testing. Instead of waiting three weeks for new hardware to come in for a stress test, we can get a few systems instantly and run the tests. On-demand infrastructure may be interesting for heavier builds or grid UI testing. At the moment, I’m working on a way to split a bunch of selenium tests that run for thirty minutes across ten or twenty boxes so that they results come back quicker. Instead of actually buying the hardware, we might just get it from a cloud.

Here are some interesting links from the conference:

• Simon Wardley, Gang Up Now before the *aas cloud gets you: video
• William Fellows, Partly Cloudy: video
• Adil Mohammed, Start-ups in the cloud: video and slides
• Alan Williamson, Pick’n'Mix, Bridging the clouds: video