Speaking of Clouds

We’re going into the second day of the OpenStack Summit here in Portland, and for me the most important item on the agenda is the last session of the day: a panel on interoperability. Several leading members of the OpenStack community have expressed their concerns about the risk of divergence between different OpenStack-based services. Josh McKenty of Piston has been very vocal on the topic, and is proposing the idea of a RefStack which could be used for compliance.

To those of us that worked on Java in the early days, this feels very familiar. Welcome to the JCP.

While RefStack may be a necessary part of the solution to the interoperability problem, I am convinced that it isn’t sufficient. The biggest concern that I have is about the OpenStack design process. Everybody seems to be busily designing individual features and mechanisms, but nobody appears to be responsible for the specification of the OpenStack service itself. And that’s a problem.

There was a great example of this yesterday: a minor discussion in the networking part of the Design Summit. The details are relatively unimportant: a proposal for a mechanism to allow Quantum to provide some network hop cost information to the Nova scheduler. (It was a pretty bad design, IMHO.) But what was interesting was the use case which the authors provided to motivate the discussion: how to optimize the placement of elements of a scale-out three-tier application. In order to exploit the proposed mechanism, some process would have to compute the cost (measured in hops, latency, or whatever) of every configuration of available resources that matched the application graph. The problem is immediately obvious: who could carry out this process? The Nova Scheduler can’t, because it has no way of predicting the topology that will result from a sequence of VM instantiations. The IaaS user can’t, because she doesn’t have access to the full list of Nova resources. Of course a declarative model (like that provided in the vCloud API) would do the trick, but OpenStack doesn’t offer such a service.

So in many ways this was a pretty futile discussion, and for me the main reason was simple: there were no service-level requirements. There was a free-floating use case, unanchored to any external API or user-level abstration.

Now I’m not saying that this is an unimportant problem. It isn’t. It is certainly useful to allow an OpenStack user to provide a hint that certain instances will be communicating intensively, and that the scheduler should try to place them in such a way as to minimize latency. (And we have a plausible example of a service-level abstraction that captures this: the EC2 “Placement Group” concept.) We also have the case where the user requires that instances are placed on different servers or racks, to provide increased availability. What should the right abstractions look like? How do we represent this in the APIs – both syntax and semantics? What kind of SLAs are appropriate? And are there any constraints or dependencies associated with this aspect of the service? All of this would seem to be necessary before diving into a discussion of intra-system mechanisms.

There is always a risk when building a complex system that the external interface will emerge as an almost accidental side-effect of the implementation process. For isolated implementations, this may be OK – the client can meet the service half-way. For durable platforms, it is almost always a very bad thing. If the API is based on the implementation, changes to the implementation usually lead to changes in the API. There is always the risk that the system winds up being usable only by people like the implementor, which simply doesn’t scale.

As I’ve written before, I’m a big fan of API-first design. Specify the requirements for a system. Develop an API that captures the service contract. Test that API: ask potential users of the system to work through how they would code to it – BEFORE YOU WRITE A SINGLE LINE OF IMPLEMENTATION CODE. Think about the variety of possible implementations – technology, scale, private vs. multi-tenant, billed vs. free. Then implement. The result is likely to be a more stable service contract, with fewer incompatible changes from release to release, and with greater interoperability between deployments.

There are two widespread ideas about open source IaaS systems. The first is that they are like “Linux for the cloud”; the second that they correspond to one layer (API with orchestration) in a stack of technology that makes up the total system. Both of these have just enough truth to be deeply misleading. A system like OpenStack is a complex distributed system, with much greater architectural complexity than a POSIX operating system or a SQL data base. This complexity is reflected in the number of system elements and the variety of alternatives that must be accommodated for each element. And while it is true that the public face of an IaaS system is represented by an API (or GUI) and the complex orchestration that each request triggers, it is more than a veneer of automation over an existing system.

In most cases, the existing infrastructure – virtualization system, compute, networking, storage – has been assembled over a number of years, and reflects both technology choices and operational policies and procedures. Traditional practices for resource allocation and configuration usually involve a change management procedure including trouble ticketing systems, management review, and implementation checklists to mitigate human error. They operate at a timescale of days or weeks, reflecting the economics and risk management associated with human activities. One cannot simply add a layer of automation requiring responses in seconds or minutes on top of such a system.

The fact is that an IaaS system is functionally complex and deep. It’s complex because of the variety of services that are being delivered to the users of the IaaS, and the way in which lower-level resources must be allocated, composed and configured to deliver these services. There are complex interdependencies between the various services and resources, which leads to the need for orchestration. All of these resources are managed through different subsystems, often supplied by different vendors, and most of these subsystems are running on different systems, communicating over a set of networking resources, and sharing a variety of common services for storage, security, and so forth. All of these elements are typically replicated for reasons of availability and scale. And everything is being driven by API calls with stringent latency requirements. Temporal constraints – and economics – means that all of this must be 100% automated.

We cannot simply add a layer of IaaS on top of existing equipment, software, and operations. Every layer of the technology and operations of an IaaS, from API and GUI to physical infrastructure lifecycle management, needs to be re-thought. This doesn’t necessarily mean that IaaS is restricted to pure “greenfield” opportunities. However, rather than asking “what will work with what we already have?”, we need to start with a holistic architecture for the IaaS that reflects the technical requirements and business objectives. When we have this, we can consider the selective re-use of existing equipment, technologies and operational policies – but only when this makes good, long-term, economic and operational sense.

On Monday I had a long conversation with Derrick Harris of GigaOM, which turned into a piece entitled How to deal with cloud failure: Live, learn, fix, repeat. This was very well received (“one of the most intelligent articles I’ve read on this topic”), and I’d like to add a couple of related observations that we couldn’t work into the conversation.

Large scale cloud computing systems are characterized by two important properties. First, many of their functions are asynchronous, not transactional. The user issues an API call to start a VM in EC2 or OpenStack, the service acknowledges the request, and at some point later on the operation is completed. Usually. Perhaps not; maybe the request will fail. This is hard enough to deal with; when we start to compose multiple asynchronous mechanisms, the complexity can spiral out of control. For example, what happens when we bring together:

• An autoscaling system which monitors traffic and adds or removes capacity (VMs) as needed.
• A software deployment system which can roll out a new release to an existing fleet of machines at a controlled rate, and supports roll-back if the new software proves buggy.
• A self-healing mechanism which detects failed or “sick” instances of a fleet of VMs, and restarts or replaces them.
• A fault injection system (like the Netflix “Chaos Monkey”) which exercises the robustness of the system by periodically killing components.
• And all of these using an asynchronous API for managing VM instances.

While it may be possible to predict the eventual state of such a system, the path to that state, and the time to reach it, is extraordinarily hard to wrap your head around.

The second characteristic of today’s cloud systems is the massive levels of replication. S3 or Swift objects are stored in multiple data centers, service instances are replicated behind load balancers, load-balanced clusters are replicated in multiple data centers behind DNS anycasting or round-robin, and so forth. This is nothing new: we’ve been replicating, clustering, and doing fail-over since the 1960s. However at scale we cannot always afford to wait until replication has been completed – we can’t wrap a neat transaction around it – and we have embraced designs based on “eventual consistency” as a way of boosting throughput.

Put these two trends together, and we have systems that are radically asynchronous, massively replicated, and non-deterministic. These systems are hard to work with. Developers and operators like things to be transactional, linear, deterministic: “get used to disappointment”. We’re still learning how to work with systems like this, and we won’t always get it right. I was particularly peeved to read a simplistic assessment of the recent AWS outage in which the author claimed that…

… affected customers — Instagram, Pinterest, Pocket and Netflix, which all suffered from Amazon’s recent outage on the weekend — hadn’t used the ability of the cloud to create geographically redundant links.

“They could operate at a higher level of redundancy, so that these sort of outages would only have a minimal impact on them. It’s a matter of cost,” Bettin said.

In a word, bullshit. I know many of the guys who built these systems, and they have all incorporated high levels of redundancy in their system. None of them are “backup cheapskates” – that’s an insult which simply betrays the author’s failure to appreciate the complexity of the domain. I don’t know all of the root causes of their outages, but I’m prepared to bet that most were due to bugs arising from complex interactions between multiple systems, of the kind that I discussed earlier.

As I said to Derrick, I don’t expect the failure rate to plateau any time soon, because we are continually expanding scale, pushing performance, and introducing new sources of complexity. I do think that our tools for modeling, diagnosing, and repairing these systems are improving rapidly, however. I’ve mentioned Chaos Monkey as an example of fault injection techniques; I also expect to see more systems adopting anti-entropy elements, and dedicating a portion of the capacity of every system to in-production testing of functionality and performance.