Prepared notes for the “AWS APIs in OpenStack” debate

This evening there’s a meetup in San Francisco at which people will be debating the role of AWS APIs in OpenStack. It should be a lot of fun; at least 150 people have RSVP’d. I’m planning to be there, but just in case I thought I’d put down my position in advance. I’ve blogged about much of this stuff previously, so for regular readers this may be tl;dr.

The Great AWS API Debate.

I want to begin by emphasizing that I’m a longtime supporter for OpenStack. I’ve been involved for the last couple of years; my employer, Brocade, has contributed to both Neutron and Cinder, including FC SAN. My day job is to enable multivendor NFV for OpenStack.

Two years ago, I developed an IaaS architecture for Yahoo that was based on OpenStack. We decided from the start that we would only use the AWS (EC2, S3, EBS) APIs; we would not expose the OpenStack API to users. There were two motivations for this. First, we wanted to be able to leverage the work of the AWS ecosystem; today I would cite the importance of being able to exploit Netflix OSS. Secondly, I was concerned about the stability of the native OpenStack API, and the likelihood of inconsistent feature support and ad hoc extension.

Today, my concerns seem quite justified. What I had not anticipated was that OpenStack development would have been so slow. I had hoped that after three years we might have replicated the major AWS features from 2009, but this has not happened. In large measure, I attribute this failure to the absence of a crisp, immutable set of requirements. AWS API compatibility would have provided these requirements.

A few open source projects have the luxury of being first movers. This was not the case for two of the most successful projects: Linux and MySQL. In each case, the project succeeded by embracing and extending an existing standard API. Linux would not have succeeded without POSIX. MySQL would not have succeeded without SQL. The reasons are straightforward. Implementing a broadly accepted standard reduces the costs of adoption; it also provides a shared goal, a clear set of requirements.

There are two common arguments against focussing on the AWS APIs. The first is that we give up control; that Amazon will set the agenda going forward. This is naive. Amazon’s ability to make radical changes to the API is severely constrained by the long-term needs of their customers. They are not stupid. (Disclaimer: I used to work there.) The second is that it would limit the ability of the OpenStack community to innovate; that it’s too soon to settle on a single definition of IaaS. This would carry more weight if there was actual evidence that OpenStack was out-innovating AWS.

Having said all of this, where do I stand on the current question? Unfortunately, I think that it may be too late. I’m concerned that we’ve accepted so many slightly divergent implementations and extensions that it may be impossible to define an OpenStack subset which does provide the right degree of compatibility. It’s worth re-reading Rob Hirschfeld’s series of blog posts on “OpenStack Core”; the relationship of a “core” to AWS compatibility is really unclear. It’s also important to recognize that OpenStack is being applied to a wider set of use cases than many of us expected. In addition to AWS-style public or private clouds, with uniform pooled resources, OpenStack is being used for general enterprise data center automation, including the management of legacy (and distinctly un-cloudy) resources. That’s not going away.

1 Comment to "Prepared notes for the “AWS APIs in OpenStack” debate"

  1. August 16, 2013 - 8:15 PM | Permalink

    Thanks for the link back and a chance to explain how defining core DOES help w/ the AWS API question. The critical thing about defining core is that it helps everyone know exactly what’s required and what’s option in OpenStack. In this case, I suspect that the OpenStack API would be considered core and AWS API would be optional. We’re already seeing that OpenStack will have a lot of optional extensions – that’s the sign of healthy ecosystems and adoption.

    The exciting direction our core discussion has been taking is expectation that we will use tests to define functionality. This has been a growing part of the interop discussion.

    It applies to the AWS APIs in this way: if community cares about an extra API then someone will code it AND write tests to verify it. If we have tests then we can validate that they remain in working condition (in the case of AWS, we could validate against BOTH OpenStack and AWS!). Then we can collect usage stats on the tests from the field and judge if those these should become “must-pass” and core function.

    The key is that we can shift the debate from “should we support” to “this is a popular feature.”

    Thanks again for the link. I hope this helps.

Comments are closed.