Last week I wrote the following on FaceBook:
I need to write a blog post about open source: specifically about those people who respond to any criticism of a project by saying, “The community is open to all – if you want to influence the project, start contributing, write code.” Because, seriously, if you want your project to be really successful, that attitude simply won’t work. It doesn’t scale. You want to have more users than implementors, and those users need a voice. (And at scale, implementors are lousy proxies for users.)
Maybe this weekend.
This is that blog post. It has been helped by the many friends who added their comments to that FaceBook post, and I’ll be quoting from some of them.
Over the last few months, I’ve been trying to figure out why I’m uncomfortable with the present state of OpenStack, and whether the problem is with me or with OpenStack. Various issues have surfaced – the status of AWS APIs, the “OpenStack Core” project, the state of Neutron, the debate over Solum – and they all seem to come down to a pair of related questions: what is OpenStack trying to do, and who’s making the decisions? And my discomfort arises from my fear that, right now, the two answers are mutually incompatible.
First, the what. That’s pretty simple on the surface. We can look at the language on the OpenStack website, and listen to the Summit keynotes, and (since actions speak truer, if not louder, than words) see the kind of projects which are being accepted. OpenStack is all about building a ubiquitous cloud operating system. There is to be one code base, with an open source version of every single function. And although the system will be delivered through many channels to a wide variety of customers, the intent is that all deployed OpenStack clouds will be fundamentally interoperable; that to use the name OpenStack one must pass a black-box compatibility test. (That’s the easy bit; the more fuzzy requirement is that your implementation must be based on some minimum set of the open source code.) This interoperability is motivated by two goals: first, to avoid the emergence of (probably closed) forks, and second to enable the creation of a strong hybrid cloud marketplace, with brokers, load-bursting, and so forth.
This means that the OpenStack APIs (and, in some less well defined sense, the code) are intended to become a de facto standard. In the public cloud space, while Rackspace, HP, IBM and others will compete on price, support, and added-value services, they are all expected to offer services with binary API compatibility. This means that their customers, presumably numbering in the tens or hundreds of thousands, are all users of the OpenStack APIs. They have a huge stake in the governance and quality of those APIs, and will have their opinions about how they should evolve.
How are their voices heard? What role do they play in making the decisions?
Today, the answers are “they’re not” and “none”. Because that’s not how open source projects work. As Simon Phipps wrote, “…open source communities are not there to serve end users. They are there to serve the needs of the people who show up to collaborate.” And he went on to say,
What has never worked anywhere I have watched has been users dictating function to developers. They get the response Geoff gave in his original remarks, but what that response really means is “the people who are entitled to tell me what to do are the people who pay me or who are the target of my volunteerism — not you with your sense of entitlement who don’t even help with documentation, bug reports or FAQ editing let alone coding.”
As I see it, there are two basic problems with this thinking. First, it doesn’t scale. If you have hundreds of thousands of users, of whom a small percentage want to help, and a couple of dozen core committers, the logistics simply won’t work. (And developers are often very bad at understanding the requirements of real users.) Maybe you can use the big cloud operators and distribution vendors as proxies for the users, but open source communities can be just as intolerant of corporate muscle as they are of users who don’t get involved. Some of that can be managed — much of the resources that Simon terms “volunteerism” actually comes from corporations — but not all.
But the second problem is that the expectation of collaborator primacy is incompatible with the goal of creating a standard. The users of OpenStack have a choice, and the standard will be what they choose. If the OpenStack community wants their technology to become that standard, they must find a way to respect the needs and expectations of the users. And that, quid pro quo, means giving up some control, which will affect what the community gets to do — or, more often, not do.
This is a much bigger issue for OpenStack than for other open source projects, for a couple of reasons. Several of the most successful projects chose to implement existing de facto or de jure standards — POSIX, x86, X Windows, SQL, SMB, TCP/IP. For these projects, the users knew what to expect, there were alternative (non-FOSS) alternatives, and the open source communities accepted the restrictions involved in not compromising the standards. Other projects were structured around a relatively compact functional idea — Hadoop, MongoDB, Xen, MemCache, etc. — and the target audience was relatively small: like-minded developers. OpenStack is neither compact, nor (by choice) standards-based. It has a large number of components, with a very large number of pluggable interfaces. The only comparable open source project is OpenOffice, which had a similarly ambitious mission statement and has also had its share of governance issues.
One obvious step towards addressing the problem would be to distinguish between the “what” and the “how” of OpenStack. Users are interested in the APIs that they use to launch and kill VMs, but they are unlikely to know or care about the way in which Nova uses Neutron services to set up and tear down the network interfaces for those VMs. Good software engineering principles call for a clear distinction between different kinds of interfaces in a distributed system, together with clear policies about evolution, backward compatibility, and deprecation for each type. This is not simply a matter of governance. For example, a user-facing interface may have specific requirements in areas such as security, load balancing, metering, and DNS visibility that are not applicable to intrasystem interfaces. There is also the matter of consistency. Large systems typically have multiple APIs — process management, storage, networking, security, and so forth — and it is important that the various APIs exhibit consistent semantics in matters that cut across the different domains.
The adoption of some kind of interface taxonomy and governance model seems necessary for OpenStack, so that even if community members have to relinquish some control over the user-facing interfaces (the “what”), they still have unfettered freedom with the internal implementation (the “how”). Today, however, we are a long way from that. OpenStack consists of a collection of services, each with its own API and complex interdependencies. There is no clear distinction between internal and external interfaces, and there are significant inconsistencies between the different service APIs. At the last OpenStack Summit I was appalled to read several BluePrints (project proposals) which described changes to user-facing APIs without providing any kind of external use-case justification.
The present situation seems untenable. If OpenStack wants to become an industry standard for cloud computing, it will have to accept that there are multiple stakeholders involved in such a process, and that the “people who show up to collaborate” can’t simply do whatever they want. At a minimum, this will affect the governance — consistency, stability — of the user-facing interfaces; in practice it will also drive functional requirements. Without this, traditional enterprise software vendors delivering OpenStack-based products and services will have a hard time reconciling their customers’ expectations of stability with the volatility of a governance-free project. The bottom line: either the community will evolve to meet these new realities, or OpenStack will fail to meet its ambitious goals.
UPDATE: Rob Hirschfeld seems to want it both ways in his latest piece. Resolving the tension between ubiquitous success and participatory primacy doesn’t necessarily require a “benevolent dictator”, but that’s one way of getting the community to agree on a durable governance model.