Honestly the one interface should be the AWS API. Eucalyptus has already embraced it and set the tone. OpenStack is partially there but political battles in the community have forced a stalemate.
If you standardize on the lowest common denominator, you lose many of the key features of AWS such as VPC and autoscaling.
The AWS API is an INTERFACE to a proprietary system which runs closed source code. The code that runs the API endpoints is also closed source. Whether or not the SDKs for the AWS API are open or not is irrelevant to this discussion (not that you said something to the contrary, but I'm covering my bases here).
Eucalyptus has no more or less 'embraced' the AWS API methods than OpenStack has - save for a bare few methods that have different connotations in OpenStack land than they do with Euca: https://wiki.openstack.org/wiki/Nova/APIFeatureComparison.
The 'political battles' you are citing in OpenStack are actually a bunch of cage rattlers who are promoting their own interests in their respective service offerings. Going into a company that uses AWS for infrastructure and trying to pitch them on OpenStack is a long, drawn out process. Selling new technology is hard.
The lowest common denominator benefit that OpenStack provides is an open and transparent code base that can be modified and utilized by anyone. It's that openness that allows the few people using features like VPC and autoscaling to write some simple code to do deployments on whatever they like. Honestly, it's a handful of lines of code at the most for any given method. Saying you 'lose' feature functionality is simply a rationalization for attacking the OpenStack movement.
Rackspace pitched OpenStack hard to competitors when they released it. I mean hard. They were practically begging various folks to give them a moment to discuss switching entire hosting services over to OpenStack. The strategic play was, and remains, fairly obvious: if you remove feature differentiation the market competes on other differentiators (Rackspace wants to compete on support, most likely). Given that historical context of OpenStack's origin and not-so-subtle strategy, it's interesting to see such a dismissal of hosters that still want to compete on features.
Think about it. If you launch a hosting shop today and adopt OpenStack, you will be able to offer a similar feature set to Rackspace right off the bat. You then have to begin work on how to differentiate yourself feature-wise, while meanwhile not really competing with Rackspace because it's the same offering. OpenStack tells us that Rackspace wants all hosters to look largely equivalent technically.
I'm pretty sure what's where the project came from. Now, however, it appears to be evolving into a community effort to compete with Amazon, which is smart on Rackspace's part: by making a community effort, Rackspace (and everybody who runs OpenStack) get to capitalize on the work of the community to build an Amazon-like system, since such a system would require a significant investment on Rackspace's part and they likely don't have the people to do it. The danger, however, is that we'll end up with a hosting industry with dozens of AWS clones.
Regardless, in the broad, virtualization was our crutch until Linux containers caught up to what zone/jail admins have known for decades: there are far more efficient ways to share a machine than virtualizing an entire OS. The smart minds are focusing on containers now. I know OpenStack just implemented Docker support in Nova, but I don't think OpenStack models container computing well.
This is a long-standing argument in OpenStack. The root problem is that the AWS API exposes only what AWS exposes (trivial example: no live-migration.) The AWS API precludes features better than AWS, which is a big driver for some private clouds.
But there's no stalemate in OpenStack: it supports both APIs, although the EC2 API definitely gets less love. If you improve the EC2 API implementation, those improvements will get merged subject to the normal code review. If you want to do an ecosystem project that proxies EC2 calls to OpenStack, there are obvious questions over code duplication, but there's not profound opposition. And if you do want to do a proxy, go right ahead; you don't need OpenStack's buy in at all: build it, make it better than what is there already, and you'll win the argument.
The root problem (as with all open source) is that there are are a lot more people saying they want others to build better EC2 compatability, than there are people saying they want to build it.
> If you standardize on the lowest common denominator, you lose many of the key features of AWS such as VPC and autoscaling.
That's not the level where libcloud is aiming for LCD - it's not at a provider level, but at a product level. As long as a couple of providers support autoscaling, and AWS and Rackspace do, there can be support for it. It's on my todo list to implement, by finding the common thread between AWS and Rackspace APIs, and developing an abstraction that aims to cover the features of both.
The AWS API is convoluted, overwhelmingly complicated, and often yields no insight to the actual behavior taking place in the Amazon system. I can think of countless scenarios wherein my API call succeeded but nothing happened, with no reporting of the reason why anywhere. A later run of the exact same call again works. No explanation.
Now, arguably, I'm discussing the implementation of the specific API, but even on an architecture level the AWS API tries to handle absolutely every case in the most complex manner possible. This is because AWS tries to handle every case in their systems; where other providers trim things for simplicity, AWS adds as much as it can. This flexibility can be useful if you need it but to someone getting started, it is a giant wall. They've gotten much better about guiding people through setup, but the API still reflects the kitchen sink philosophy. Everybody I've ever known who loves the AWS API has only ever used boto. As an exercise, launch an instance with ephemeral and EBS storage mounted using the bare API on a clean account. You'll see what boto hides from you.
The correct approach is something like libcloud with the ability to interact with platform-specific things in a way that the library does not anticipate. However, there's a school of thought in that if you are digging yourself in to platform-specific features, your ability to move is now a significant line item of technical debt that could put you in a position where you are completely stuck and cannot hire enough people to get out.
As a former hosting employee it is immeasurably frustrating that everybody mentally defaults to AWS these days without even a second thought. All of Silicon Valley is basically wired to funnel venture capital directly to Amazon. The VC firms might as well just invest in Amazon and skip the startup risk, since inevitably most of the cash burn of a startup will go directly to AWS (because it takes a genius of reservations, flexible scaling, and other tricks to not spend six figures a month on your hosting). AWS was designed for flexible scaling, not running your 700 instances 24x7 regardless of load, but nobody operates that way. You think Amazon is going to tell you that?
With Amazon, the API is full of issues, and that's even before speaking about the service itself as well as the support; even with paid support, 12-hour ticket responses were the norm for me including in an outage situation. My personal favorite was a six-month ticket that culminated in "we don't know, so we're closing this ticket," when reporting buggy behavior with their API. There are many aspects of AWS that are dreadful, the API being only a small example, and the suggestion that we should standardize on the AWS way of doing things as an industry is very harmful. I'd contend we need a viable alternative, but Amazon has made themselves a Kitchen Sink Provider and other companies can only bite at little pieces of the pie.
Very different experience with AWS/OpenStack/running our own here. We indeed had a largish AWS bill ~50k/month. Decided to move dev/test in house into an OpenStack cluster I built with the help of Mirantis/FuelWeb. Decent experience standing up the cluster but spurts of trouble thereafter. No real win with anything API-like at all with OpenStack. Feels half-baked. Getting help is damn near impossible.
Meanwhile in AWS-land, things are humming along. Using more and more of the AWS infrastructure and removing pieces we had previously built in-house. Getting lots of mileage out of boto/Java/awscli and starting to see a real paradigm shift in how to run ephemeral servers instead of the 700 instances 24x7 approach you refer to above.
I came from running my own servers, floundered for a bit on AWS, built an OpenStack cluster and strongly prefer AWS for simplicity and sheer flexibility of the API. I'm a year in running on AWS and haven't had a single issue requiring any support.
The largest benefit has come from the concepts and projects put out by Netflix. If I didn't see companies like them leading the way, I'd likely be a bigger OpenStack fanboy.
If you standardize on the lowest common denominator, you lose many of the key features of AWS such as VPC and autoscaling.