Happy New Year! I hope you are all having a fantastic holiday. This is a year end posting that I think you will find particularly compelling. Rather than predicting the future I thought I would take a look back at five long years of ‘cloud computing’.
The Cloudscaling blog has a loyal following as can be seen from the website and RSS feed stats. As many of you long time readers know, I’ve been ‘in the game’ working on cloud computing technology or blogging about as long as anyone except perhaps those at AWS. In all of that time, my thinking and assessment of what’s happening and how it’s evolving has changed continuously. What was interesting for me this year is that this continuously changing perspective slowed to a crawl or perhaps even stopped. 2011 is the year that much of my thinking and perspective on cloud computing, particularly infrastructure clouds (aka “IaaS”) hardened.
That sounds tough. “Hardened.” I don’t mean hardened in the sense of rigid, but rather in the notion of wet cement drying. Many things that have seemed up in the air now seem settled and my doubts about the future of infrastructure clouds are gone. They are not only here to stay, but the shape and direction of them seem very clear. I’m not certain everyone else is clear, but I am. Perhaps I will be wrong, but I doubt it.
Let’s take a look back at the arc of my thinking and how things did NOT change in 2011. That will tell us what 2012 is likely to look like.
The Evolution of My Cloud Thinking
My thinking evolved through three clear phases:
[1] KT’s private and public compute clouds, their OpenStack storage cloud, Internap’s OpenStack storage cloud, and another I can’t currently discuss. [2] By my current estimation AWS is closing out at 71 significant feature releases this year, up 5 from my estimate of 66 for 2011. I will provide a more detailed update soon. [3] The one possible exception here is the Rackspace team who I give full props to for understanding the nature of the change and doing their best to adapt. [4] I plan to explore VLANs and the confusion there and explain why VPC is meaingful, but mostly for legacy apps in a future posting; the biggest AWS users, like Zynga and Netflix don’t use VPC or VLANs at all. [5] Surfacing vendor specific ‘features’ to differentiate your load balancing service simply provides a layer of lock-in that end-users don’t want while making your infrastructure cloud less compatible with others. [6] I strongly recommend reading Simon Wardley’s piece on enterprise clouds. [7] If you haven’t you *really* should watch this great video interview I did with Lew Tucker, CTO of Cisco Cloud Computing on operational and capital costs for building infrastructure clouds. [8] Also be sure to watch this video interview I did with Adrian Cockcroft at CloudBeat 2011. [9] I think my posting from late 2010 on why ‘Elasticity is NOT Cloud Computing’ still holds up well in this context. [10] You really should listen to this great podcast (audio, text summary)I did with Horace Dediu of Asymco where we cover a lot of crowd in the relationship between the post-PC era and cloud computing. [11] Unfortunately, I don’t have a reference for this. I’ve heard it ‘off the record’ from a number of sources at Cisco and VCE, but I can’t find a public reference on it. If anyone has such a reference I would appreciate a link in the comments below. Full credit will be provided. [Freebie] Quora question: In what ways is AWS better than it’s competitors?
Automation -> VMs & Virtual Datacenters -> New IT ParadigmPhase 1: Automation About this time of year, in late 2006, a short time after Amazon EC2 launched, myself and others prototyped a cloud application management framework similar to RightScale. At that time RightScale was named something else and had not been funded or publicly launched. These were early days. As someone with a deep passion for automation, I remember thinking then that a lot of my lifetime interests (networking, storage, security, and systems management) were all converging and being managed by automation. For me, what was happening was all about automation … and lots of it. Phase 2: Virtual Machines & Virtual Datacenters Roughly summer of 2008, the first “CloudCamp” was thrown where a number of the cloud bloggers and thought leaders came together for the first time. Unknowingly we all centered about using the term ‘cloud computing’ to explain what this new emerging phenomena was. It was right after this event and over the summer of 2008 that the term “cloud computing” really took hold. This also led to the formation of the “clouderati” and I simultaneously joined GoGrid as the VP Technology Strategy. Perhaps GoGrid biased my thinking, but I started to move from a perspective that was cloud application centric back into my sweet spot of physical infrastructure and a focus on virtual datacenters or what I called at the time, “cloud centers”. In this view, virtual machines were king and inevitably, the question was: “how will we model existing datacenter environments?” Phase 3: Cloud Computing is a new kind of IT After leaving GoGrid in the summer of 2009 I had the opportunity to step back and take a fresh look at how things had evolved. I wanted to build my own cloud business again, but I wanted to skate to where the puck would be, not where it was today. I also could see that most everyone involved in the cloud computing space was spending time trying to retrofit the notion of ‘cloud computing’ to their existing business models and technology. Simultaneously, I still didn’t see any serious competitors to AWS. What were they doing that was so different?? It’s not well known, but in the beginning of Cloudscaling’s (re)formation in fall of 2009 into mid 2010, I did a number of strategic and due diligence engagements on various IaaS vendors for VC firms, Platform-as-a-Service startups, enterprises, and enterprise vendors. During that time I was involved in deep technical dives on the technology and business models for these IaaS vendors. They ranged from GoGrid competitors to more of an enterprise cloud model. By late 2010 Cloudscaling, collectively, had deep architectural and business model understanding of roughly 10 different Infrastructure-as-a-Service (IaaS) players, not including the IaaS clouds that we helped build [1]. I am not sure anyone else had or has that understanding today. What we saw, was telling. My primary takeaway was that even when it came to startups and direct AWS competitors, absolutely none of the infrastructure cloud players were developing their clouds like AWS. For the most part, they were simply integrating common-off-the-shelf (COTS) components to mimic an AWS-like environment. None of them had AWS velocity [2], nor were they paying attention to what made AWS special [3]. All too often, they identified ‘flaws’ in AWS that were instead unrecognized strengths. Examples of this include:
- Constrained feature set
- Standardized instance sizes
- Lack of VLANs [4]
- Ephemeral storage
- Generic load balancing service without fancy vendor lock-in features [5]
- Why isn’t VMware more successful in the public cloud space if it’s just VMs and VDCs?
- Why isn’t there a VMware-based competitor at similar scale to AWS? Or even close?
- There are now 100+ “VMs on demand” competitors, but almost none have the same growth rate as AWS … why not?
- What do the largest Internet giants (Amazon, Google, Facebook, SFDC) all have in common from an architectural standpoint and how is that different from a typical enterprise datacenter?
- How do I virtualize and manage my existing datacenter apps?
- How do I achieve bottom line cost savings and extend server consolidation?
- How can my existing vendors help me create a ‘private cloud’?
- How can I be compatible with everything I own today?
- What can we do to allow application developers to experience ‘infinite scalability’?
- How can we simplify the allocation of traditional IT resources of networking, storage, and compute?
- What will it take to help next generation web applications ‘scale’ by simply adding more of these IT resources?
- How do we make it continually less expensive such that application developers can consume as much as they need?
- How can I, the service provider, make my cost of capital equipment and operational management as low as possible so I can pass those savings on to app developers? [7]
- Big data, the explosion of data and data processing needs
- The post-PC era, or the notion of the rise of appliances and mobile platforms as the long term predominant platform, and the shift to ‘apps’ from ‘desktops’ [10]
- Consumerization of IT (TrendMicro whitepaper in PDF), or the notion that knowledge workers prefer more adaptive and flexible environments to get their work done such as they experience in their private lives with the large web application providers (Google, Amazon, Facebook, etc.)
[1] KT’s private and public compute clouds, their OpenStack storage cloud, Internap’s OpenStack storage cloud, and another I can’t currently discuss. [2] By my current estimation AWS is closing out at 71 significant feature releases this year, up 5 from my estimate of 66 for 2011. I will provide a more detailed update soon. [3] The one possible exception here is the Rackspace team who I give full props to for understanding the nature of the change and doing their best to adapt. [4] I plan to explore VLANs and the confusion there and explain why VPC is meaingful, but mostly for legacy apps in a future posting; the biggest AWS users, like Zynga and Netflix don’t use VPC or VLANs at all. [5] Surfacing vendor specific ‘features’ to differentiate your load balancing service simply provides a layer of lock-in that end-users don’t want while making your infrastructure cloud less compatible with others. [6] I strongly recommend reading Simon Wardley’s piece on enterprise clouds. [7] If you haven’t you *really* should watch this great video interview I did with Lew Tucker, CTO of Cisco Cloud Computing on operational and capital costs for building infrastructure clouds. [8] Also be sure to watch this video interview I did with Adrian Cockcroft at CloudBeat 2011. [9] I think my posting from late 2010 on why ‘Elasticity is NOT Cloud Computing’ still holds up well in this context. [10] You really should listen to this great podcast (audio, text summary)I did with Horace Dediu of Asymco where we cover a lot of crowd in the relationship between the post-PC era and cloud computing. [11] Unfortunately, I don’t have a reference for this. I’ve heard it ‘off the record’ from a number of sources at Cisco and VCE, but I can’t find a public reference on it. If anyone has such a reference I would appreciate a link in the comments below. Full credit will be provided. [Freebie] Quora question: In what ways is AWS better than it’s competitors?