Four trends that shaped cloud computing in 2011

Luis Robles is a venture capitalist at Sequoia Capital.

In a few short years, we all have witnessed cloud computing unleash a wave of innovation in IT. Enterprises are continuing to adopt cloud-based services and entrepreneurs are finding plentiful low cost and low friction compute resources to transform ideas into enduring new companies. With the year coming to a close, and VentureBeat’s CloudBeat 2011 conference just a day away, here’s a look at some of the more notable cloud trends of the past year:

1. Amazon web services hiccuped a few times but continued to set the pace for public clouds

We’re seeing a growing number of cloud offerings, but developers are doubling down on the ones that are innovating fastest — and AWS is leading the pack. Driven by a steady stream of feature enhancements, AWS continued its explosive growth in 2011, recently announcing that its S3 service was processing over 370K requests per second and had doubled in 9 months to store a staggering 566 billion objects! Most of the entrepreneurs we met in 2011 were using AWS somehow, and even Amazon is finding interesting ways to leverage its own infrastructure (such as the new AWS-powered Silk browser).

Amazon’s cloud outages in April and August were certainly disruptive to many and raised doubts about the stability and scalability of a service possibly expanding too quickly. But the lesson to be drawn was not to rewind the clock and return to hosting your own infrastructure. Rather, it was to distribute your infrastructure across multiple locations and plan for failures. Enterprise IT departments make considerable investments outfitting secondary datacenters and maintaining disaster recovery plans that rarely work as planned. In contrast, many of the startups that diversified geographically with AWS availability zones recovered quickly or avoided service interruptions altogether because they had become accustomed to spinning up new instances of their applications quickly, or because they built their applications with robust levels of redundancy that would have been a stretch to achieve on their own.

2. Private clouds grew and alternatives to VMware matured

Server virtualization remains a core building block for clouds. According to industry analysts, some 40-50% of all workloads running on x86 servers will be virtualized by the end of 2011. For enterprises building and maintaining their own private clouds, VMware delivered a number of compelling enhancements to the vSphere 5 platform, such as “monster” virtual machines (up to 32 virtual CPUs & 1TB of RAM), storage DRS, and a virtual storage appliance. The VMware vCloud Director ecosystem also received a boost from new features and partners, with 25 different datacenters across 13 countries slated to be powered by vCloud by the end of the year.

However, smaller players and new open source “cloud operating system” offerings are finding open doors amongst enterprises considering alternatives to VMware’s premium features and expanding the overall market. For instance, OpenStack (the NASA and Rackspace-backed open source cloud platform), gained significant traction, celebrating its first birthday in July and now claiming more than 130 participants and contributors. Rackspace has done a nice job building an ecosystem around OpenStack and is driving a wedge into the market with its Cloudbuilders and RackConnect initiatives, which enable enterprises to get private instances of an OpenStack cloud and augment them with resources from Rackspace’s public cloud. Citrix acquired Cloud.com to boost its product offerings for service providers and is surely gaining valuable insights from marquee customers creating their own clouds such as Zynga. Startups like Eucalyptus and Nimbula also continued to win customers by addressing hybrid use cases that combine some of the benefits of private and public clouds.

3. Cloud productivity apps became good enough

Cloud-based productivity applications like Google Docs have been around for a few years, but in 2011 they went mainstream and became credible replacements to Microsoft Office for most people. Google rounded out its Docs suite with an updated Presentations program, and the commenting, revision history, and sharing features in Docs are cleverly implemented to make emailing documents back and forth seem archaic. Microsoft responded with Office365, and its licensing plan for small businesses has even replaced the flagship Office desktop software with the new web-based offerings. Other startups such as SlideRocket and Prezi delighted their users by demonstrating innovations of their own.

Of course, many companies still use Office, and it’s not going away anytime soon. But most startups don’t install any software at all, and larger businesses are starting to consider foregoing desktop software upgrades. Many of us are rarely without access to the Internet or a 3G connection, and we’re spending more and more of our day in a browser (or across browsers on different devices). Many common, everyday tasks are now easier and better done on the web, like working on a budget or collaborating on a memo. The powerful combination of utility computing, ‘good enough’ HTML5-based apps, and the proliferation of new devices suggests the demand for cloud applications is set to match the hype and truly takeoff in 2012.

4. Tools for wrangling big data showed that volume of data is only part of the equation

Consumers, enterprises, mobile devices, and sensors in the world today generate more data than ever — so much that an estimated 90% of the world’s data has been created in the last 2 years, and much of it is landing in the cloud. Volume of data is certainly not slowing down, but in the past year we’ve seen a shift in the variety of data and an increase in how fast its generated. Unstructured or loosely structured data is adding to the variety and will likely alter the budgets being allocated to databases and storage systems. Time sensitivity is also increasing, and data must be streamed through applications and enterprises at higher speeds than before to maximize its value.

Given these trends, MapReduce, Hadoop, and “big data” analytics are crossing the chasm from early adopters to more mainstream use cases. Companies across a variety of verticals such as e-commerce, financial services, life sciences, and telecommunications are deploying new tools to pipeline data analysis through their organizations more efficiently. Management tools and user interfaces for the new data processing engines are also making complex queries and analyses accessible to audiences beyond the “data scientists”. The Hadoop World conference last month showcased a thriving ecosystem of vendors and household names such as eBay and JP Morgan Chase using products from those vendors to crunch their data. There’s similar momentum behind MongoDB and other open source NoSQL initiatives, with developers using new tools to build new applications.

In aggregate, developers are downloading map-reduce frameworks and non-relational databases more than 200,000 times per month. The business intelligence companies are deriving from data is becoming an increasingly important competitive advantage, and companies are creating new jobs and paying significant salary premiums to attract and retain the best data scientists — inside and outside the San Francisco Bay Area.

These are just a few of the highlights from a remarkable year in cloud computing. What other trends would you consider to be truly disruptive? As customers and early adopters, what feature updates or new product offerings would you like to see from some of these technology vendors? I’d love to hear your thoughts in the comments below and look forward to meeting many of you at VentureBeat’s CloudBeat 2011 conference this week.

Disclosure: Sequoia Capital is a business partner of 10Gen (which develops MongoDB) and Nimbula, and a former business partner of Google and Rackspace. At Sequoia, Luis works with cloud computing, software, and enterprise companies, and he is a former employee of VMware. He can be reached at robles[at]sequoiacap.com.

CloudBeat 2011 takes place Nov 30 – Dec 1 at the Hotel Sofitel in Redwood City, CA. Unlike any other cloud events, we’ll be focusing on 12 case studies where we’ll dissect the most disruptive instances of enterprise adoption of the cloud. Speakers include: Aaron Levie, Co-Founder & CEO of Box.net; Amit Singh VP of Enterprise at Google; Adrian Cockcroft, Director of Cloud Architecture at Netflix; Byron Sebastian, Senior VP of Platforms at Salesforce; Lew Tucker, VP & CTO of Cloud Computing at Cisco, and many more. Join 500 executives for two days packed with actionable lessons and networking opportunities as we define the key processes and architectures that companies must put in place in order to survive and prosper. Register here. Spaces are very limited!