Why Amazon created AWS Kinesis, its live data processing service

NEW YORK — Metering is an essential part of Amazon Web Services.

Amazon records how much data its AWS customers require so companies only pay for what they use, explained Amazon’s Ryan Waite onstage at GigaOM’s Structure Data conference on Thursday. Because Amazon customers generate tens of billions of records daily — and lots of apps could benefit from access to that data — Amazon needed a system that could capture those tiny pieces of data in real time.

[aditude-amp id="flyingcarpet" targeting='{"env":"staging","page_type":"article","post_id":1181157,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"big-data,business,cloud,","session":"A"}']

Kinesis is a service for real-time processing of streaming data that Amazon launched in December. It’s designed to capture those tiny records, which make up a huge amount of backend data in aggregate. Amazon wanted to be able to capture all of that data, aggregate it, and put it into its AWS cloud storage service (S3) so it could go back and look at it later.

“This enables us to scale the metering service to new limits and give alerts in real time,” said Waite, who serves as general manager of data services for AWS.

Kinesis enabled Amazon to increase its load to billions of records from millions of files, or terabytes of data per hour. All of the data gets stored in Kinesis for 24 hours, and it can be offloaded to Amazon S3 for later examination.

Waite characterized the service as “incredibly cheap” — a million PUT transactions costs a customer 2.8 cents — and immensely useful for AWS customers. He highlighted digital marketing company Bizo, which uses Kinesis to bolster its real-time analytics of its clients’ online ad campaigns.

“With Kinesis, they can sit down side-by-side and watch the campaign as it’s happening,” said Waite.

Waite also pointed to billion-dollar game developer Supercell, which makes the massively popular mobile game Clash of Clans.

By using Kinesis, “They can see what’s working in the game and what’s not working in the game and make changes.”