Twitter details Heron, a real-time stream-processing system that outperforms Apache Storm

Twitter today is showing off Heron, a new real-time stream processing system its engineers have built. Heron offers several advantages over Apache Storm, the stream-processing system Twitter developed in house and later open-sourced years ago.

For one thing, Heron is less demanding on the data center infrastructure that Twitter relies on.

[aditude-amp id="flyingcarpet" targeting='{"env":"staging","page_type":"article","post_id":1741284,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"big-data,dev,enterprise,","session":"D"}']

“Since Heron is efficient in terms of resource usage, after migrating all Twitter’s topologies to it we’ve seen an overall 3x reduction in hardware, causing a significant improvement in our infrastructure efficiency,” Twitter engineering manager Karthik Ramasamy wrote in the blog post.

Several tests prove that Heron provides performance gains over Storm, too.

Heron is now the main stream-processing engine inside Twitter, Ramasamy wrote — but it is backward-compatible with Storm, which was widely used across Twitter in the past.

Several companies — and cloud providers such as Google Cloud Platform, Microsoft Azure, and Amazon Web Services — have embraced Storm over the years. It’s certainly possible that Heron will follow a similar trajectory.

Ramasamy did not say anything about releasing Heron under an open-source license, although he did say sharing an academic paper on Heron at the 2015 ACM SIGMOD International Conference on Management of Data was a “first step” toward sharing lessons learned “in order to further develop these programs.”

For more on Heron, check out the full academic paper here.