Twitter unveils Heron, a real-time analytics system that can withstand social traffic spikes

To better support analytics for the scale and activity of its social stream, Twitter has announced Heron.

Heron replaces Storm, Twitter’s existing open source platform, but remains compatible with the legacy service via APIs.

Like Storm, Heron is tasked with providing real-time analytics for Twitter’s feed. With Heron, Twitter can process billions of events per minute with sub-second latency, and Twitter says it’s up to 15-times faster than Storm. Heron is also designed to withstand activity spikes when popular events occur without bottlenecking.

Twitter designed Heron to be easily managed for better developer productivity, and will provide better performance predictability.

Heron runs tasks in process-level isolation, which will make debugging easier. It has an off-the-shelf scheduler, and utilizes Storm APIs to submit topologies. A data flow pressure mechanism will handle any traffic spikes, but keeps data accuracy on-point throughout.

Heron is already in use at Twitter, where the company says it’s seen a three-fold reduction in hardware, “causing a significant improvement in our infrastructure efficiency.”

To learn more about Heron in-depth, read Twitter’s research paper from SIGMOD 2015.

➤ Flying faster with Twitter Heron [Twitter]