How Hotstar streams cricket matches to 10 million passionate fans at once

Last week, Google’s YouTube TV suffered an hour-long outage while it was airing a soccer World Cup match between Croatia and England, disappointing scores of subscribers in the process. It even led some to question whether it’s possible for any company to handle live streaming such a major event.

We're watching the FIFA on YouTube TV and it's completely down. If Google can't keep it online in a surge like this, nobody can. https://t.co/Jeed9kZY5b

— ⚡️ Owen (@ow) July 11, 2018

The truth is that while live streaming high-definition video to large audiences can be incredibly challenging, others have risen to the challenge. India-based streaming service Hotstar, which has garnered some 350 million followers across its lifetime and is owned by 21st Century Fox, streamed a cricket match to 10.3 million viewers concurrently earlier this year.

It happened on May 27, when the Chennai Super Kings faced off against the Sunrisers Hyderabad in the final match of the 11th edition of the Vivo Indian Premier League at Mumbai’s Wankhede Stadium. The IPL is the most-attended cricket league on the planet, and brings together the best players from across the world in a fast-paced format.

The 💜 of EU tech

The latest rumblings from the EU tech scene, a story from our wise ol' founder Boris, and some questionable AI art. It's free, every week, in your inbox. Sign up now!

How did Hotstar pull it off? CEO Ajit Mohan told TNW that it’s learned a lot from being in the business for years and cutting its teeth in nail-biting circumstances:

One of the very first events that we streamed on Hotstar was the 2015 ICC Cricket World Cup match between India and Pakistan. If I recall correctly, we ran into trouble with the stream within the first five minutes. For Hotstar, the crucible moment in streaming sports came in the very first day of a big sporting event. Thankfully, we were able to get the stream back up within a few minutes.

He added that supporting millions of concurrent stream in real-time isn’t just about setting up a robust video delivery platform or content delivery network; it also requires building out a solid API and end-user app:

I think a lot of the conversation sometimes gets limited to just talking about video and content delivery network (CDN) infrastructure. But fundamentally, to make sure that you have a high quality experience, you have to solve for not just the video infrastructure but also the client app and API. And I think that kind of a thoughtful approach is what we have pioneered in hotstar over the last few years.

In sports, you can move from zero to four million streams in less than a second. With most other content categories, there’s a much smoother curve over hours of streaming. And that essentially pressure tests the platform at different levels. So, your apps have to be able to scale. There are different API points for the apps to access your content, and you have to make sure that all of those API calls can handle the scale that comes with the surge.

I was particularly interested to learn about how those surges of millions of users happen in a matter of seconds. That’s often due to people hearing about major turning points during a match. Mohan explained that Hotstar draws viewers in during these moments too, with a simple little feature in its mobile app:

We’ve understood that users may not have the time to watch a match from start to finish, the full match, but are quite comfortable glancing at a scoreboard on their phones. We offer a lock screen widget for that, and that ends up being a trigger to start watching the match at the right point. But, as with most sports, these trigger points (such as a wicket falling – the equivalent of a hitter striking out in baseball) happen to be common for millions of fans. That creates the surge.

Plus, we do a lot of segmented notifications, by understanding the sports viewing patterns of different kinds of fans, and sending specific alerts depending on what is of interest to them. Because they’re targeted in this way, the response rate for these notifications is extremely high. So within milliseconds of a notification, we’ve seen three to four million fans come onto Hotstar as a response to our alerts.

To that end, analyzing viewer behavior helps a lot. Mohan shed light on what Hotstar learns from usage data about its audience:

We’ll know, for example, that viewer A is a fan of the Royal Challengers Bangalore (RCB) team, and that if they’re not watching the match, we might draw them in to the stream with an alert about (star batsman) Virat Kohli walking onto the field with his bat.

We know the connections between geography and team affinity. We know whether fans respond to wicket falls (the equivalent of striking out in baseball), or fours and sixes (similar to home runs). We know the kind of fans who come in at the back end of a close finish. There are tens of different parameters on which these notifications are built.

There’s a dedicated group within the retention team that focuses on these segmented notifications. Its entire approach is to use the algorithm for creating customized notifications.

Data also helps prepare Hotstar for spikes in traffic, noted Mohan:

We look at so much data modeling to build for performance. So in terms of cricket, the number of people who are tuning in for the first ball of the match gives you an indication of the curve (of users who will log onto the livestream). Then, you constantly update your traffic estimates based on what happens. With the final match for the last IPL tournament, our data told us that we were headed towards concurrent viewership exceeding 10 million.

Hotstar’s head of technology, Akash Saxena, elaborated on how its apps are built for scale in a blog post about prepping for the IPL:

We build our client applications to be resilient and gracefully degrade. While we maintain a very high degree of availability, we also prepare for the worst by reviewing all the client: server interactions and indicating either gracefully that the servers were experiencing high load or by a variety of panic switches in the infrastructure. These switches indicate to our client applications that they should ease off momentarily, either exponential back-off or sometimes a custom back-off depending on the interaction so as to build jitter (a variation in the delay of received data packets) into the system that provides the back-end infrastructure time to heal.

The team also rebuilt the Hotstar Android app from scratch about five months ahead of the IPL, taking a modular approach as they went, so they could improve functionality for various parts of the software independently.

Next, Hotstar created ‘Game Day’ test scenarios in which it simulated server loads with millions of concurrent viewers – not unlike what the performance team would face when IPL matches were on. Ashutosh Agrawal, a senior technology consultant with the firm, explained that this involved not only developing server test plans, but also running them at the same time of day as the matches. This gave the engineers a taste of what it’d be like to work in tandem to troubleshoot issues across every point of Hotstar’s infrastructure.

That said, a number of Hotstar viewers told TNW that the platform faces occasional issues with sports streams. But it’s clearly focusing on tackling those challenges for as many people as possible. Mohan said that the next big event that will test Hotstar’s capabilities is the India vs. Pakistan match on the Asia Cup roster; the event will take place in the United Arab Emirates in September, and will see one of cricket’s most notable rivalries unfold over roughly eight to 10 hours.

We are pretty clear that if there’s an opportunity to reset our own benchmark for concurrency, that’s going to be the one. We’re now perusing all of the historical data that we have, including that from previous India-Pakistan matches to get a sense of how far can we push the envelope on that match.

It’ll likely become more complex because fundamentally, the experience that we offer today is more than just video. We are going to have a game, and a social experience like we did for the IPL. We’re also working on a couple of other things that we haven’t talked about yet, so we aren’t simply replicating what you saw in the IPL finals.

H/T Mihir Patkar