About Bleacher Report

Bleacher Report (often abbreviated as B/R) is a global digital destination for millennial sports fans creating and collaborating on content at the intersection of sports and culture.

How did it start?

The story begins with a good dream, an angry clock, and a cup of coffee. Ben had pager duty, and the site wasn’t working. Let’s go back to the beginning.

Bleacher Report is the second biggest sports platform in the world and quickly grew to that size by offering sports fans something that no one else did: personalization and rapid content production, while following local to international sports. Most early users accessed the site through a computer or from newsletters that would, in turn, lead them back to the site.

In general, it was a passive model. We had to go to the site to get the information. Since most pages were static, they could cache them for performance. Spikes were relatively gentle, even when news broke.

Mobile changes the rules of the game

Almost overnight, that landscape changed. When the iPhone exploded, companies quickly developed applications (apps) that transformed the way customers interact with the web. Suddenly, when news broke, push notifications or emails would send everyone to their favorite sports site at once. Spikes were instant and violently sharp.

It’s hard to express just how much impact this kind of change could have. Say a famous star like Kevin Durant, one of the most famous basketball players in the world, suddenly gets traded to the San Francisco area. The news site would push out 10 million notifications. If only 5% clicked, they’d have an instant spike of a half million users, and the site couldn’t handle it.

With a mobile phone in everyone’s pocket, the definition of what was major news changed too. Now, the end of a good game or an upset became major news. While this shift certainly wasn’t limited to Bleacher Report, it hit them particularly hard. They knew they had to do something, but deciding exactly what to do wasn’t easy.

Early responses to the challenge

Bleacher Report took two primary steps to overcome the challenges. Let’s see what those were.

Use of hardware

At first, the sports news publisher did what everyone does. They threw hardware at the problem. For a while, it worked. Then, an unexpected guest named “Success” crashed the party. As more sports fans found the site, the cost for scaling through hardware went up too quickly to absorb. Soon, the site was running on over a hundred servers. To save money, the team introduced auto-scaling to add servers on demand. That too failed because ramping up a new environment takes time, and they were failing users at peak demands.

Caching

They had to make their next logical play. They cached. The early results were promising, serving content many times faster. At this point, a guest named “Personalization” slammed through the door without an invitation. As athletes published their own Twitter feeds and they started showing highly variable content like scores, caching made less sense because each user saw something different. Imagine the user subscribes to college basketball, international soccer, major league baseball, and a couple of the streams that are topic focused. We simply can’t cache that. Increasingly, the scrambling software engineers decided to only build pages based on what they could cache, drastically hurting personalization.

They realized that caching was a deal with the devil, and he had collected his price: the very innovation that made Bleacher Report great.

Shift to Elixir

So, Bleacher Report developers were on call because their existing technology stack was not up to the problems the next few years would throw at it. The industry was changing, the servers were overloaded, and the developers were going as hard as they could go. Something had to give.

At some point, the development leadership and management teams both agreed that they had to take some bitter medicine to get better. For many of the same reasons as icanmakeitbetter, they settled on Elixir: they wanted scalability and reliability without compromising productivity. And with that decision made, they set out to find some problems to prove their hypothesis.

Establishing early wins

Matt Pruitt and Michael Schäfermeyer were the Elixir advocates at Bleacher Report, who first championed the language. They were able to quickly develop a proof of concept. It was still a prototype when Ben came to Bleacher Report but they’d done the hard work of demonstrating its viability to the company.

Ben first worked on a tiny service that fetched titles, descriptions, and the like from external services. By fetching all of the metadata concurrently instead of doing each request sequentially, his tiny team reduced the response time dramatically. The simple service took a day to prototype, a few more to finish out and deploy, and solved a key business need.

Neither prototype was aesthetically pleasing, but both were conceptually beautiful. The code was explicit, concurrent, and well organized, running much faster with better stability than the code it replaced. They’d established a few quick political wins.

Based on the early success, the anxious but excited initial group made a firm commitment to Elixir and sought ways to expand on their initial successes and kept chipping away. It wasn’t always easy. Those were the early days of the Elixir community. Elixir and Phoenix, the dominant web framework in the community, were changing rapidly, so early code had a great deal of churn. Elixir’s flexibility for rapid prototyping mitigated the damage somewhat, and the team was eager to rewrite early attempts to more idiomatic Elixir.

Over time, Bleacher Report increased the scope of their prototypes and moved more substantial pieces of their application into production. Elixir was improving both scale and reliability, one bottleneck at a time. The developers gathered confidence. Management and customers also started to notice.

Eventually, over four years, they’d moved the bulk of their system. The benefits weren’t just tangible. They were transforming.

Enjoying the benefits on draft day

For the first time in years, the platform was stable and response times were largely traffic independent. The graph below shows the number of requests (y) over time (x) for the sports site’s busiest week of the year, the NFL draft for 2017, after the migration. The massive peaks and valleys are the first three draft rounds. The specific details are proprietary, but it’s the highest number of concurrent users in their history:

Traffic is higher than usual throughout the week because of the news coming out about the draft. They expected to see a relative increase in traffic on the first nights of the draft, but they didn’t expect to break their highest number of concurrent users by almost 30,000!

In the past, when traffic would ebb and flow, there would be corresponding, sustained fluctuations in the response times. This NFL draft also happened to be the first NFL draft where the majority of Bleacher Report’s stack was Elixir. The graph below shows how the Elixir app API gateway that serves their client apps, iOS, Android, and web, fared during this record-breaking time.

Now, for the kicker. Let’s look at the response times in seconds (y) against time (x):

The view from management

We’ve heard some anecdotes from the Bleacher Report rank and file. Sometimes, a manager’s words can be more convincing. Let’s talk to Dave Marks, the Senior Engineering Director at Bleacher Report. He’ll tell us about Elixir adoption from a senior manager’s perspective:

That’s powerful stuff. A development director’s two biggest goals for cutting costs are often at odds. Choosing to emphasize development productivity often reduces the size and cost of the development staff. Sometimes, this thrust comes at the cost of performance, as it once did at Bleacher Report. This project, though, shows a simplification of the development stack and better development buy-in. It reduced the costs of feature development and made it easier to bring in new staff when it’s necessary.

Questions and answers about development

The Bleacher Report story is a struggle that took several years. They successfully waded through the politics we identified in the first story, setting expectations and building consensus. They identified a problem that was too big for their current technology stack and solved it by establishing some quick wins. Then, they methodically migrated the rest of their system, piece by piece. Though they’re still not done, they’re close enough to reap major rewards of more effective development staff, better stability, and excellent performance. Getting to that point was hard. Ben made it his mission to write articles and speak at conferences to tell others about questions Bleacher Report developed through their process:

  • With a larger migration, how do we decide what to do first?
  • How do we train a large team on Elixir?
  • How do we address a legacy monolith?
  • How do we maintain good code quality with measurable heuristics for quality given an inexperienced team?

Ben and Bruce met at a conference in Mexico City and decided to write a course, “Adopting Elixir,” about their growing number of questions surrounding adoption. Bleacher Report is the most pervasive adoption story in this course. We’ll help you identify the questions you should be asking as you develop your application. We’ll then help you build some intuition around potential solutions and point you toward tools and techniques that can help you find the answers that you seek.