How BuzzFeed uses GitHub

BuzzFeed stays on the pulse with GitHub and DevOps-ready workflows.

Industry: Digital Media
Location: New York, NY
GitHub Seats: 220

Overview

For hundreds of millions of users, BuzzFeed is the leading digital media publisher—a first stop for breaking news, entertainment, quizzes, and lifestyle content. Managing a cross-platform network of articles, lists, videos, original series, and world-class reporting requires flexible infrastructure that supports multiple deployments per day.

Behind BuzzFeed.com is a distributed team of developers and engineers located from California to London to Argentina. Tech users, IT teams, and BuzzFeed News reporters all build and run code together.

From Manual to Automated Workflows

At the start, these teams deployed manually, pushing code to production themselves. As the number of deployments multiplied, engineers spent more time on frustrating manual tasks and less time building tools that mattered.

"The mindset we carry within our team is that we always want to automate ourselves into a better job," explained Director of Engineering Andrew Mulholland. "We want to make sure that the task we're doing manually today becomes mostly automated. We focus on building tools that serve as an enabler for developers."

With an automation-first approach, BuzzFeed turned to GitHub Enterprise, which had already been used within the company by developers and journalists for years. It quickly became the foundation for a modern development workflow built on collaboration and monorepos.

"One of the great things about BuzzFeed's culture is that, if you see something broken, and you offer up a pull request to fix it, people will be happy to review the code and approve it," Mulholland said. "Monorepos make that even easier. You have access to all code by default and this serves as an enabler for learning."

Transparency Across the Organization

Beyond the tech team, users across the organization have access to BuzzFeed's GitHub, including members of the BuzzFeed News team. The same transparency that applies to code applies to reporting as well: data journalists publish their data to a public GitHub repository, whether it's Jupyter Notebooks or stand-alone code.

"We want to try to show our work as much as possible and not take readers' trust for granted," Mulholland said. "When we've done analysis for a story, we make the tools that we used for that analysis available so people can run and verify it if they want to."

Innersourcing and Code Reuse

With GitHub, the process of open code by default is seamless. Anyone within BuzzFeed's GitHub organization can open a feature request in another service—a practice known as innersourcing. This approach has allowed BuzzFeed developers to contribute to improving the company's infrastructure.

For example, when an engineer needed custom autoscaling, the infrastructure team wasn't available to develop it. But since the developer could access the team's code, they were able to self-service and suggest code changes themselves. "They came to us with some requests for improvements, and we said that if they helped, we'd be happy to accept that code. So they added the code in, and it's now widely used across dozens of different services."

Rapid Response to Outages

Every second counts in an outage—especially for one of the world's most-visited websites. In one instance, a service outage occurred while the original development team wasn't available. The problem was quickly resolved because the application was in a monorepo in GitHub with high standardization between services. An infrastructure engineer was familiar enough to make changes and solve the issue. The next day, the original team approved and merged the changes.

Local Development and CI/CD

One of the most important roles for the infrastructure team is serving internal developers. Local development environments are key to that. "Being able to have developers bring up a service locally on their machines is crucial to ensuring a quick service when they run it," Mulholland said.

BuzzFeed created libraries and templates for its supported languages—Python, Golang, and Node—to enable developers to quickly start building locally. The team used GitHub as a core part of their highly-automated CI/CD workflows. Now, developers can immediately package and push code.

"As soon as your branch is pushed up to GitHub, that will kick off a build in our CI system. Then once it has been reviewed and merged, it will be automatically deployed to production."

By moving from manual deployments to automation with GitHub, BuzzFeed's teams deploy 30 percent more frequently, sometimes up to 300 times per day. This freed up bandwidth for other top priorities—including security and open source.

Open Source and Community Contribution

Open source is important to BuzzFeed. As a user of many open source technologies, the company wanted to give back to the community. In 2018, BuzzFeed open sourced its single sign-on (SSO) solution.

Before sharing the SSO project as open source on GitHub, the team worked to improve the readability and structure of its code. "When you're under pressure to get things done, there's an element of technical debt that creeps up," Mulholland explained. "We had our SSO project internal for a year before making it open source. Since making it open source meant making the code public, it forced us to raise the bar and fix our code internally, and created a great learning opportunity for our developers."

Participating in open source brings better code and credibility. "A lot of people obviously know us for our buzzy content, quizzes, massive food brand Tasty, and BuzzFeed News, but now people understand that, 'Oh yeah, there's good software engineering happening there as well.' That's been very helpful when we've been hiring."

Security and Bug Bounty Program

By making the code to one of BuzzFeed's most critical projects open source, it became eligible for the company's bug bounty program. "With SSO being open source it's easier for researchers to find issues—and that's a good thing," Mulholland said. "We then encourage the use of our bug bounty program to ensure we can reward researchers for responsible disclosure as we triage and remediate it."

Initially, the bug bounty program was a struggle. The all-in approach created another backlog: "In theory everyone was responsible for the triage of tickets, but by having everyone responsible, it meant that really no one was responsible. Often non-critical reports took a long time to be triaged."

Having worked at companies with large security organizations, Mulholland knew a siloed approach didn't fit BuzzFeed's values. "It creates a very much us-versus-them mentality, and that doesn't fit well how we do things. We wanted to create a more collaborative, inclusive approach."

Instead, BuzzFeed built a security program uniquely suited to the company, with a rotation plan that assigned an engineer from a different team each week to be on triage. "Our mindset was that by sharing the burden across the tech organization, it wouldn't be too much of a burden to any individual or any one team."

The rotation worked—someone was always responsible, and the mean time to triage and bounty dropped from weeks to days. The improved process also brought in more bug reports. "Suddenly we were getting way more reports because security researchers got the signal that we care about security, and we were paying bounties quickly. They realized it was worth their while, so they spent more time looking at BuzzFeed." Mulholland and his team even began working with researchers periodically, inviting them to preview new features for potential vulnerabilities.

Modernizing the Frontend

Using GitHub has introduced more secure features into BuzzFeed's infrastructure and enabled significant changes. The team recently migrated BuzzFeed.com article pages onto a new React-based front end. "GitHub played a key role in that, with its collaborative features being a boon to development," Mulholland noted. "By moving to the React front end, it's more modular. It's going to enable us to work to evolve the user experience to make it easier to find the content that you want to read, and easier for us to enable new features to put around the reading experience."

Impact

"GitHub has become so essential to the developer experience," Mulholland said. "If we can keep enabling our engineers like we did using continuous deployments with GitHub, it saves our team a considerable amount of time. And those time savings are a boost to productivity."