How Datadog uses GitHub

Datadog taps GitHub Advanced Security to integrate security into developer workflows.

About Datadog

Location: New York, New York
Industry: Cloud Monitoring
Company Size: 2,185

Datadog makes it simple to answer critical questions about cloud infrastructure. The company gathers logs from servers, containers, databases, and other services to shed light on entire infrastructures, allowing teams to monitor application performance and security without guessing.

GitHub as core infrastructure

The Datadog team keeps up-to-date with cutting-edge technologies and development practices. "It would be hard to help customers be more effective in their cloud platforms and DevSecOps practices if we're not practicing that ourselves," Chief Information Security Officer Emilio Escobar says. The company uses Amazon Web Services, Google Cloud Platform, and Microsoft Azure for different parts of their operations, with GitHub serving as an abstraction layer to make deploying to each cloud as simple and uniform as possible.

Because Datadog's software agent is open source, many developers move seamlessly between public and private repositories throughout their day. "We've used GitHub from the inception of Datadog," Escobar says. "It's a high-quality product, and a lot of our engineers contribute to open source so there's a sense of community there. GitHub is one of those products that it's ingrained in the DNA of our engineering, it's become part of the culture."

Integrating security into developer workflows

Datadog is leveraging GitHub Advanced Security seamlessly into developers' existing workflows to ensure code is secure as applications are being built. Security is crucial to Datadog's operations since they handle not only their own internal data, but sensitive data on behalf of customers. "The culture of the company has been built from communication and interactions where security is everyone's responsibility," Escobar says. "Whether you're an engineer or you're a product manager, you have to care about security, just like you have to care about the functionality and quality of the product."

That said, there's a strong need to avoid overburdening developers with too many tools or cumbersome security processes. "We prefer to have security that leverages what developers are already using rather than trying to force them to use some other tool," Escobar says. "That feels interruptive and it always causes friction."

We've used GitHub from the inception of Datadog. It's a high-quality product, and a lot of our engineers contribute to open source so there's a sense of community there. GitHub is one of those products that it's ingrained in the DNA of our engineering, it's become part of the culture.

Managing compliance with GitHub Actions

Datadog uses GitHub Actions to manage changes to "red zones"—particular parts of their products that have strict compliance requirements. Using Actions, Datadog's security team can receive alerts when changes are made and track approvals for those changes. "Only about 2% of those changes require any sort of action, the other 98% are things that get closed because we either knew about them or there's a good rationale for the change," Escobar says. For less sensitive repositories, Datadog is increasingly turning to automated scanning.

Reducing false positives with CodeQL

False positives are a real worry for any scanning tools Datadog deploys, because they can reduce developers' trust in the results. Developers might start ignoring a tool that produces too much noise. Datadog is rolling out new scans gradually with tools like CodeQL to ensure quality results. So far, they're seeing fewer false positives with CodeQL than with the other static analysis tools they're evaluating.

Datadog is also exploring CodeQL as part of their provable security initiative. "We'd like to make facts about our security queryable through Datadog," Escobar explains. "We might be able to put a query, such as the number of times a particular vulnerability has been introduced, into a dashboard."

Protecting customers with Secret Scanning

Datadog uses GitHub not just to improve their own internal security practices, but to help their customers as well. Datadog partnered with GitHub to use Secret Scanning to find and notify customers who had accidentally published their Datadog API keys in public repositories.

"We don't automatically revoke keys because that could break things, but we can communicate with impacted customers and let them know they published something they probably didn't mean to," Escobar says. During the second quarter of 2021, Datadog detected over 250 leaked Datadog API keys. The service enabled Datadog to build an extra layer of trust with their customers by enabling them to remediate issues before they could lead to compromised data.

Shifting left on security

All of these efforts have enabled Datadog to use GitHub to "shift left" on security by weaving security efforts into the day-to-day work of developers, giving them immediate feedback that they can use to improve the quality and security of their code. It's an approach that enables developers to fix problems up-front, instead of waiting weeks or months for a security review. "We want security to be collaborative," Escobar says. "We want to work with engineers and not block productivity."