How Postmates uses GitHub

Postmates uses GitHub Advanced Security to automatically surface vulnerabilities before hackers do.

Location: San Francisco, CA
Industry: Online Food Delivery
Customer Since: 2016

The Challenge

In 2020, COVID-19 transformed Postmates from a convenience service into an essential lifeline. The company now delivers from more than 600,000 merchants across over 4,200 U.S. cities. With so many variables and an increasingly complex security landscape, Postmates is focused on keeping all users safe—customers, merchants, and couriers alike.

A small team of engineers manages security across two areas: Platform Security, which focuses on network security policies, and AppSec, which performs security reviews and handles the bug bounty program. Despite their broad coverage, the team is lean, often leaving security professionals feeling like emergency responders.

Staff Security Engineer David Ross recognized that performing static analysis on code to systematically find bugs was the missing piece. Some merchants Postmates partnered with wanted assurance that the company was taking every precaution possible. "We clearly needed to find a solution," said Ross.

Evaluating Solutions

Ross had considered static analysis tools before, but the options he evaluated either didn't support the languages in Postmates' codebase or didn't meet his expectations. "One criteria we needed was around language support: Python, JavaScript, Java, TypeScript, and Go. That limited us," he explained. After reviewing results from SonarQube and Veracode, he wasn't convinced. "I've seen static analysis tools uncover really interesting bugs—I just wasn't impressed by what these two found."

Many tools felt like "black boxes" to Ross. "You let it do its thing, analyze your codebase, and the results may or may not be useful." When he discovered he could write custom queries with CodeQL, he was intrigued. "We started a proof of concept for Advanced Security. We wanted to prove it out, but the flexibility of CodeQL looked really promising."

Finding Variants Automatically

For Postmates' security team, CodeQL's real value lies in customization and automation. Finding one issue is a start, but upon closer inspection, it can unfold into many more variants. A third-party researcher might find an issue, but the same one—or a variant of it—can exist elsewhere in the codebase.

Hunting down these variants is typically a tedious, manual process requiring deep knowledge of the codebase and development team habits. For example, if an issue appears in Postmates' buyer app, Ross knows he might also find it in the merchant and courier apps. "You'll often see the same issue in every code base," he said.

CodeQL adds value by finding variants coming through other channels and in other products. It can track data from source to sink.

With CodeQL queries, Postmates can more consistently and automatically find variants. "CodeQL adds value by finding variants coming through other channels and in other products," said Ross. "It can track data from source to sink, which is particularly useful for things like cross-site scripting vulnerabilities: you can follow their path through your code. It's possible to find these things by searching certain strings, but to find a lot of them, you need something a bit better—and I haven't seen anything else like CodeQL."

Standardizing Security Practices

Writing queries has also leveled up the standardization, consistency, and rigor of Postmates' security standards. One-off issues can signal less-than-perfect practices, like checking in hard-coded passwords or disabling certificate validation. In the query results, Ross can see the holistic impact of these problems and where they originated. If any team's specific actions introduce a vulnerability, he can show them a better way—helping developers learn security best practices while writing higher quality code.

The Postmates team feels they're only scratching the surface of this approach. CodeQL comes with an open source repository containing thousands of queries. "The base of queries is only expanding, not just through GitHub but through other companies' expertise," says Ross. "It makes the potential for the value that we get, just at static analysis, to be much greater than it is today."

For the team, this potential lies in robust sets of CodeQL queries customized to issues unique to Postmates' codebase. Running a larger set of queries on every pull request makes it possible to automatically catch any instance of an issue before it's checked into the code.

Dependency and Secret Management

Postmates engineers have also found Dependabot and secret scanning helpful in finding crucial issues to focus on. Ross has them enabled on every repository in the organization, as well as newly created repositories. "We found a ton of important things to address," he said. "On the AppSec side, it's often the best way for us to get visibility into issues in the code."

Vulnerability alerts and automated pull requests help the team better understand and more easily update their open source dependencies. "For modern startups, everything is open source," said Ross. "Closed source software can sit for years with critical issues." But actively maintained open source projects surface and fix those issues at a rapid rate. "Even if you wind up updating your dependencies frequently, it means vulnerabilities are being found that otherwise wouldn't be," he says. "It's a good problem to have."

Integrated Tracking and Triage

Postmates' automated approach to security extends to their triage and tracking processes. They use the GitHub API to aggregate and collect issues identified by CodeQL and create Jira tickets to track these issues. These entries are added to a tool called ZenGRC that manages all compliance issues, creating tasks to track and automatically ping developers. Low false positive rates are key to helping engineers avoid unnecessary work.

ZenGRC also pulls in secret scanning and dependency graph issues, making it easier to track necessary updates to repositories and vulnerable dependencies found by GitHub via Dependabot. Even without a Product Manager, the team has been able to tackle easy fixes and "low-hanging fruit" alongside the most pressing updates.

Shifting Security Left

Developers and security teams are often pitted against each other, but Postmates' security team is thoughtful about how they engage engineers. "We're here to help," said Ross, "Not slow them down."

Part of this means fitting security into the development cycle as organically and as early as possible—sometimes called "shifting security left." Using GitHub Actions, the team ensures that whenever there's a push to the main branch or someone checks in code, it's scanned. It's also scanned once a week, regardless. This way, engineers can find bugs as they code, rather than at the end when issues become more time intensive to resolve.

Continuous Improvement

CodeQL is now helping the team find vulnerabilities they'd never found before, but an objective end-state of "most secure" doesn't exist for security professionals. Because of the nature of threats, vulnerabilities, and open source software, it's necessary for teams to continuously take stock of their processes and improve.

Postmates is currently working toward a comprehensive security program running from report to remediation. Ross sees the progress they've made so far as an important part of a much longer journey. "Advanced Security and CodeQL makes this possible," he said.