I’m assuming you are caught up with the original incident. A quick test: what does the number “4” mean to you?

Like literally everyone’s, our infrastructure was impacted, and we pushed a remediation shortly after. This blog post covers what we did, with a short detour on how Chrome trusts certificates, what makes Chrome with Playwright different, and a quick test for you to confirm whether your usage of Playwright is impacted (it probably is, even as of today).

Our infra

Our infrastructure for the scope of this incident covers

  1. Runs infra, which runs thousands of browsers for Playwright tests
  2. Sandboxes that agents use as “computers” to write and maintain Playwright tests and push them to GitHub

This means that if an attacker manages to get into the network path between our infra and either the app-under-test or github.com, they could forge a certificate that causes test user credentials or our GitHub App tokens to get leaked.

Both of these run two runtimes, Node.js (where Playwright runs) and the web browser (where the app-under-test runs). We fixed the former by upgrading to the patched Node.js (v22.23.4 for us), which was shipped and adopted on October 21.

The web browser runtime - essentially, Chrome - was trickier.

How Chrome decides to trust a site

When you open https://example.com in Chrome, the server presents a certificate saying “I am example.com, signed by Intermediate X.” Chrome checks the certificate signature chain, and if the chain ends at a trusted root certificate authority, the connection is allowed to go ahead.

example.com (the website's certificate)
        │
        │ signed by
        ▼
Intermediate certificate authority (key leaked)
        │
        │ signed by
        ▼
Root certificate authority (on Chrome's built-in trusted list)

If any link’s private key leaks, an attacker can sign new certificates at that point in the chain and everything up the chain still vouches for them. In this incident, eMudhra’s intermediate certificates were at risk.

Where does Chrome get trusted certificates?

On Linux, Chrome uses five inputs to know about trusted certificates.

  1. Enterprise policy: certificates an enterprise admin adds or blocks through Chrome’s policy files (FYI: this does not work for all Chrome’s builds: chrome-headless-shell ignores it)
  2. NSS database: a per-user certificate database on disk, and any user can mark certificates as trusted or distrusted here
  3. Chrome Root Store: the public roots that Google vets. This list is compiled into every release, and newer versions can arrive through background downloads
  4. CRLSets: an emergency brake that rejects certificates that Google has blocked. This list arrives through background downloads
  5. Certificate Transparency check: all public certificates carry signed receipts proving they were recorded in public logs. A forged certificate that was never logged has no receipts, so Chrome rejects it

Notice what’s missing: the operating system’s own certificate bundle (/etc/ssl/certs), which is why updating that fixes curl but changes nothing for Chrome. (Also, I hope you’ve done this on macOS, as Apple recommends.)

In this incident, normal users of Chrome relied on CRLSets.

CRLSets: Chrome’s emergency brake

The standard way to revoke a certificate is a CRL (Certificate Revocation List). Every major browser has its form of this (Firefox does OneCRL), and Chrome calls theirs CRLSets. CRLSets arrive through Chrome’s component updater, a background channel that downloads data files without a browser update or restart.

Chrome used CRLSets to block the leaked keys on October 17, about two hours after the disclosure.

Why Playwright misses this

Playwright launches Chromium with --disable-component-update and a brand-new user profile every time. This ensures deterministic test behavior, as nothing changes underneath as your tests run. To strengthen this determinism guarantee, Playwright versions are also coupled with browser versions, and it’s not a good idea to move them independently.

However, this means Playwright’s bundled browser cannot rely on CRLSets, and depends on the Chrome Root Store compiled into the build. Chrome is shipping the eMudhra removals with their 158 builds, which is what the next version of Playwright will support.

If you are on current Playwright stable (v1.64.1) or older, you are at risk.

What we did: NSS database

The NSS database is usually used to add trust, for example, for a corporate CA. Thankfully, it also works the other way, and can be used to mark certificates as explicitly distrusted. The NSS database belongs to the Linux user, and not the Chrome user profile, so it works with Playwright’s ephemeral user profiles.

To ship our fix, we distrusted 17 certificates: the eight compromised intermediates, their eight Sectigo cross-signs from impacted-intermediates.json, and the root that Chrome eventually removed. This fix went into our infra and sandbox images.

Test your own setup

Add this to your Playwright suite to confirm whether you are impacted. On current stable Playwright v1.64.1, two of the five tests fail (dv-g3b and dv-g3b-noct), and your test browser will accept a forged certificate.

import { expect, test } from "@playwright/test";

const compromisedHosts = [
  "https://dv-g3b-emudhra.we-have-a-problem.com/",
  "https://dv-g3b-aaa-emudhra.we-have-a-problem.com/",
  "https://dv-g3b-noct-emudhra.we-have-a-problem.com/",
  "https://dv-g3b-aaa-noct-emudhra.we-have-a-problem.com/",
];

for (const url of compromisedHosts) {
  test(`rejects: ${url}`, async ({ page }) => {
    await expect(page.goto(url)).rejects.toThrow(
      /net::ERR_CERT_(AUTHORITY_INVALID|REVOKED)/,
    );
  });
}

test("still trusts a normal site", async ({ page }) => {
  const response = await page.goto("https://example.com/");
  expect(response?.ok()).toBe(true);
});