Home/Blog/Testing

How we test a patch series before it ships

2026-04-15Testing8 minutes

Four checks run on every change to the patch catalogue. None of them is clever. Three exist because of a specific bug that reached a release, and the fourth because a reviewer asked a question we could not answer.

Check one: the header lint

bep series lint runs in under a second and catches the boring failures: a missing base, a duplicate id, a touches prefix that does not exist in the tree, a dependency cycle. It exists because a patch with a typo in family was accepted by 1.3.0 and then disabled itself at apply time. Nobody noticed for two releases.

$ bep series lint
patches/0100-build-flags.patch     ok
patches/0200-debug-hooks.patch     ok  requires build-flags
patches/0300-devtools.patch        error  touches a path that is not in revision 8f2c41d9
3 patches, 1 error

Check two: apply on a clean tree

The lint cannot see whether a hunk applies. Every catalogue change is applied to a freshly fetched tree in CI, twice: once with default settings, once with series.optional = false. The second run is the one that matters, because an optional patch that silently stops applying is a feature disappearing from a build without a message.

This check caught the CRLF bug fixed in 2.8.1 only after it shipped, because our CI runs on Linux and checks out with LF. The fix was not a better test; it was adding a Windows runner that applies the catalogue to a checkout made with core.autocrlf=true. Testing the platform you ship to is not optional, and we had been treating WSL2 as a stand-in for Windows.

Check three: a build, twice

A full build takes 70 minutes, so it does not run on every commit. It runs on every release candidate, on three machines, and the second run on each machine must produce byte-identical code sections. A patch that adds a timestamp, reads the hostname or sorts a list by locale fails here. Three have so far, all found before the release.

Check four: the five-minute install

Someone who did not write the change follows the install page on a clean virtual machine, with no shell history and no compiler caches. The rule is five minutes from a fresh terminal to bep doctor reporting a ready machine, excluding the engine fetch. If it takes longer, the page is wrong, not the reader.

This check exists because a reviewer asked, in a documentation pull request, “how long does this take?” and nobody knew. The first run took nineteen minutes: a missing pkg-config was mentioned three paragraphs below the command that needed it. Moving one line fixed it.

What we do not test

• Every engine revision. We test the revision pinned by each supported series, not the moving head of upstream. A patch that applies to head today may not tomorrow; that is what base is for.
• Every Linux distribution. Ubuntu 24.04, Fedora 41 and Arch cover the three package layouts we see in reports. A glibc-specific failure on an older distribution is fixed when reported, not pre-emptively.
• Interactive behaviour of the patched engine. We test that the build is what the series says, not that the resulting browser behaves well; that belongs to the team using it.

The release checklist

1. Lint passes with zero warnings, on the release branch.
2. Apply passes on Linux, macOS and Windows checkouts, with optional patches mandatory.
3. Two builds per machine match in code sections.
4. The install page works on a clean machine in under five minutes.
5. The release notes name every merged patch and every known issue, including the ones we decided not to fix.

The checklist is a file in the repository, not a ritual in somebody's head. When a release ships with a known issue, the issue is written into the notes with the words “known issue”, as in the 2.8.1 notes, so a reader can find it without reading the tracker.