Browser Engine Patcher

Home/Blog/Reproducible builds

Reproducible builds: what we pin, and what we stopped pinning

2026-09-24Build engineering9 minutes

A build is reproducible when the same inputs produce the same outputs, byte for byte, on another machine at another time. The inputs are easy to list and easy to pin. The pipeline is the part that hides state, and it took us a year to stop pretending otherwise.

Four things we pin

  1. The source revision. A commit id, not a branch and not a tag. Tags move; commits do not. The revision lives in bep.toml and is copied into bep.lock at fetch time.
  2. The patch series. Every patch hash, in order. A patch edited after the lock was written is a modified input, and the tool says so instead of applying it quietly.
  3. The compile flags that change output. Optimization level, debug info and the target triple. Not the job count: parallelism changes timing, not bytes.
  4. The clock. Build systems embed timestamps in archives and generated headers. SOURCE_DATE_EPOCH is set from the commit date unless the environment overrides it, and the effective value is printed in the build summary and stored in the lock.

What we stopped pinning, and why

The first lockfile recorded absolute paths of the compiler and the working tree. That looked thorough and was actively harmful: two machines with the same toolchain produced different hashes because /home/ana and /build/agent-3 are different strings, and those strings end up in debug info. We now pin the compiler's identity — version banner and binary hash — and let paths differ, with one rule: the build must not embed the working directory. The two flags that control that are in the catalogue's build-flags patch, where they can be reviewed.

The second thing we stopped pinning is the fetch transport. A mirror URL is an input to the fetch, not to the build; recording it in the lock made a mirror move look like an output change. The lock keeps a hash of the fetched tree instead, which is the fact we actually care about.

The cache key that cost a day

Until 2.8.0 the object cache was keyed by the path of the fetched tree plus its modification time. Both are proxy facts. A rename invalidated 30 GB of objects; a touch did the same; and on a shared build server one team's cleanup invalidated another team's cache without either noticing until the next build took an hour.

# before 2.8.0
key = sha256(path + str(mtime) + patch_list)

# after 2.8.0
key = sha256(revision + fetch_command + patch_hashes + target)

The new key costs one pass over the patch files — milliseconds — and removed a category of support reports rather than an instance of one. This is the general shape of a reproducibility bug: a fact that is true locally is used as if it were true globally.

Verification is not a rebuild

bep verify re-applies the series to a clean copy of the sources and compares file hashes with the manifest written by the last build. It catches the common failure — an edit outside the series, a line ending normalised by an editor, a generated file that changed — in seconds. It does not prove the binaries match: that needs a second build on another machine and a comparison of the outputs.

We run that second build for every release, on a different distribution and a different compiler minor version from the one used to cut the archive. The outputs differ in debug sections by design (compiler version strings), so the check compares section hashes for the code sections only, and a human reads the diff when it fails. It has failed three times: once because of an embedded timestamp, once because of a locale-dependent sort in a generated file, and once because a patch read the hostname. All three were real problems that a single-machine check would have missed.

What a user should take from this

The checklist we use before calling a release reproducible is in the repository next to the release script, and it is short enough to read in five minutes. Start with Quick start if you want to see the pins in a real series, and read the CLI reference for what verify does and does not check.