Caching Nix Builds for Faster Team Onboarding

Stop new laptops compiling Nix packages from source for an hour: push dev-shell closures to a team binary cache from CI and configure substituters and trusted keys.

A new engineer runs nix develop on their first morning and watches building '/nix/store/…-custom-protoc-plugin-1.4.0.drv' scroll for fifty minutes, followed by an hour compiling a patched OpenSSL. Every other developer did the same on their first day, and CI does it on every cold runner. The public cache at cache.nixos.org only contains packages built from unmodified nixpkgs; anything custom, overridden or from a non-default input is built locally. This page builds those packages once in CI and serves them to everyone from a team binary cache, as part of reproducible dev shells with Nix, Devbox and direnv.

The payoff is usually dramatic: first-run time on a fresh laptop drops from about an hour to a few minutes, which is a direct improvement to time-to-first-PR.

Diagnostic

List which store paths in the dev shell's closure are missing from the configured caches — those are what a new machine builds from source:

#!/usr/bin/env bash
set -euo pipefail
sys=$(nix eval --impure --raw --expr builtins.currentSystem)
drv=$(nix eval --raw ".#devShells.$sys.default.drvPath")
nix-store --query --requisites "$drv" | grep '\.drv$' > /tmp/drvs.txt
nix build --dry-run ".#devShells.$sys.default" 2>&1 | sed -n '/will be built/,/will be fetched/p' | head -12
nix config show | grep -E '^(substituters|trusted-public-keys)'

Expected bad output:

these 7 derivations will be built:
  /nix/store/2x9…-openssl-3.0.14-patched.drv
  /nix/store/8kq…-custom-protoc-plugin-1.4.0.drv
  /nix/store/c1m…-terraform-provider-internal-0.9.2.drv
  /nix/store/jz0…-nix-shell-env.drv
these 312 paths will be fetched (1.21 GiB download, 5.02 GiB unpacked):
substituters = https://cache.nixos.org/
trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=

312 paths come from the public cache, but seven — including a patched OpenSSL, which drags in a long rebuild — must be built locally, and only the public cache is configured.

Where First-Run Time Goes Bar chart splitting first nix develop time into downloads from the public cache and local source builds. Where First-Run Time Goes 312 cached paths 4 min patched openssl 38 min custom protoc plugin 9 min other local builds 6 min
The seven local builds dominate; the 312 cached paths download in minutes.

Root cause

Nix decides per store path whether to download or build. Before building, it asks each configured substituter (binary cache) whether it has that exact path, identified by the hash of all its inputs. cache.nixos.org is populated by Hydra, the nixpkgs build farm, so it has every package built from an unmodified nixpkgs revision — but anything with a changed input has a different hash and is not there. Overlays, overrideAttrs, custom derivations, pinned private sources and packages from a non-standard nixpkgs revision all fall into that category. Worse, overriding a low-level dependency such as OpenSSL changes the hash of everything that depends on it, turning a single patch into dozens of local rebuilds. Without a team cache, each machine repeats all of that work independently.

The cache also has a trust dimension. Nix only accepts downloaded paths signed by a key listed in trusted-public-keys, so adding a cache means distributing its public key, and on multi-user installs only trusted users can add substituters. That is why a cache someone "set up" still does not help if the key never reached developers' nix.conf.

Resolution

  1. Create a cache. Cachix is the quickest hosted option (cachix create shop-dev); self-hosted options include an S3 bucket with nix copy --to s3://… or Attic. The steps below use Cachix.

  2. Push the dev-shell closure from CI on every change to flake.lock or flake.nix, so the cache is warm before developers pull:

name: nix-cache
on:
  push:
    branches: [main]
    paths: [flake.nix, flake.lock]
jobs:
  cache:
    strategy:
      matrix:
        os: [ubuntu-24.04, macos-14]
    runs-on: ${{ matrix.os }}
    steps:
      - uses: actions/checkout@v4
      - uses: DeterminateSystems/nix-installer-action@v14
      - uses: cachix/cachix-action@v15
        with:
          name: shop-dev
          authToken: ${{ secrets.CACHIX_AUTH_TOKEN }}
      - run: nix build --no-link .#devShells.$(nix eval --impure --raw --expr builtins.currentSystem).default

The cachix-action watches the store and pushes every path built during the job. The matrix covers Linux and Apple Silicon, since cached paths are per system.

  1. Configure developers' machines to use the cache. Put the settings in the flake so Nix offers them on first use:
{
  nixConfig = {
    extra-substituters = [ "https://shop-dev.cachix.org" ];
    extra-trusted-public-keys = [ "shop-dev.cachix.org-1:Xb4mLq2kH9fT3sVwYc8dRn1pJ6uA0eZ5gK7iO2yQ4tM=" ];
  };
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-24.05";
  outputs = { self, nixpkgs }: { };
}

On multi-user installs, nixConfig settings are applied only for trusted users. Add developers to trusted-users in /etc/nix/nix.conf during bootstrap, or write the substituter and key into /etc/nix/nix.custom.conf directly:

#!/usr/bin/env bash
set -euo pipefail
conf=/etc/nix/nix.custom.conf
sudo touch "$conf"
grep -q shop-dev.cachix.org "$conf" || sudo tee -a "$conf" >/dev/null <<'EOF'
extra-substituters = https://shop-dev.cachix.org
extra-trusted-public-keys = shop-dev.cachix.org-1:Xb4mLq2kH9fT3sVwYc8dRn1pJ6uA0eZ5gK7iO2yQ4tM=
EOF
sudo launchctl kickstart -k system/org.nixos.nix-daemon 2>/dev/null || sudo systemctl restart nix-daemon
  1. Reduce what needs caching. Before caching a patched OpenSSL, ask whether the patch is needed in the dev shell at all. Overriding low-level libraries multiplies rebuilds; scoping the override to the one package that needs it keeps everything else on the public cache.
Build Once in CI, Download Everywhere Flow from a lockfile change through a CI build and cache push to developers downloading prebuilt paths. Build Once in CI, Download Everywhere flake.lock change merged to main CI builds shell Linux and macOS push to cache signed paths laptops download no local builds
Each custom path is built once per system instead of once per laptop.

Expected output

$ nix build --dry-run .#devShells.aarch64-darwin.default 2>&1 | head -3
these 319 paths will be fetched (1.34 GiB download, 5.41 GiB unpacked):
  /nix/store/2x9…-openssl-3.0.14-patched
  /nix/store/8kq…-custom-protoc-plugin-1.4.0
$ time nix develop --command true
real    4m12s

No derivations are listed under "will be built"; everything is fetched, and a fresh laptop is ready in about four minutes instead of an hour.

Prevention

  1. Fail CI when the dev shell would build locally. Run nix build --dry-run on a clean runner with only the configured caches and fail if the "will be built" list is non-empty after the cache job has run. That catches a cache job that stopped pushing.

  2. Rotate the cache auth token like any other CI secret; only CI needs write access. Developers need only the public key.

  3. Track first-run time as an onboarding metric. A sudden jump means something stopped being cached.

Public Cache Only vs Team Cache Comparison of relying on cache.nixos.org alone against adding a team binary cache. Public Cache Only vs Team Cache public cache only plus team cache custom paths built per laptop built once in CI first run about an hour first run minutes CI rebuilds on cold runners CI downloads too nothing to operate token and key to manage
The team cache matters in proportion to how much the flake customises nixpkgs.

Platform caveats

macOS: the Nix daemon must be restarted after editing nix.conf or nix.custom.conf; use launchctl kickstart -k system/org.nixos.nix-daemon.

Apple Silicon (ARM64): cached paths are per system. A cache populated only by Linux runners gives macOS developers nothing; include a macos-14 (arm64) runner in the cache job.

WSL2: behaves like Linux. If systemd is not enabled in the distribution, restart the daemon manually or use a single-user install, where nix.conf in the home directory applies immediately.

Rollback

Remove the substituter lines; Nix falls back to the public cache and local builds:

#!/usr/bin/env bash
set -euo pipefail
sudo sed -i.bak '/shop-dev.cachix.org/d' /etc/nix/nix.custom.conf
sudo systemctl restart nix-daemon 2>/dev/null || sudo launchctl kickstart -k system/org.nixos.nix-daemon

Frequently Asked Questions

Why does Nix ignore the nixConfig substituters in my flake?

On multi-user installs, only users listed in trusted-users may add substituters, and Nix prompts or ignores the setting otherwise. Add the cache directly to the system nix.conf during bootstrap, or add developers to trusted-users.

Can a team cache serve stale or malicious binaries?

Paths are identified by the hash of their inputs and signed. Nix only accepts a path whose signature matches a trusted public key, and the path name encodes its inputs, so a cache cannot substitute a different build for the same inputs without the signing key.

How much does a team cache store?

Only paths not in the public cache: typically a few hundred megabytes to a few gigabytes per system. Hosted caches offer retention policies; self-hosted S3 caches can use lifecycle rules.

Is magic-nix-cache in CI a replacement for a team cache?

It caches store paths between runs of the same repository's CI, which speeds up CI but does not help developer laptops. A team cache serves both.