Testing Onboarding Instructions in a Clean Container

A new hire follows the README exactly and step four fails with make: *** No rule to make target 'db-seed'. Stop. — the target was renamed to db:seed three weeks ago and nobody who already had a working setup noticed. Setup instructions rot because the only people who run them from scratch are new hires, a few times a year. This page runs the onboarding path in a clean container automatically, on every change to the files it depends on and nightly, so broken instructions fail CI instead of someone's first day. It is part of README-driven automation.

Two things get tested: the one-command bootstrap target, and the shell code blocks in the README itself, so the documentation and the tooling cannot drift apart silently.

Diagnostic

Try the documented path in a clean container right now and see where it breaks:

#!/usr/bin/env bash
set -euo pipefail
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v "$PWD":/src:ro ubuntu:24.04 bash -euo pipefail -c '
  apt-get update -qq && apt-get install -y -qq git make curl docker.io >/dev/null
  git clone -q /src /work && cd /work
  awk "/^## Setup/,/^## [^S]/" README.md | awk "/^\`\`\`bash/{f=1;next} /^\`\`\`/{f=0} f" > /tmp/setup.sh
  cat /tmp/setup.sh
  bash -euo pipefail /tmp/setup.sh
' 2>&1 | tail -8

Expected bad output:

make bootstrap
make db-seed
make: *** No rule to make target 'db-seed'.  Stop.

The README's Setup section still calls a target that no longer exists. Every engineer with an existing environment skips that step, so only a clean run finds it.

Root cause

Onboarding instructions are the one part of a repository whose users change every time: each new hire runs them once, on a machine with no prior state, and then never again. Everyone who could notice a broken step already has a working environment and skips it. Changes to Makefiles, scripts, Compose files and tool versions affect the setup path without anyone re-reading the README. The documentation is also written in prose around code blocks, so even a careful reviewer cannot tell whether the blocks still run. Screenshots and prose explanations make this worse, not better: they are the parts of documentation most likely to go stale and least likely to be noticed, because nothing executes them. The fewer setup steps exist only as prose, the less there is to rot. The only reliable test is to execute the setup path from zero — a clean machine, a fresh clone, the literal commands from the README — and to do it on every change that could affect it and on a schedule to catch external drift, such as a tool's installer URL changing.

Resolution

1. Mark README setup blocks as executable with a consistent fence and keep them free of placeholders:

## Setup

Run these from a fresh clone:

```bash
make bootstrap
make db:seed
make doctor
```

Only fenced bash blocks inside the Setup section are extracted; explanatory blocks elsewhere use text or console fences so they are not executed.

2. Add a script that extracts and runs them in a clean environment:

#!/usr/bin/env bash
set -euo pipefail
readme="${1:-README.md}"
awk '/^## Setup/{f=1;next} /^## /{f=0} f' "$readme" \
  | awk '/^```bash$/{b=1;next} /^```$/{b=0} b' > /tmp/readme-setup.sh
test -s /tmp/readme-setup.sh || { echo "no bash blocks found in Setup section"; exit 1; }
echo "running $(wc -l < /tmp/readme-setup.sh) setup lines from $readme"
bash -euo pipefail -x /tmp/readme-setup.sh

Save as scripts/test-readme.sh. -x echoes each command, so a failure shows exactly which documented line broke.

3. Run it in CI in a fresh container whenever setup-related files change, and nightly:

name: onboarding
on:
  pull_request:
    paths: [README.md, Makefile, 'scripts/**', compose.yaml, '.tool-versions', mise.toml]
  schedule: [{ cron: '0 4 * * *' }]
jobs:
  fresh-setup:
    runs-on: ubuntu-24.04
    container: ubuntu:24.04
    steps:
      - run: apt-get update -qq && apt-get install -y -qq git make curl ca-certificates docker.io
      - uses: actions/checkout@v4
      - run: ./scripts/test-readme.sh README.md

4. Also test the documented platforms that matter. A macOS runner job running the same script catches Homebrew and Apple Silicon differences; a Windows job belongs only if Windows is documented as supported natively.

Expected output

running 3 setup lines from README.md
+ make bootstrap
==> tools ok, .env written, services healthy
+ make db:seed
seeded 4 tenants, 120 products
+ make doctor
docker ............. ok
db reachable ....... ok
all checks passed

Every documented line runs in order on a clean machine and the doctor target confirms the result. A pull request that renames a target without updating the README now fails this job with the exact line that broke.

The nightly run catches a different class of failure: nothing in the repository changed, but something the setup depends on did — an installer script moved, a base image tag was retagged, a package registry dropped a version. Those failures used to be discovered by whoever onboarded next; now they arrive as a red nightly build with a precise log, usually the same morning.

Prevention

1. Treat a red onboarding job as a release blocker for the repository's main branch; otherwise it becomes background noise.

2. Keep the Setup section short — ideally one or two commands that call the bootstrap target — so the test exercises tooling, and prose stays small enough to keep accurate. The make bootstrap guide covers the target itself.

3. Record the duration of the clean setup as an onboarding metric; a sudden increase is as important as a failure, as discussed in instrumenting bootstrap scripts with timing telemetry.

Platform caveats

Docker in CI containers: the setup usually starts Compose services, so the job needs Docker access — mount the runner's socket or use a service container. GitHub-hosted Linux runners have Docker available when the job runs directly on the runner rather than in a container.

macOS runners: they cannot run Linux containers without a VM; test macOS setup steps that do not need Docker there, and the container-based steps on Linux.

Apple Silicon (ARM64): use an arm64 runner (macos-14 or ubuntu-24.04-arm) for at least one job if most developers are on M-series Macs, since architecture-specific failures only appear there.

Rollback

The job and script are additive; disable the workflow if it blocks urgent work, then fix the README:

#!/usr/bin/env bash
set -euo pipefail
gh workflow disable onboarding.yml

Frequently Asked Questions

Related

• Keep onboarding docs in sync with CI checks
• Build the health check the test finishes with
• Audit transitive dependencies that break fresh installs