Instrumenting Bootstrap Scripts With Timing Telemetry

Everyone agrees onboarding is slow, but nobody can say which step is slow for whom: the new hire says "setup took most of the day", the platform team's own laptop bootstraps in eight minutes, and CI reports nothing because CI does not run the parts that hurt — first image pulls over hotel Wi-Fi, Homebrew on a fresh Mac, the Nix store download. Without per-step data from real machines, improvement work targets guesses. This page adds lightweight timing telemetry to the bootstrap script so the team sees where setup time actually goes, as part of time-to-first-PR metrics.

The design goal is useful data with minimal intrusion: step names, durations, exit codes and a coarse platform label — never paths, usernames, hostnames or environment values.

Diagnostic

Check whether the bootstrap produces any timing information today:

#!/usr/bin/env bash
set -euo pipefail
grep -nE '^bootstrap:' -A12 Makefile
/usr/bin/time -p make bootstrap 2>&1 | tail -3
ls ~/.cache/acme-bootstrap 2>/dev/null || echo "no local timing records"

Expected bad output:

bootstrap: check-tools env certs deps images seed
	@echo "done"
real 1893.41
user 212.60
sys 88.14
no local timing records

The only number is the total — 31 minutes on this machine — with no breakdown by step, no record of failures, and nothing collected across machines.

Root cause

Bootstrap scripts are usually written as a sequence of targets with no instrumentation, and they run on the machines where problems happen — new laptops, unusual networks, first runs — exactly once per person. Anecdotes survive; numbers do not. CI measures its own clean setup, which is useful but different: CI runners have fast networks, warm registries and no OS-level setup. The result is that onboarding improvement work optimises what the platform team can see on its own machines, while the steps that dominate for new hires (large first pulls, cold dependency caches, VM provisioning) go unmeasured. Timing each step locally, and optionally reporting it, turns the anecdote into a distribution the team can act on.

Failures are the other half of the picture. A step that fails and is retried by hand — an image pull that times out, a seed that hits a port conflict — adds minutes that a total-time measurement attributes to "slow setup" in general. Recording the exit code of every attempt shows which steps fail, how often, and on which platforms, which is often more actionable than raw duration: a step that fails one run in five on Apple Silicon is a bug to fix, not a performance problem to optimise.

Local logs matter even without reporting. When a new hire asks for help, cat ~/.cache/acme-bootstrap/steps.tsv shows exactly which step took forty minutes or failed three times, which turns a vague support conversation into a specific one within seconds.

Resolution

1. Wrap each step with a timer that records name, duration and exit code locally:

#!/usr/bin/env bash
set -euo pipefail
# scripts/timed.sh <step-name> <command...>
step="$1"; shift
log_dir="${XDG_CACHE_HOME:-$HOME/.cache}/acme-bootstrap"; mkdir -p "$log_dir"
run_id="${BOOTSTRAP_RUN_ID:-$(date +%s)}"
start=$(date +%s)
set +e; "$@"; code=$?; set -e
dur=$(( $(date +%s) - start ))
printf '%s\t%s\t%s\t%s\n' "$run_id" "$step" "$dur" "$code" >> "$log_dir/steps.tsv"
printf '  [%4ss] %-12s %s\n' "$dur" "$step" "$([ $code -eq 0 ] && echo ok || echo "FAILED ($code)")"
exit "$code"

2. Call it from the Makefile so every step is timed with no change to what the steps do:

export BOOTSTRAP_RUN_ID := $(shell date +%s)
T := ./scripts/timed.sh

bootstrap:
	@$(T) check-tools ./scripts/check-tools.sh
	@$(T) env ./scripts/write-env.sh
	@$(T) certs $(MAKE) --no-print-directory certs
	@$(T) deps npm ci --no-audit --no-fund
	@$(T) images docker compose pull --quiet
	@$(T) seed $(MAKE) --no-print-directory db-seed
	@./scripts/report-timing.sh

3. Report anonymously, with an opt-out, so the team sees a distribution across machines:

#!/usr/bin/env bash
set -euo pipefail
[ "${ACME_TELEMETRY:-on}" = off ] && exit 0
log="${XDG_CACHE_HOME:-$HOME/.cache}/acme-bootstrap/steps.tsv"
run_id="${BOOTSTRAP_RUN_ID:?}"
platform="$(uname -s)-$(uname -m)"
steps=$(awk -v r="$run_id" -F'\t' '$1 == r {printf "%s{\"step\":\"%s\",\"s\":%s,\"exit\":%s}", (n++ ? "," : ""), $2, $3, $4}' "$log")
curl -fsS -m 3 -X POST https://telemetry.acme.dev/bootstrap -H 'content-type: application/json' \
  -d "{\"platform\":\"$platform\",\"steps\":[$steps]}" >/dev/null || true
echo "timing reported (set ACME_TELEMETRY=off to disable)"

Only step names, durations, exit codes and an OS-architecture label leave the machine. A three-second timeout and || true ensure telemetry never blocks or fails setup.

4. Summarise the slowest steps — per platform, at the median and the 90th percentile — in a dashboard or a weekly query:

SELECT platform, step,
       percentile_cont(0.5) WITHIN GROUP (ORDER BY s) AS p50_s,
       percentile_cont(0.9) WITHIN GROUP (ORDER BY s) AS p90_s,
       count(*) FILTER (WHERE exit <> 0) AS failures
FROM bootstrap_steps
WHERE received_at > now() - interval '30 days'
GROUP BY platform, step
ORDER BY p90_s DESC
LIMIT 10;

Expected output

$ make bootstrap
  [  12s] check-tools  ok
  [   4s] env          ok
  [   5s] certs        ok
  [ 486s] deps         ok
  [1104s] images       ok
  [ 282s] seed         ok
timing reported (set ACME_TELEMETRY=off to disable)

And after a month, the summary query answers the question that started this:

platform         step    p50_s  p90_s  failures
Darwin-arm64     images  610    1480   2
Darwin-arm64     deps    240    520    0
Linux-x86_64     images  190    410    0
Darwin-arm64     seed    150    300    5

Image pulls on Apple Silicon dominate, and their slowest runs are very slow; seeding fails occasionally on the same platform. Those two rows are the improvement backlog — for example a smaller seed dataset and prebuilt images from a nearby registry mirror — and re-running the query after each change shows whether it helped.

Prevention

1. Keep telemetry minimal and documented. List exactly what is sent in the README, keep the opt-out, and review the payload whenever the script changes.

2. Alert on regressions. A weekly job that compares p90 per step with the previous month catches a new slow step — a heavier image, an extra dependency — soon after it lands.

3. Time the doctor and the first test run too, since time-to-first-PR includes getting a green local test, not just a finished bootstrap; see how to measure developer onboarding time in distributed teams.

Platform caveats

macOS: date +%s is available by default; avoid date +%s%N, which GNU supports but macOS date does not. Second-level resolution is enough for bootstrap steps.

Windows (native): the scripts are Bash; run bootstrap in WSL2, where they behave as on Linux, or add a PowerShell equivalent using Measure-Command.

Corporate networks: outbound telemetry may be blocked by proxies. The short timeout keeps setup unaffected; local timings are still recorded.

Rollback

Remove the wrapper from the Makefile and delete local logs; nothing else depends on them:

#!/usr/bin/env bash
set -euo pipefail
git checkout HEAD~1 -- Makefile
rm -rf "${XDG_CACHE_HOME:-$HOME/.cache}/acme-bootstrap"

Frequently Asked Questions

Related

• Write the make bootstrap target being timed
• Cut first-build time for a fresh clone
• Run an onboarding friction survey alongside the data