Writing a Troubleshooting Runbook Linked to Diagnostics

The same five setup problems — port 5432 already in use, an expired local certificate, Docker out of memory, a stale node_modules volume, a missing .env key — are answered in the team chat every month, each time from memory and slightly differently. The existing troubleshooting page is a wall of prose that nobody searches, and make doctor prints FAIL: db not reachable without saying what to do next. This page builds a runbook where every entry has a stable ID, a symptom, a runnable check and a fix — and where the doctor script prints the ID and link for any failing check. It is part of README-driven automation.

The design principle: a developer who sees a failure should be one click from the exact fix, and the runbook entry should be tested by the same code that detects the problem.

Diagnostic

Measure how often the same issues recur and whether the doctor output points anywhere useful:

#!/usr/bin/env bash
set -euo pipefail
make doctor 2>&1 | grep -E 'FAIL|WARN' | head -5 || true
test -f docs/runbook.md && grep -c '^## ' docs/runbook.md || echo "no runbook"
grep -rhoE 'RB-[0-9]{3}' scripts/doctor.sh docs 2>/dev/null | sort -u | wc -l | xargs echo "runbook IDs referenced:"

Expected bad output:

FAIL: db not reachable
FAIL: certificate check
no runbook
runbook IDs referenced: 0

Failures are reported without a cause or a next step, there is no runbook to link to, and nothing connects checks to documentation.

Root cause

Troubleshooting knowledge accumulates in the place people ask: chat threads, which are hard to search and impossible to keep current. When a runbook exists, it is usually organised by component ("Database", "Docker") rather than by what the developer sees, written in prose that drifts from the actual scripts, and disconnected from the tool that detects problems. The doctor script knows exactly which check failed but has no way to hand over the fix; the runbook has the fix but no way to be found at the right moment. Giving each known problem a stable identifier and making the check print it closes that gap: the detection and the documentation share a key, and CI can verify that every key exists on both sides.

Resolution

1. Write runbook entries with a fixed structure — ID, symptom as the developer sees it, check command, fix, and why it happens:

## RB-003 — Port 5432 already in use

**Symptom:** `docker compose up` fails with `Bind for 0.0.0.0:5432 failed: port is already allocated`.

**Check:**
    lsof -nP -iTCP:5432 -sTCP:LISTEN

**Fix:** stop the other Postgres (`brew services stop postgresql@16`) or set `DB_PORT=5433` in `.env` and rerun `make up`.

**Why:** a Postgres installed with Homebrew or another project's stack already listens on 5432.

Keep entries short, symptom-first and ordered by ID so links stay stable.

2. Make each doctor check print its runbook link on failure:

#!/usr/bin/env bash
set -euo pipefail
RUNBOOK="https://github.com/acme/shop/blob/main/docs/runbook.md"
fails=0
check() {
  local id="$1" name="$2"; shift 2
  if "$@" >/dev/null 2>&1; then
    printf '  ok    %s\n' "$name"
  else
    printf '  FAIL  %s  -> %s#%s\n' "$name" "$RUNBOOK" "$(echo "$id" | tr 'A-Z' 'a-z')"
    fails=$((fails + 1))
  fi
}
check RB-001 "docker daemon reachable" docker info
check RB-002 "docker memory >= 6 GiB" sh -c '[ "$(docker info --format "{{.MemTotal}}")" -ge 6000000000 ]'
check RB-003 "port 5432 free or used by our db" sh -c '! lsof -nP -iTCP:5432 -sTCP:LISTEN | grep -qv com.docke'
check RB-004 "local certificate valid 30+ days" openssl x509 -checkend 2592000 -noout -in .certs/local.pem
check RB-005 ".env has every key in .env.example" sh -c 'diff <(cut -d= -f1 .env.example | sort) <(cut -d= -f1 .env | sort) >/dev/null'
exit "$fails"

The anchor is the lower-cased ID, which matches the heading anchor most Markdown renderers generate for ## RB-003 — ... headings when entries start with the ID.

3. Keep IDs and checks in sync with a CI test that fails when a check references a missing entry or an entry has no check:

#!/usr/bin/env bash
set -euo pipefail
in_checks=$(grep -oE 'RB-[0-9]{3}' scripts/doctor.sh | sort -u)
in_runbook=$(grep -oE '^## RB-[0-9]{3}' docs/runbook.md | cut -c4- | sort -u)
diff <(echo "$in_checks") <(echo "$in_runbook") && echo "runbook and doctor checks in sync"

4. Add an entry whenever a question repeats. The rule of thumb: the second time a setup question is answered in chat, it becomes a runbook entry and, where possible, a doctor check.

Expected output

$ make doctor
  ok    docker daemon reachable
  ok    docker memory >= 6 GiB
  FAIL  port 5432 free or used by our db  -> https://github.com/acme/shop/blob/main/docs/runbook.md#rb-003
  ok    local certificate valid 30+ days
  ok    .env has every key in .env.example
make: *** [doctor] Error 1
$ brew services stop postgresql@16 && make doctor | grep 5432
  ok    port 5432 free or used by our db

The failure names the problem and links to the entry; the documented fix resolves it; rerunning the doctor confirms.

Over a few months the runbook becomes a record of the environment's real failure modes, ordered by when they were first seen. Reviewing it every quarter is a useful input for platform work: entries that are hit often are candidates for elimination rather than documentation — a port that always collides can be moved, a certificate that keeps expiring can be regenerated automatically by bootstrap.

Prevention

1. Run the sync test in CI on every change to scripts/doctor.sh or docs/runbook.md.

2. Never renumber entries. Deprecated entries keep their ID with a note pointing to the replacement, so old links in chat history still work.

3. Track which checks fail most — a count of failing IDs from the doctor's optional telemetry — and prioritise removing the cause over improving the entry.

Platform caveats

macOS and Linux: lsof is available on both; on minimal Linux images use ss -ltnp in the check instead.

WSL2: port checks inside WSL do not see ports held by Windows processes; add a PowerShell-side check or note in the entry that netstat -ano on Windows may be needed.

Rendering differences: heading anchor generation differs between GitHub, GitLab and static site generators. Add explicit anchors (<a id="rb-003"></a>) above each heading if the runbook is rendered in more than one place.

Rollback

The runbook and links are additive; remove the link printing to return to the previous doctor output:

#!/usr/bin/env bash
set -euo pipefail
git checkout HEAD~1 -- scripts/doctor.sh

Frequently Asked Questions

Related

• Build the onboarding health-check script
• Fix port-already-allocated errors in Compose
• Reduce setup friction for junior engineers