Debugging Works-on-My-Machine Runtime Drift

A test or build passes on your laptop and fails in CI with an identical commit hash — the classic "works on my machine" symptom that the runtime parity frameworks parent topic exists to eliminate. This guide gives you a repeatable procedure: fingerprint both environments, diff the fingerprints, and pin every axis that diverges until the failure reproduces on demand. The goal is not to make the red job green by luck but to make local and CI byte-for-byte comparable, so the same test yields the same result wherever it runs.

The trap with these failures is that the source tree is provably identical. git rev-parse HEAD matches on both sides, git status is clean, and yet the outcome differs. That rules out the code and points at the runtime: the interpreter version, the shell environment, the C library locale, the wall-clock timezone, and the CPU that executes native instructions. None of those are stored in the repository, so a plain code review can never surface them. You have to measure them directly.

Diagnostic

Capture a normalized runtime fingerprint in both environments and diff them. Run this script locally and as a CI step, then compare the two outputs. Sorting the keys makes the diff stable, so only genuinely divergent rows appear.

#!/usr/bin/env bash
set -euo pipefail
# runtime-fingerprint.sh — emit a deterministic snapshot of the runtime.
{
  echo "arch=$(uname -m)"
  echo "kernel=$(uname -s)"
  echo "node=$(node -v 2>/dev/null || echo none)"
  echo "python=$(python3 -V 2>&1 || echo none)"
  echo "libc=$(ldd --version 2>/dev/null | head -n1 || echo none)"
  echo "tz=$(date +%Z)"
  echo "lc_all=${LC_ALL:-unset}"
  echo "lang=${LANG:-unset}"
  echo "node_env=${NODE_ENV:-unset}"
  echo "umask=$(umask)"
} | sort

Expected BAD output — diffing local (left) against CI (right) reveals the divergent rows in one glance:

$ diff <(./runtime-fingerprint.sh) ci-fingerprint.txt
< arch=arm64
> arch=x86_64
< node=v20.18.0
> node=v18.20.4
< tz=CEST
> tz=UTC
< lang=en_US.UTF-8
> lang=C

Four rows differ, and each one is a candidate root cause. Emit the CI fingerprint as a build artifact (or simply cat it in the job log) so you can pull it down and diff it against your workstation without SSH access to the runner. Keep the script dependency-free — only coreutils and the language binaries you already require — so it runs on a bare Alpine runner as readily as on macOS. When the diff is non-empty, the exit code is non-zero, which lets you wire the same script into a gate later without rewriting it.

Root cause

"Works on my machine" is almost never the code; it is an undeclared dependency on the runtime. Five axes account for the overwhelming majority of these failures, and the fingerprint above exposes each one.

Language and runtime version. A method, syntax feature, or standard-library default that exists locally may be missing or behave differently in CI. Array.prototype.at() landed in Node 16.6, structuredClone in 17, and Python f-string debugging (f"{x=}") in 3.8. A test written against Node 20 that fails on the runner's Node 18 is not flaky — it is running on a different interpreter. Even patch versions matter when a bug fix changes rounding or regex behaviour.

Environment variables. A NODE_ENV, TZ, or feature-flag variable set in your shell profile but absent on the runner silently changes control flow. The most dangerous variety is the one exported in your ~/.zshrc months ago and forgotten — it is part of your machine's identity, not the project's, so it never made it into version control.

Locale. String sorting, uppercasing, number parsing, and collation all depend on LANG/LC_ALL. A test that sorts ["Åke", "Bob", "ake"] produces one order under en_US.UTF-8 and a different one under C, where bytes sort ASCII-first. CSV parsing that relies on a decimal comma versus point breaks the same way.

Timezone. TZ shifts every naive datetime. A test asserting that an event falls "today" passes at 14:00 CEST and fails when the runner in UTC has already rolled past midnight, or a formatted timestamp gains an hour. Any date arithmetic without an explicit zone is a latent CI failure.

CPU architecture. arm64 locally versus x86_64 in CI exposes native-module ABI mismatches, differing floating-point intermediate precision, and byte-order-sensitive code. A dependency that ships a prebuilt binary for one arch and compiles from source on the other can behave subtly differently.

None of these live in the repo, so the diff is invisible until you fingerprint both sides. Pinning each axis turns the runtime into a declared, version-controlled dependency — the same discipline the parent topic applies to whole-environment parity.

Resolution

Work the divergent rows in the order the fingerprint reports them. Each step pins one axis and is independently verifiable by re-running runtime-fingerprint.sh.

1. Pin the language and runtime version so both sides resolve the identical interpreter.
2. Declare every required env var in .env.example and inject the same set into CI.
3. Force locale and timezone explicitly in both environments.
4. Pin the container platform so architecture matches the CI runner.

Pin runtime versions in a manifest both the workstation and CI consume. A version manager such as asdf or mise reads this file, so a developer running mise install and a CI job running the same command land on byte-identical interpreters:

# .tool-versions (asdf/mise) — single source for local and CI
nodejs 20.18.0
python 3.12.4

Normalize locale, timezone, and architecture in Docker Compose so local execution mirrors the runner. Setting these in the service definition means every contributor inherits them without editing a shell profile — the same reasoning behind multi-service orchestration with Compose:

# docker-compose.yml
services:
  app:
    build: .
    platform: linux/amd64   # match CI's x86_64 runners
    environment:
      - LANG=C.UTF-8
      - LC_ALL=C.UTF-8
      - TZ=UTC
      - NODE_ENV=test

Propagate the non-secret half of the environment from a committed template with dotenv configuration management, then validate the contract so a missing variable fails loudly rather than drifting silently — see catching missing env vars before container startup:

#!/usr/bin/env bash
set -euo pipefail
# Assert every key declared in .env.example is exported before running tests.
while IFS= read -r key; do
  [ -z "$key" ] && continue
  [ -n "${!key:-}" ] || { echo "missing env var: $key"; exit 1; }
done < <(grep -vE '^\s*#|^\s*$' .env.example | cut -d= -f1)
echo "env contract satisfied"

The four axes are not equally common. Across a representative sample of "works on my machine" incidents, runtime-version and environment-variable drift dominate, with locale and timezone close behind and architecture the rarest but hardest to spot. The chart below is a rough distribution to help you decide where to look first when the diff shows several suspect rows.

Expected output

Once each divergent axis is pinned, the fingerprints match and the diff is empty:

$ diff <(./runtime-fingerprint.sh) ci-fingerprint.txt
$ echo $?
0

The previously CI-only failure now reproduces (and, after the fix, passes) identically in both places. An empty diff is the proof that you are testing parity rather than relying on coincidence: the same interpreter, the same environment, the same locale and clock, and the same instruction set. If the test still fails after the diff is empty, you have found a genuine code or data-fixture bug — which is exactly the state you want, because it is now reproducible on your laptop.

Prevention

Pinning once is not enough; drift creeps back as runners upgrade and shells accumulate exports. Lock the gains in with automation.

1. Run runtime-fingerprint.sh in CI and fail the job if it diverges from a committed baseline — the lightweight companion to automating runtime parity checks between local and staging.
2. Keep the .tool-versions pin and the Compose platform/locale block under review in every pull request, so a version bump is a deliberate, reviewed change rather than an accident.
3. Add the env-contract assertion to your onboarding health-check script so drift surfaces before tests even run.

Commit the baseline fingerprint alongside the workflow so any silent runner upgrade — a new default Node on the hosted image, a locale change in a base container — trips the gate on the next push instead of surfacing as a mysterious red build weeks later.

Platform caveats

macOS (Docker Desktop): the host TZ and LANG come from system settings, not the container; never assume the host locale — always set TZ/LC_ALL inside Compose so the container is deterministic regardless of who runs it. WSL2: the WSL distro often defaults LANG to C.UTF-8 while Windows is en_US; pin locale explicitly so WSL and CI agree, and be aware that clock skew between the WSL VM and the Windows host can perturb timezone-sensitive tests. Apple Silicon (ARM64): without platform: linux/amd64, native modules compile for arm64 and behave differently than CI's x86_64 — pin the platform to reproduce CI faithfully, accepting the emulation overhead as the cost of fidelity.

Rollback

If the pins break a local-only workflow — for example a native tool that only ships arm64 binaries — revert them in one step and iterate on a narrower fix:

#!/usr/bin/env bash
set -euo pipefail
git checkout -- .tool-versions docker-compose.yml   # revert pins if they break local-only workflows

Frequently Asked Questions

Related

• Automate runtime parity checks between local and staging
• Catch missing env vars before container startup
• Reproduce CI-only test failures locally with act
• Fix boolean and number env coercion bugs