Matching CI Runner Images to Local Dev Containers

A test passes in the dev container and fails on the GitHub runner with sh: 1: protoc: not found, or passes on both but with different output because the runner's ubuntu-24.04 image has jq 1.7.1 and git 2.46 while the dev container has jq 1.6 and git 2.39. The dev container defines the development environment precisely; CI runs on a general-purpose runner image with hundreds of preinstalled tools at versions nobody on the team chose. This page runs CI inside the same image developers use, so "passes locally" and "passes in CI" mean the same environment, as part of CI/CD pipeline parity checks.

The approach reuses the dev container definition rather than maintaining a separate CI image, so there is exactly one environment to keep correct.

Diagnostic

Compare tool versions in the dev container with those on the CI runner:

#!/usr/bin/env bash
set -euo pipefail
probe='for t in node npm python3 go protoc jq git make; do printf "%s\t%s\n" "$t" "$($t --version 2>&1 | head -1 || echo missing)"; done'
devcontainer up --workspace-folder . >/dev/null
devcontainer exec --workspace-folder . bash -c "$probe" | sort > /tmp/dev.txt
gh run download "$(gh run list --workflow tool-probe.yml --limit 1 --json databaseId -q '.[0].databaseId')" -n probe -D /tmp/ci 2>/dev/null
sort /tmp/ci/probe.txt > /tmp/ci.txt 2>/dev/null || true
diff -u /tmp/dev.txt /tmp/ci.txt || true

Expected bad output:

--- /tmp/dev.txt
+++ /tmp/ci.txt
-git	git version 2.39.5
+git	git version 2.46.0
-jq	jq-1.6
+jq	jq-1.7.1
-node	v20.17.0
+node	v20.16.0
-protoc	libprotoc 25.1
+protoc	sh: 1: protoc: not found

Four tools differ, and one is missing entirely from the runner.

Root cause

Hosted CI runners are general-purpose machines. Their images are maintained by the CI provider, include hundreds of tools, and are updated weekly — the versions on the runner are whatever the provider shipped that week. Workflows then install a few pinned tools on top with setup actions and rely on the preinstalled ones for everything else. The dev container, by contrast, is a precise definition the team controls. The two drift in both directions: the runner has tools the dev container lacks (so CI passes for reasons local runs cannot reproduce), lacks tools the dev container has (so CI fails), and has different versions of shared tools (so behaviour differs). Running CI jobs inside the dev container image removes the runner image from the equation; the runner becomes a host for the container, nothing more.

The "extra tools" direction is the sneaky one. A test that shells out to jq or zip passes in CI because the runner happens to have them, and fails for a new hire whose dev container does not — or the reverse, where the dev container has a tool the production image lacks and CI never notices. Both directions come from the same cause: two environments defined by different people for different purposes. Collapsing them into one definition does not only fix version mismatches; it makes missing tools fail in the same place for everyone, which is where they can be fixed once.

Runner image updates also stop being surprises. Providers announce changes, but few teams read those announcements, and a default version bump — Python, Git, a compiler — shows up as an unexplained CI failure on an unrelated pull request. Inside the team's own image, those versions change only when the team changes them.

Resolution

1. Prebuild and publish the dev container image from CI, tagged by commit and pinned by digest, so jobs pull rather than rebuild:

name: devcontainer-image
on:
  push:
    branches: [main]
    paths: ['.devcontainer/**']
jobs:
  build:
    runs-on: ubuntu-24.04
    permissions: { contents: read, packages: write }
    steps:
      - uses: actions/checkout@v4
      - uses: docker/login-action@v3
        with: { registry: ghcr.io, username: '${{ github.actor }}', password: '${{ secrets.GITHUB_TOKEN }}' }
      - uses: devcontainers/[email protected]
        with:
          imageName: ghcr.io/acme/shop-devcontainer
          imageTag: ${{ github.sha }},latest
          cacheFrom: ghcr.io/acme/shop-devcontainer
          push: always

2. Run test jobs inside that image. devcontainers/ci runs commands in the container built from the same devcontainer.json, including Features and postCreateCommand:

jobs:
  test:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@v4
      - uses: devcontainers/[email protected]
        with:
          cacheFrom: ghcr.io/acme/shop-devcontainer
          push: never
          runCmd: make lint test

Alternatively, for jobs that do not need the full dev container lifecycle, use a container job with the published image pinned by digest:

jobs:
  unit:
    runs-on: ubuntu-24.04
    container:
      image: ghcr.io/acme/shop-devcontainer@sha256:9b1c7e2a4f0d3c6b8e5a7f1d2c4b6a8e0f3d5c7b9a1e3f5d7c9b2a4e6f8d0c1b
    steps:
      - uses: actions/checkout@v4
      - run: make test

3. Point the dev container at the same prebuilt image so developers pull it too, keeping laptops and CI on identical layers:

{
  "image": "ghcr.io/acme/shop-devcontainer:latest",
  "remoteUser": "vscode"
}

4. Keep the tool probe as a CI job that uploads its output, so the diff from the diagnostic can be rerun any time.

Expected output

$ diff -u /tmp/dev.txt /tmp/ci.txt && echo "dev container and CI identical"
dev container and CI identical
$ gh run view --log | grep -m1 'protoc'
libprotoc 25.1

CI runs make lint test inside the same image as the dev container, the probe shows no differences, and failures reproduce locally by opening the dev container.

Debugging a CI failure now starts with devcontainer up on the commit in question, which gives the exact environment the job used. The previous loop — push a debug commit, wait for the runner, read logs, repeat — is only needed for problems caused by the runner host itself, such as resource limits, which the CI-only timeout guide covers.

Prevention

1. Rebuild the image when its definition changes and weekly for base image security updates, with the digest recorded in the job output.

2. Fail CI if a job runs on the bare runner for steps that should use the image — a quick check that /.dockerenv or the image label is present at the start of the job.

3. Keep the probe job and alert when its output differs from the committed expected list, which catches changes to the definition that were not intended.

Platform caveats

Apple Silicon (ARM64): developers pull the arm64 variant and CI the amd64 variant of the same tag; build the image for both platforms so the tool versions match, as in building multi-arch images locally with buildx.

Docker inside the job: tests that start containers need Docker access from within the dev container job; use the docker-outside-of-docker Feature and mount the runner's socket, as covered in running Docker inside a dev container.

Job startup time: pulling a multi-gigabyte image adds time per job; keep the image lean and rely on layer caching in the registry.

Rollback

Switch jobs back to the bare runner with setup actions; the prebuilt image can keep serving developers:

#!/usr/bin/env bash
set -euo pipefail
git checkout HEAD~1 -- .github/workflows/test.yml

Frequently Asked Questions

Related

• Reproduce CI-only test failures locally with act
• Prebuild dev container images to cut startup time
• Enforce toolchain versions in CI