Detecting OpenSSL and libc Version Drift

A service connects to a partner API from production but fails locally with SSL routines::unsafe legacy renegotiation disabled; a Node app that worked on Node 16 now throws error:0308010C:digital envelope routines::unsupported in a teammate's container; and a Go binary built in CI crashes on an Alpine runtime image with /lib64/ld-linux-x86-64.so.2: not found. None of these are application bugs. They come from differences in the system libraries under the application — OpenSSL, the C library (glibc or musl) and the CA certificate bundle — which vary between base images, distribution versions and host installs. This page detects that drift across environments and removes it, as part of runtime parity frameworks.

These libraries sit below the level most version pinning covers: a pinned Node version can link against OpenSSL 1.1 in one image and OpenSSL 3 in another.

Diagnostic

Print the library versions every runtime actually uses, in each environment:

#!/usr/bin/env bash
set -euo pipefail
probe='
  echo "os:       $(. /etc/os-release && echo "$ID $VERSION_ID")"
  echo "libc:     $( (ldd --version 2>&1 | head -1) || echo musl)"
  echo "openssl:  $(openssl version 2>/dev/null || echo none)"
  command -v node >/dev/null && echo "node ssl: $(node -p process.versions.openssl)"
  command -v python3 >/dev/null && echo "py ssl:   $(python3 -c "import ssl; print(ssl.OPENSSL_VERSION)")"
  echo "ca certs: $(grep -c "BEGIN CERT" /etc/ssl/certs/ca-certificates.crt 2>/dev/null || echo 0)"
'
for img in shop/api:dev ghcr.io/acme/shop-api:prod; do
  echo "== $img"; docker run --rm --entrypoint sh "$img" -c "$probe"
done

Expected bad output:

== shop/api:dev
os:       debian 12
libc:     ldd (Debian GLIBC 2.36-9+deb12u7) 2.36
openssl:  OpenSSL 3.0.14 4 Jun 2024
node ssl: 3.0.13+quic
ca certs: 146
== ghcr.io/acme/shop-api:prod
os:       debian 11
libc:     ldd (Debian GLIBC 2.31-13+deb11u10) 2.31
openssl:  OpenSSL 1.1.1w  11 Sep 2023
node ssl: 3.0.13+quic
ca certs: 139

The development image is Debian 12 with OpenSSL 3; production is Debian 11 with OpenSSL 1.1 and an older CA bundle. Node's bundled OpenSSL matches, but anything using the system library — curl, Python, native modules — behaves differently.

Root cause

Language runtimes are pinned; the operating system underneath usually is not. Base images such as node:20 or python:3.12 track a Debian release that changes when the tag is rebuilt — node:20 moved from Debian 11 to 12 in 2023 — and different teams pin different variants (-bullseye, -bookworm, -slim, -alpine). OpenSSL 3 disabled legacy algorithms and unsafe renegotiation that OpenSSL 1.1 allowed, so the same TLS handshake or hash call succeeds in one image and fails in another. glibc versions are backwards-compatible but not forwards: a binary built against glibc 2.36 fails on 2.31 with version 'GLIBC_2.34' not found, and anything built for glibc fails on musl-based Alpine entirely. CA bundles age with the image, so a newer root CA may be trusted in one environment and not another. Because nothing prints these versions by default, the drift is invisible until one of these errors appears.

Resolution

1. Standardise one base image family and release for development, CI and production, and pin it by digest:

# syntax=docker/dockerfile:1.7
FROM node:20.17.0-bookworm-slim@sha256:5f0e8a0c7d3b2e6f41a9c8d7e5b4a3f2c1d0e9f8a7b6c5d4e3f2a1b0c9d841b9 AS base
RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates openssl && rm -rf /var/lib/apt/lists/*

Name the Debian release in the tag (-bookworm) so a future default change cannot move it silently; the digest pins it exactly, as in pinning base image digests for multi-arch builds.

2. Build the production image and the dev image from the same base stage using multi-stage targets, so they cannot drift:

FROM base AS dev
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci
CMD ["npm", "run", "dev"]

FROM base AS runtime
WORKDIR /app
COPY --from=dev /app/node_modules ./node_modules
COPY . .
CMD ["node", "server.js"]

3. Build Go binaries statically or on the runtime's libc. For Alpine or distroless runtimes, disable cgo:

#!/usr/bin/env bash
set -euo pipefail
CGO_ENABLED=0 GOOS=linux go build -trimpath -o out/server ./cmd/server
file out/server | grep -q 'statically linked' && echo "static binary: runs on any libc"

4. Fix legacy-crypto errors at the source rather than re-enabling legacy algorithms: upgrade the tool that uses MD4 (old webpack versions caused the digital envelope routines::unsupported error), and ask partners to support secure renegotiation. Use --openssl-legacy-provider only as a documented, temporary exception.

5. Record the library versions as a parity artifact in CI for every image, so drift is visible in review:

#!/usr/bin/env bash
set -euo pipefail
docker run --rm --entrypoint sh "$1" -c '. /etc/os-release; echo "$ID-$VERSION_ID glibc=$(ldd --version | head -1 | awk "{print \$NF}") openssl=$(openssl version | awk "{print \$2}")"' | tee "parity-$(echo "$1" | tr '/:' '__').txt"

Expected output

== shop/api:dev
os:       debian 12
libc:     ldd (Debian GLIBC 2.36-9+deb12u7) 2.36
openssl:  OpenSSL 3.0.14 4 Jun 2024
ca certs: 146
== ghcr.io/acme/shop-api:prod
os:       debian 12
libc:     ldd (Debian GLIBC 2.36-9+deb12u7) 2.36
openssl:  OpenSSL 3.0.14 4 Jun 2024
ca certs: 146

Development and production now share the same OS release, C library, OpenSSL and CA bundle, so TLS and crypto behaviour is identical and binaries built in one run in the other.

With the libraries aligned, the original errors become ordinary, reproducible bugs. The partner's legacy renegotiation now fails in development too, where it can be investigated and raised with the partner before release rather than discovered in an incident. That is the real benefit of parity: failures move earlier, not away.

Prevention

1. Diff library versions in CI between the dev and production targets, and fail when they differ.

2. Update the base image deliberately with Renovate digest updates, reviewing the OS release and OpenSSL major version in the diff.

3. Include library versions in the runtime parity report, alongside runtime and tool versions, as described in automating runtime parity checks between local and staging.

Platform caveats

macOS hosts: macOS ships LibreSSL as /usr/bin/openssl, so host-side scripts and containers disagree even when images match. Run parity-sensitive checks inside containers, or use the Homebrew openssl@3 explicitly.

Apple Silicon (ARM64): base image variants are multi-arch, and library versions match across architectures for the same digest; verify with the probe on both.

Alpine: uses musl and a different OpenSSL packaging; mixing Alpine and Debian images in one system is the most common source of libc drift. Prefer one family.

Rollback

Restore the previous Dockerfile and base image; behaviour returns to the old library mix:

#!/usr/bin/env bash
set -euo pipefail
git checkout HEAD~1 -- Dockerfile
docker compose build --no-cache api

Frequently Asked Questions

Related

• Pin a local runtime to match the production image
• Diagnose native module build failures on ARM64
• Match CI runner images to local dev containers