Rotating Secrets Without Restarting Containers

You rotated a database password, but every running container still holds the old one in process.env and only picks up the new value after a full restart — which drops in-flight connections and cold-starts your pool. This guide is part of local secret vaults and rotation within the environment sync, secrets and CI parity baseline. It shows how to make a live process re-read a rotated secret and re-establish its connections without dropping traffic, whether the secret comes from a plain file, Vault, dotenv-vault, or SOPS.

Diagnostic

Confirm that the running process is pinned to the old secret after rotation. The point of this step is to prove that the file changed while the process value did not — the two facts together localize the fault to the application, not the mount.

#!/usr/bin/env bash
set -euo pipefail
# Rotate the mounted secret file
echo "new-password-v2" > ./secrets/db_password.txt

# The container still serves the old value from its env
docker compose exec app printenv DB_PASSWORD

Expected BAD output:

old-password-v1

The file changed on disk, but the process loaded the secret once at boot and never re-read it. You can confirm the process is holding a stale copy in memory by dumping its live environment block directly from the kernel — this reads the values the process actually has, not what the shell would inject:

#!/usr/bin/env bash
set -euo pipefail
pid=$(docker compose exec -T app pgrep -f 'node server.js')
docker compose exec -T app tr '\0' '\n' < /proc/"$pid"/environ | grep DB_PASSWORD

If that still prints old-password-v1 while ./secrets/db_password.txt on the host shows new-password-v2, the divergence is confirmed: the mount propagated the change and the process is simply not reacting to it. A second useful signal is the database's own view — query pg_stat_activity (or your engine's equivalent) for the app's connections and note that they are still authenticated and serving traffic under the revoked credential, which is exactly the window in which a naive restart would drop them.

Root cause

Environment variables are copied into a process's memory at execve() time and are immutable for that process's lifetime — nothing an external tool writes to disk, and no change to the parent shell, can mutate the environment of a process that is already running. Mounting a secret as a file is the prerequisite for live rotation, but it is not sufficient — the application must be told to re-read the file, and then to act on the new value. There are two distinct problems hiding here. The first is discovery: without a reload trigger (a file watcher, a signal handler, or a sidecar that refreshes the value), the new secret only takes effect on the next restart. The second, and the one most rotation guides omit, is propagation: even after your code re-reads the file, any open database connection was authenticated with the old credential and keeps working until it is closed. A password rotation that revokes the old credential server-side will start returning authentication errors on the next reconnect unless your pool is drained and rebuilt with the new secret. Solving rotation without downtime therefore means solving both — re-read the file and recycle the connections that depend on it.

Resolution

The steps below build the mechanism bottom-up: first make the value changeable under a running process, then make the process notice, then make it reconnect.

1. Mount the secret as a file rather than an env var, so the value can change under a running process. Point the application at the path, never at a baked-in variable.

# docker-compose.yml
services:
  app:
    image: app:local
    secrets:
      - db_password
    environment:
      DB_PASSWORD_FILE: /run/secrets/db_password
secrets:
  db_password:
    file: ./secrets/db_password.txt

2. Have the app re-read the file on SIGHUP, so an operator or a rotation hook can signal a reload with zero downtime. Keep the current value in a single accessor so every caller reads through the same reload-aware getter rather than caching its own copy.

// secret-reload.ts
import { readFileSync } from 'node:fs';

const secretPath = process.env.DB_PASSWORD_FILE!;
let dbPassword = readFileSync(secretPath, 'utf8').trim();

process.on('SIGHUP', () => {
  const next = readFileSync(secretPath, 'utf8').trim();
  if (next !== dbPassword) {
    dbPassword = next;
    console.log('secret reloaded on SIGHUP');
    reconnectPool(next).catch((err) => console.error('pool reconnect failed', err));
  }
});

export const getDbPassword = () => dbPassword;

3. Rebuild the connection pool with the new credential instead of merely storing the string. Create the fresh pool first, swap the reference, then drain the old pool so in-flight queries finish on the connection they started on. This is the step that turns "the variable updated" into "no request ever failed".

// reconnect-pool.ts
import { Pool } from 'pg';

let pool = new Pool({ password: process.env.DB_PASSWORD_INITIAL });

export async function reconnectPool(password: string): Promise<void> {
  const next = new Pool({ password });
  await next.query('SELECT 1');   // fail fast if the new credential is wrong
  const previous = pool;
  pool = next;                    // new checkouts use the new pool immediately
  await previous.end();           // drains: waits for in-flight queries, then closes
}

export const getPool = () => pool;

4. Send the signal after rotation — or use a file watcher or sidecar agent to do it automatically. The wrapper script writes the value and signals in one atomic operation so the two never drift.

#!/usr/bin/env bash
# rotate-and-reload.sh
set -euo pipefail
printf '%s' "$1" > ./secrets/db_password.txt
docker compose kill -s SIGHUP app
echo "rotated and signalled reload"

#!/usr/bin/env bash
# sidecar: watch the file and SIGHUP the app on change (inotify)
set -euo pipefail
while inotifywait -e close_write /run/secrets/db_password; do
  kill -HUP "$(pgrep -f 'node server.js')"
done

Choosing a reload trigger

Three mechanisms deliver the reload, and they are not interchangeable. A signal handler (SIGHUP) is the simplest and most portable — it works anywhere you can address the PID, and it is deterministic because you decide when it fires. A file watcher (inotify in a sidecar) removes the manual step but depends on filesystem events surviving the mount, which they do not always do across a virtualized bind mount. An application poll — the app stats the file every few seconds and reloads when the mtime changes — is the most robust across platforms because it depends on nothing but reading the file, at the cost of a small reaction delay. Pick by asking whether your rotation source can call a hook: if it can, signal directly; if it cannot, watch or poll the file it writes.

Expected output

With the file mount, the SIGHUP handler, and the pool rebuild in place, a rotation produces a reload log line and no dropped connections:

$ ./rotate-and-reload.sh new-password-v2
rotated and signalled reload
$ docker compose logs app | tail -2
app-1  | secret reloaded on SIGHUP
app-1  | pool reconnected: 8 idle connections rebuilt, 0 queries failed

Re-running the kernel-level check from the Diagnostic now shows the process value tracking the file:

$ docker compose exec -T app tr '\0' '\n' < /proc/1/environ | grep DB_PASSWORD_FILE
DB_PASSWORD_FILE=/run/secrets/db_password
$ docker compose exec app cat /run/secrets/db_password
new-password-v2

The process now uses new-password-v2 without a restart, and every in-flight request completed on the connection it started on.

Reload latency and dropped connections

The measurable payoff is in connections dropped per rotation. A full container restart tears down every connection; a rolling restart across replicas still drops whatever was in flight on the replica being cycled; the live reload drops none because the old pool drains instead of closing abruptly. The chart below shows a representative single-node run with an eight-connection pool under steady load. The reaction latency differs too: a SIGHUP reload completes in the low tens of milliseconds because it only rebuilds idle connections, while a full restart pays the container's entire boot and warm-up cost before it serves the first request.

Prevention

1. Establish the pattern at design time: read every rotatable secret from a file through a reload-aware accessor, never from a static env var, and never cache the raw string outside that accessor.
2. Add a smoke test in CI that rotates a dummy secret, sends the reload signal, and asserts the app reports both a reload and a successful pool rebuild — so the mechanism cannot silently regress. This pairs naturally with the checks in catching missing env vars before container startup.
3. For Vault-managed secrets, drive the reload from the same lease lifecycle that rotates the credential, so renewal and reconnection are one event rather than two independent timers that can drift.
4. Verify the new credential before swapping the pool reference, as the reconnectPool example does with SELECT 1; a bad rotation should surface as a logged reconnect failure that keeps the old pool alive, not as a wave of authentication errors.

Platform caveats

macOS (Docker Desktop): inotify events do not always propagate across the virtualized bind mount, so a file-watch sidecar can miss changes. Prefer the explicit SIGHUP trigger, or fall back to a short mtime poll loop inside the app. WSL2: file-watch reload only fires reliably when the secret file lives on the Linux filesystem (for example under ~/code), not under /mnt/c, where cross-boundary writes do not raise inotify events. Apple Silicon (ARM64): ensure inotify-tools is the arm64 build inside the sidecar image, or the watcher exits immediately with exec format error; pin the platform in the sidecar's FROM line if you build multi-arch.

Rollback

If a rotated secret is bad, write the previous value back and signal another reload — no restart needed, and the SELECT 1 guard means a still-bad value fails closed on the old pool rather than taking the service down:

#!/usr/bin/env bash
set -euo pipefail
./rotate-and-reload.sh "old-password-v1"

Frequently Asked Questions

Related

• Compare Vault, dotenv-vault, and SOPS for local secrets
• Keep local secrets out of version control
• Block startup on missing environment variables