Rotating Secrets Without Restarting Containers

You rotated a database password, but every running container still holds the old one in process.env and only picks up the new value after a full restart — which drops in-flight connections. This guide is part of local secret vaults and rotation within the environment sync, secrets and CI parity baseline.

Diagnostic

Confirm that the running process is pinned to the old secret after rotation.

#!/usr/bin/env bash
set -euo pipefail
# Rotate the mounted secret file
echo "new-password-v2" > ./secrets/db_password.txt

# The container still serves the old value from its env
docker compose exec app printenv DB_PASSWORD

Expected BAD output:

old-password-v1

The file changed on disk, but the process loaded the secret once at boot and never re-read it.

Root cause

Environment variables are copied into the process at startup and are immutable for that process's lifetime. Mounting a secret as a file is the prerequisite for live rotation, but it is not sufficient — the application must be told to re-read the file. Without a reload trigger (a file watcher, a signal handler, or a sidecar that refreshes the value), the new secret only takes effect on the next restart, which is exactly the downtime you are trying to avoid.

Resolution

Mount the secret as a file rather than an env var, so the value can change under a running process.

# docker-compose.yml
services:
  app:
    image: app:local
    secrets:
      - db_password
    environment:
      DB_PASSWORD_FILE: /run/secrets/db_password
secrets:
  db_password:
    file: ./secrets/db_password.txt

Have the app re-read the file on SIGHUP, so an operator (or rotation hook) can signal a reload with zero downtime.

// secret-reload.ts
import { readFileSync } from 'node:fs';

let dbPassword = readFileSync(process.env.DB_PASSWORD_FILE!, 'utf8').trim();

process.on('SIGHUP', () => {
  dbPassword = readFileSync(process.env.DB_PASSWORD_FILE!, 'utf8').trim();
  console.log('secret reloaded on SIGHUP');
});

export const getDbPassword = () => dbPassword;

Send the signal after rotation — or use a file watcher / sidecar agent to do it automatically.

#!/usr/bin/env bash
# rotate-and-reload.sh
set -euo pipefail
echo "$1" > ./secrets/db_password.txt
docker compose kill -s SIGHUP app
echo "rotated and signalled reload"

#!/usr/bin/env bash
# sidecar: watch the file and SIGHUP the app on change (inotify)
set -euo pipefail
while inotifywait -e modify /run/secrets/db_password; do
  kill -HUP "$(pgrep -f 'node server.js')"
done

Expected output

$ ./rotate-and-reload.sh new-password-v2
rotated and signalled reload
$ docker compose logs app | tail -1
app-1  | secret reloaded on SIGHUP

The process now uses new-password-v2 without a restart and without dropping connections.

Prevention

Establish the pattern at design time: read every rotatable secret from a file with a reload path, never from a static env var.
Add a smoke test in CI that rotates a dummy secret and asserts the app reports a reload, so the mechanism cannot silently regress.
For Vault-managed secrets, pair this with lease renewal so rotation and reload are driven by the same lifecycle.

macOS (Docker Desktop): inotify events do not always propagate across the virtualized bind mount; prefer the explicit SIGHUP trigger or a short poll loop on the file's mtime. WSL2: file-watch reload only fires reliably when the secret file lives on the Linux filesystem (~/code), not under /mnt/c. Apple Silicon (ARM64): ensure inotify-tools is the arm64 build inside the sidecar image, or the watcher exits immediately with exec format error.

Rollback

If a rotated secret is bad, write the previous value back and signal another reload — no restart needed:

#!/usr/bin/env bash
set -euo pipefail
./rotate-and-reload.sh "old-password-v1"