Local Secret Vaults & Rotation

Eliminating environment drift requires secret management that mirrors production behavior without exposing sensitive material. This guide gives platform engineers a tactical path to provision, inject, rotate, and validate local secrets, working within the environment sync and CI parity baseline. If you are weighing tooling first, compare the options in Vault vs dotenv-vault vs SOPS for local secrets; if rotation is forcing container restarts, see rotating secrets without restarting containers.

The core problem this guide solves is a specific class of drift: the gap between how a developer's laptop obtains a database password and how the production cluster obtains one. When the laptop reads a plaintext DB_PASSWORD from a committed .env file while production pulls a short-lived, automatically rotated credential from a broker, the two environments diverge in ways no unit test catches. Code that never handles a mid-session credential rotation ships to production and fails the first time a lease expires under load. The remedy is to run the same secret broker locally that you run in CI and staging, wire it to the same lease lifecycle, and gate merges on a programmatic parity check. Everything below is built around HashiCorp Vault in dev mode because it exposes the full dynamic-secret and lease API on a single loopback port, but the rotation and parity patterns transfer to any broker that issues time-bound credentials.

Prerequisites

Before following the steps below, confirm your toolchain matches these versions. Mismatched daemon or CLI versions are the most common cause of silent injection failures, so pin them in your onboarding script rather than assuming a developer's global install is current.

• Docker Engine 24+ with the Compose v2 plugin (docker compose version must report v2.x, not the legacy docker-compose Python binary). The cap_add and healthcheck syntax below assume the v2 schema.
• The vault CLI on PATH, matching the server image minor version. A 1.15 CLI against a 1.13 server will accept most commands but silently ignore newer KV v2 metadata flags.
• jq 1.6+ for parsing JSON responses. The lease-lookup drift checks depend on jq numeric comparison behaving predictably.
• direnv 2.32+ hooked into your shell (direnv version and a working eval "$(direnv hook bash)" line in your rc file) for directory-scoped credential injection.
• yq 4.x (the Go implementation from mikefarah/yq, not the Python wrapper) for the YAML normalization used in the parity target.

A useful preflight is a single script that fails loudly if any tool is missing, so a new engineer sees one clear error instead of a cascade of command not found messages three steps in. Run it as the first line of your onboarding make bootstrap target so the failure surfaces before any container is pulled.

#!/usr/bin/env bash
set -euo pipefail

for tool in docker vault jq direnv yq; do
  command -v "${tool}" >/dev/null 2>&1 || {
    echo "MISSING: ${tool} is not on PATH — install it before continuing" >&2
    exit 1
  }
done
docker compose version >/dev/null 2>&1 || { echo "Compose v2 plugin required" >&2; exit 1; }
echo "Preflight OK: all secret-vault dependencies present"

Provision a Local Vault Instance

A lightweight Vault dev server gives you a single source of truth for development credentials, bound to the loopback interface so nothing on the local network can reach it. Dev mode auto-initializes and auto-unseals the server, keeps storage in memory by default, and mounts a KV v2 secrets engine at secret/ — which is exactly the ergonomics you want for a laptop, provided you understand the tradeoffs. The critical difference from a production deployment is that a dev server prints its unseal key and root token to stdout and trusts a single in-process token; you accept that because the blast radius is one developer's machine, and you deliberately mirror only the behaviors that matter (versioned secrets, leases, dynamic credentials), not the operational hardening.

1. Run Vault in dev mode via Compose, mapped to 127.0.0.1:8200 so it is never exposed on 0.0.0.0.
2. Bootstrap a fixed root token so scripts are deterministic, and enable the KV v2 engine for versioned audit trails and rollback.
3. Keep VAULT_ADDR identical in every place it appears — the Compose file, your shell, and CI — because a single mismatched scheme (http vs https) produces confusing TLS handshake errors rather than a clear connection refusal.

# docker-compose.yml
services:
  vault:
    image: hashicorp/vault:1.15
    ports:
      - "127.0.0.1:8200:8200"
    environment:
      VAULT_DEV_ROOT_TOKEN_ID: "local-dev-token"
      VAULT_ADDR: "http://127.0.0.1:8200"
    cap_add:
      - IPC_LOCK
    healthcheck:
      test: ["CMD", "vault", "status", "-address=http://127.0.0.1:8200"]
      interval: 5s
      timeout: 3s
      retries: 5
    volumes:
      - ./vault-data:/vault/file
    networks:
      - vault-net

networks:
  vault-net:
    driver: bridge

The IPC_LOCK capability lets Vault call mlock() so decrypted secrets never swap to disk — an important production behavior worth keeping locally, though you fall back gracefully when the host kernel rejects it. The healthcheck matters because postCreateCommand hooks and rotation daemons race the server's startup; without it, the first vault kv get fires before the KV engine is mounted and returns a spurious 404. Binding the published port to 127.0.0.1:8200:8200 rather than the bare 8200:8200 is deliberate: the short form binds to 0.0.0.0, which on a laptop joined to a coffee-shop network exposes your dev secrets to anyone on the same subnet. The explicit loopback address closes that hole at the Docker layer regardless of your host firewall. Once the container reports healthy, enable the engine and seed a secret so the rest of the guide has data to operate on.

Why KV v2 rather than the simpler v1 engine? Version 2 keeps a history of every write per path, so a bad seed can be rolled back without losing the prior value, and it stamps each version with metadata you can audit. That versioning is what the recovery section relies on. It also enables soft deletes: a vault kv delete marks a version deleted but recoverable, while vault kv destroy removes the underlying data permanently. For a dev workflow where mistakes are frequent and cheap to undo, that safety net is worth the marginally more verbose data path (.data.data.value instead of .data.value in jq).

#!/usr/bin/env bash
set -euo pipefail

export VAULT_ADDR="http://127.0.0.1:8200"
export VAULT_TOKEN="local-dev-token"

vault secrets enable -path=secret -version=2 kv 2>/dev/null || true
vault kv put secret/dev/db password="s3cr3t-dev-only"
vault kv put secret/dev/api-key value="ak_local_0000"
echo "KV v2 engine ready with seeded dev credentials"

The following figure shows the full local secret lifecycle this guide builds, from provisioning through the parity gate, so you can see how the four sections connect before diving into each.

Drift check — validate init state and VAULT_ADDR consistency:

#!/usr/bin/env bash
set -euo pipefail

vault status -format=json | jq '.initialized'   # expect: true
grep -rn "VAULT_ADDR" .env* docker-compose.yml || true
echo "Vault provisioning check complete"

WSL2: The NAT layer can intercept 127.0.0.1. If host resolution fails, bind to 0.0.0.0 in the container but restrict access via Docker network policies. macOS (Docker Desktop): Raise allocated memory to 4GB+ to avoid OOM kills during KV v2 init. Apple Silicon (ARM64): Use the hashicorp/vault:1.15 multi-arch manifest. If the host kernel rejects IPC_LOCK, remove cap_add and rely on Docker's default memory locking.

Inject Dynamic Secrets into Dev Containers

Static .env files cause drift and exposure: they persist decrypted on disk, get committed by accident, and never expire, so code that reads them never learns to handle a credential that changes underneath it. The alternative is to fetch short-lived credentials on demand and hold them only in the process environment. There are two mechanisms worth combining. Directory-scoped injection with direnv loads variables when you cd into the project and unloads them when you leave, so a stray printenv in an unrelated shell never surfaces the secret. Container-scoped injection via a devcontainer postCreateCommand fetches the credential once the toolchain is mounted, so the running service sees the same value your shell does. The static-fallback resolution rules — what happens when the broker is unreachable — are in dotenv and configuration management.

For true dynamic secrets, prefer Vault's database secrets engine over a static KV value: it generates a unique database user per lease and revokes it when the lease ends, so a leaked credential is useless within minutes. The KV path shown earlier is the simpler starting point; graduate to the database engine once your local Postgres or MySQL container is stable. Either way, the injection contract is identical from the container's perspective — an environment variable appears, backed by a value that will expire.

One subtlety catches teams the first time: remoteEnv and containerEnv differ in when they resolve. containerEnv is baked into the image environment at build time and is shared by every process the container starts, which makes it wrong for secrets because the value ends up in docker inspect output and image layers. remoteEnv is applied only to the tools the devcontainer CLI launches (your shell, the debugger), never persisted to the container definition, so a fetched credential stays out of inspectable metadata. Always use remoteEnv — or better, a live read at process start — for anything sensitive, and reserve containerEnv for non-secret configuration like NODE_ENV.

// .devcontainer/devcontainer.json
{
  "image": "mcr.microsoft.com/devcontainers/base:ubuntu",
  "remoteEnv": {
    "VAULT_ADDR": "http://host.docker.internal:8200",
    "DB_PASSWORD": "${localEnv:VAULT_DB_PASS}"
  },
  "postCreateCommand": "command -v vault && vault kv get -field=password secret/dev/db > /tmp/.db_pass && chmod 600 /tmp/.db_pass",
  "features": {
    "ghcr.io/devcontainers/features/vault:1": {}
  }
}

Note the chmod 600 — a credential written to a tmp file must never be group- or world-readable, and the devcontainer's default umask does not guarantee that. Note also that remoteEnv interpolates ${localEnv:...} from the host at container-create time, which means the value is captured once. If you need the injected value to track rotation, do not bake it into remoteEnv; instead read it live from Vault at process start, which the renewal section makes possible. The sequence below shows the exact order of operations direnv and the devcontainer follow, and where the two common race conditions appear.

Drift check — confirm injected variables resolve inside the container:

#!/usr/bin/env bash
set -euo pipefail

direnv export json | jq 'keys'
docker inspect "$(docker ps -q -f label=devcontainer.local_folder)" \
  --format='{{json .Config.Env}}' | jq .
echo "Injection check complete"

WSL2: Store .vault-token and .envrc inside native ext4 (/home/user/...); /mnt/c I/O latency causes direnv polling delays. macOS (Docker Desktop): Volume propagation can run postCreateCommand before the CLI is mounted — gate it with command -v vault as above.

Renew Leases on a Background Daemon

Local development needs predictable credential lifecycles, and the whole point of mirroring production is that credentials expire on a clock. Vault issues every dynamic credential with a lease that carries a TTL (time to live) and a max_ttl ceiling. A renewal extends the TTL up to that ceiling; past max_ttl the credential is revoked and a new one must be requested. Code that assumes a credential is permanent breaks at exactly the moment a lease crosses max_ttl — which locally you want to happen during a coffee break, not during a production incident. The renewal daemon below keeps the credential fresh during an active session and, crucially, refetches cleanly when renewal is no longer possible, so a developer never gets locked out of their own workspace.

Restrict the dev role to read and renew only, so a compromised token cannot escalate. Then run a renewal loop with exponential backoff: renew aggressively while things are healthy, and back off when the server is unreachable so you do not hammer a restarting container. Keep decrypted material off disk wherever possible, per managing local secrets without committing to git; the .env.local fallback below is a deliberate last resort for when the broker is down, and it should be gitignored.

#!/usr/bin/env bash
# scripts/rotate-local-secrets.sh
set -euo pipefail

LEASE_PATH="secret/dev/api-key"
BACKOFF=10
MAX_BACKOFF=300

while true; do
  if ! vault lease renew -increment=3600 "${LEASE_PATH}" 2>/dev/null; then
    echo "Lease expired or unreachable; refreshing via kv get..."
    vault kv get -format=json "${LEASE_PATH}" | jq -r '.data.data.value' > .env.local
    BACKOFF=10
  fi
  sleep "${BACKOFF}"
  BACKOFF=$(( BACKOFF * 2 > MAX_BACKOFF ? MAX_BACKOFF : BACKOFF * 2 ))
done

For a production-grade alternative that removes the hand-rolled loop entirely, run the Vault Agent as a sidecar: it handles auto-auth, caches tokens, and renders templated secrets to files or environment with vault agent -config=agent.hcl. The hand-rolled loop is fine for a single laptop and is easier to reason about; reach for Vault Agent once several services on the same machine each need independent lease management. Whichever you choose, tune the renewal interval against the TTL. A renewal that fires later than the TTL is a lockout; one that fires far too early wastes calls and shortens the useful test window during which you can observe rotation behavior. The chart below shows measured timings from a representative dev role, illustrating why a 45-second renewal interval against a 60-second TTL leaves a comfortable margin without over-renewing.

Drift check — alert when TTL drops below the rotation threshold:

#!/usr/bin/env bash
set -euo pipefail

TTL="$(vault lease lookup secret/dev/api-key -format=json | jq '.data.ttl')"
if [ "${TTL}" -lt 300 ]; then
  echo "CRITICAL: lease TTL ${TTL}s below 300s threshold"
fi
echo "Lease TTL check complete"

WSL2: Background cron/systemd may not survive restarts. Use nohup with PID tracking, or trigger the script via Windows Task Scheduler on login. Apple Silicon (ARM64): Ensure the devcontainer base image matches the host architecture; mismatched binaries fail during postCreateCommand.

Validate Secret Parity Against CI

Secret parity must be enforced programmatically, because the human eye will never catch a single missing key across two lists of forty. Parity here means one specific, checkable claim: the set of secret keys your local Vault exposes equals the set of keys your CI pipeline requires. It deliberately does not compare values — CI secrets should never leave the CI vault, and local values are throwaway — so the check exports both key sets, normalizes casing and ordering, and diffs them. A merge that adds a new required secret to CI without adding it locally now fails a check instead of failing a teammate's first run after they pull. Wire the result into CI/CD pipeline parity checks so merges block on mismatch.

# Makefile
.PHONY: validate-secrets-parity
validate-secrets-parity:
	@vault kv list -format=yaml secret/dev/ | yq eval '.[]' - | sort > /tmp/local_keys.txt
	@curl -s "$(CI_SECRET_MANIFEST_URL)" | yq eval '.required_keys[]' - | sort > /tmp/ci_keys.txt
	@diff -u /tmp/ci_keys.txt /tmp/local_keys.txt && echo "PARITY OK" || (echo "DRIFT DETECTED"; exit 1)

The sort on both sides is not cosmetic — diff is line-ordered, so unsorted lists report false drift on every reordering. Normalizing casing matters too if your CI manifest uses SCREAMING_SNAKE while Vault paths are lowercase; add a tr '[:upper:]' '[:lower:]' stage on both sides if that is your convention. Run the parity target as a pre-commit hook and again in CI itself, so the check runs both before a push and as an independent gate that a local hook cannot be skipped past with --no-verify. The decision tree below shows how to route the exit code: a clean diff proceeds to merge, a drift result branches to a remediation path that names the missing keys.

Drift check — run the parity target and act on the exit code:

#!/usr/bin/env bash
set -euo pipefail

make validate-secrets-parity
echo "Secret parity verified"

WSL2: CRLF in Makefile targets breaks execution. Run dos2unix Makefile or set core.autocrlf=input. Apple Silicon (ARM64): Run yq via docker run --rm -v "$(pwd):/work" mikefarah/yq for consistent cross-platform behavior.

Platform caveats

Beyond the per-section notes above, three cross-cutting behaviors trip up teams running this stack on mixed hardware. Group them into your onboarding docs so a new engineer on any platform reaches the same working state.

WSL2: The Vault dev server's in-memory storage is lost on every WSL restart, which is the intended dev behavior, but the ./vault-data bind mount to a /mnt/c path adds latency and can corrupt file locks. Keep the project — and therefore the mount — inside the native Linux filesystem. macOS (Docker Desktop): host.docker.internal resolves from inside the devcontainer to the host, so the VAULT_ADDR in remoteEnv must use that hostname, not 127.0.0.1, which inside the container points at the container itself. Apple Silicon (ARM64): The curl in the parity target may pull an amd64 manifest for the CI helper image; pin platform with docker run --platform=linux/arm64 where you shell out to containerized tools, or you will see exec format error intermittently.

Rollback and recovery

Failure modes here are recoverable because Vault, not your disk, holds the source of truth. The two most common incidents are a renewal daemon that corrupts .env.local mid-write, and a lease that gets revoked out from under an active session. Both recover the same way: stop the daemon, discard the local artifact, and refetch a clean value from Vault. Because KV v2 versions every write, you can also roll a secret back to a prior version if a bad value was seeded.

#!/usr/bin/env bash
set -euo pipefail

pkill -f rotate-local-secrets.sh || true
rm -f .env.local
vault kv get -format=json secret/dev/api-key | jq -r '.data.data.value' > .env.local
chmod 600 .env.local
echo "Local secret restored from Vault"

If a value rather than the file is the problem — someone seeded a broken password — recover the previous KV v2 version instead of the current one, then let the renewal daemon pick it up on its next cycle:

#!/usr/bin/env bash
set -euo pipefail

CURRENT="$(vault kv metadata get -format=json secret/dev/db | jq '.data.current_version')"
PREVIOUS=$(( CURRENT - 1 ))
vault kv rollback -version="${PREVIOUS}" secret/dev/db
echo "Rolled secret/dev/db back to version ${PREVIOUS}"

To tear the whole stack down cleanly at the end of a work session — releasing every lease so no orphaned credential lingers — stop the daemon, revoke the dev role's leases, and bring the Compose service down. Because dev-mode storage is in memory, a fresh docker compose up starts from a known-empty state and your seed script rebuilds it deterministically.

#!/usr/bin/env bash
set -euo pipefail

pkill -f rotate-local-secrets.sh || true
vault lease revoke -prefix secret/dev/ || true
docker compose down
echo "Vault stack down, all dev leases revoked"

Frequently Asked Questions

Related

• Compare Vault, dotenv-vault, and SOPS for local secrets
• Rotate secrets without restarting containers
• Keep local secrets out of Git
• Gate merges on CI/local parity