Gracefully Stopping Dependent Services in Order

Running docker compose down kills your API mid-request — clients see connection reset by peer, background workers drop in-flight jobs, and the container exits with code 137 after hanging for exactly ten seconds — because the process was terminated with SIGKILL instead of being allowed to drain, and because a dependency can stop before the services that depend on it finish their work. This is the shutdown half of boot ordering, the mirror image of the startup sequencing covered in Multi-Service Orchestration with Compose: the same depends_on graph that decides who starts first also decides who stops last, and just as resolving service startup order and healthcheck races is about waiting for readiness on the way up, this is about waiting for drain on the way down. The fix has two parts: teach Compose the stop order, and make sure the termination signal actually reaches the application inside the container.

Ungraceful shutdown is quiet in development and loud in production-shaped environments. A single developer pressing Ctrl-C rarely notices a lost request, but the same compose file wired into an integration test that opens a streaming connection, or a load test that leaves sockets open, produces flaky failures that look like network faults. The exit code is the tell: 137 is 128 + 9, a SIGKILL, meaning the container was force-killed after ignoring the polite request to stop.

Diagnostic

Reproduce the failure by bringing the stack up, opening a slow request, and timing the teardown. A clean shutdown returns in well under the grace period; a broken one takes the full ten seconds and then force-kills.

#!/usr/bin/env bash
# reproduce the ungraceful stop and time it
set -euo pipefail
docker compose up -d
# fire a slow request in the background so there is in-flight work
curl -s "http://localhost:${APP_PORT:-3000}/slow" &
sleep 1
time docker compose down

# BAD: down blocks for the full grace period, then SIGKILLs
[+] Running 3/3
 ✔ Container app-worker-1  Removed   10.2s
 ✔ Container app-api-1     Removed   10.1s
 ✔ Container app-db-1      Removed    0.4s
real    0m10.6s
curl: (56) Recovery failed: Connection reset by peer

The ten-second wall is diagnostic on its own: 10s is the default stop_grace_period, so a service that takes exactly that long to stop never handled SIGTERM at all — Compose waited the full grace window and then sent SIGKILL. Confirm it by reading the exit code Compose recorded before the container was removed.

#!/usr/bin/env bash
# inspect how each container actually exited
set -euo pipefail
docker compose up -d
docker compose kill -s SIGTERM api
docker inspect --format '{{.Name}} exited {{.State.ExitCode}} ({{.State.Error}})' \
  "$(docker compose ps -q api)"

# BAD: 137 = 128 + 9 = SIGKILL, i.e. the app never stopped on its own
/app-api-1 exited 137 ()

An exit code of 137 means the kernel killed the process. A cleanly shutting-down service exits 0, or 143 (128 + 15) if it terminates on SIGTERM without an installed handler. Anything that reaches 137 was force-killed, which is the signature you are hunting.

Root cause

Two independent mechanisms combine to produce the failure, and both must be correct. First, ordering: when you run docker compose stop or down, Compose stops services in the reverse of their depends_on graph, so a service is stopped only after everything that depends on it. If you never declared depends_on, Compose has no graph, stops containers in an unspecified order, and a database can be torn down while the API is still writing to it. Second, signal delivery: Compose asks a container to stop by sending stop_signal (default SIGTERM) to PID 1 inside the container, waits up to stop_grace_period (default 10s), and then sends the un-catchable SIGKILL. If PID 1 is not your application — most commonly because the image uses the shell form CMD npm start, which runs your process as a child of /bin/sh — then the shell receives SIGTERM, does not forward it, and your application is never asked to stop. It runs until the grace window expires and is then killed outright, which is precisely the 137 exit and ten-second hang from the diagnostic.

The distinction between shell form and exec form is the crux. CMD npm start (shell form) becomes /bin/sh -c "npm start"; the shell is PID 1 and most shells do not propagate signals to their children. CMD ["npm", "start"] (exec form) makes npm — and ideally the node process it becomes via exec — PID 1, so SIGTERM lands on code you control. Even with exec form, a process manager that spawns children (or a runtime that will not reap zombies as PID 1) benefits from a tiny init such as tini, which Compose provides through init: true. So graceful shutdown is a three-legged stool: Compose must know the order, the signal must reach your code, and your code must actually handle it by closing listeners and finishing in-flight work before exiting.

Resolution

1. Declare depends_on on every dependent so Compose derives the reverse stop order and tears services down from the outside in.

# docker-compose.yml
services:
  db:
    image: postgres:16-alpine
    environment:
      POSTGRES_PASSWORD: postgres
      POSTGRES_DB: app_db

  api:
    build: .
    depends_on:
      - db
    ports:
      - "${APP_PORT:-3000}:3000"

  worker:
    build: .
    command: ["node", "worker.js"]
    depends_on:
      - api

With this graph Compose stops worker first, then api, then db. The dependency is only removed once nothing points at it, which is exactly what "drain before the dependency stops" requires. The short-form list is sufficient here because shutdown ordering needs only the edges, not the readiness conditions that startup gating needs.

2. Make your application PID 1 so the signal reaches it. Use the exec form in the Dockerfile, and add init: true as a safety net for runtimes that spawn children.

# Dockerfile
FROM node:20-alpine
WORKDIR /app
COPY package*.json ./
RUN npm ci --omit=dev
COPY . .
# exec form: node becomes PID 1 and receives SIGTERM directly
CMD ["node", "server.js"]

# docker-compose.yml (add to the api and worker services)
  api:
    build: .
    init: true
    depends_on:
      - db
    stop_grace_period: 30s
    stop_signal: SIGTERM

init: true inserts tini as PID 1, which forwards SIGTERM to your process and reaps any zombies it leaves behind. stop_signal is explicit here for readability even though SIGTERM is the default; set it only if your runtime expects a different signal (some servers drain on SIGQUIT). stop_grace_period: 30s widens the window so a real drain has time to finish before SIGKILL — size it to the longest request or job you expect, not to a round number.

3. Install a signal handler in the application that stops accepting new work, finishes what is in flight, closes its dependency connections, and exits 0.

// server.js — drain in-flight requests, then close the DB pool
const http = require("http");
const { pool } = require("./db"); // a pg Pool, closed on shutdown

const server = http.createServer((req, res) => {
  setTimeout(() => res.end("ok\n"), 2000); // simulate slow work
});
server.listen(process.env.PORT || 3000);

function shutdown(signal) {
  console.log(`received ${signal}, draining...`);
  server.close(async () => {          // stop accepting, finish in-flight
    await pool.end();                 // release the dependency last
    console.log("drain complete, exiting 0");
    process.exit(0);
  });
  // safety valve: never exceed the grace period silently
  setTimeout(() => {
    console.error("drain timed out, forcing exit");
    process.exit(1);
  }, 25000).unref();
}

process.on("SIGTERM", () => shutdown("SIGTERM"));
process.on("SIGINT", () => shutdown("SIGINT"));

The ordering inside the handler mirrors the ordering across services: stop the listener first so no new requests arrive, let in-flight requests complete, and only then release the connection to the dependency. Closing the database pool before server.close() finishes would reintroduce the exact failure at the application layer — requests still running would lose their connection.

4. Tear the stack down and confirm it stops cleanly and in order. Prefer docker compose stop when you want to keep volumes and networks, and pass an explicit timeout in scripts.

#!/usr/bin/env bash
# graceful, ordered teardown with an explicit timeout
set -euo pipefail
docker compose stop --timeout 30
docker compose ps --all --format '{{.Name}}\t{{.Status}}'

The --timeout flag overrides stop_grace_period for this invocation, which is useful in CI where you want a hard ceiling. It applies per container, so the whole teardown can take up to timeout multiplied by the number of services in the worst case, though a correctly draining stack finishes far sooner.

Expected output

[+] Stopping 3/3
 ✔ Container app-worker-1  Stopped   1.9s
 ✔ Container app-api-1     Stopped   2.1s
 ✔ Container app-db-1      Stopped   0.3s

# api logs during the stop — the handler ran
app-api-1  | received SIGTERM, draining...
app-api-1  | drain complete, exiting 0

docker inspect --format '{{.State.ExitCode}}' "$(docker compose ps -aq api)"
# 0

Three things confirm success: the stop order is worker, then api, then db; each service stops in roughly its real drain time rather than the flat ten-second wall; and the exit code is 0, not 137. If any service still takes the full grace period, its signal is not landing on your code — recheck that the Dockerfile uses exec form and that init: true is set.

Prevention

1. Assert exit codes in CI so a regression to 137 fails the build. A single check catches both a dropped signal handler and a Dockerfile that slips back to shell form during an edit.

#!/usr/bin/env bash
# ci: prove the stack stops gracefully, not by SIGKILL
set -euo pipefail
docker compose up -d
docker compose stop --timeout 30
for svc in worker api; do
  code="$(docker inspect --format '{{.State.ExitCode}}' "$(docker compose ps -aq "$svc")")"
  if [ "$code" = "137" ]; then
    echo "FAIL: $svc was SIGKILLed (exit 137) — signal not handled" >&2
    exit 1
  fi
done
echo "graceful shutdown OK"
docker compose down -v

2. Pin the stop signal in the image with STOPSIGNAL so the contract travels with the artifact rather than living only in the compose file, and keep CMD in exec form.

# Dockerfile
STOPSIGNAL SIGTERM
CMD ["node", "server.js"]

3. Right-size stop_grace_period to the measured worst-case drain, and treat any service that regularly hits its timeout as a bug in the handler, not a reason to raise the number. A grace period that keeps growing is hiding a handler that does not actually finish its work.

Platform caveats

macOS (Docker Desktop): signals cross the Linux VM boundary, so a docker compose stop can appear a few hundred milliseconds slower than on native Linux; do not confuse that overhead with a failed handler. Ctrl-C on a foreground docker compose up sends SIGINT, not SIGTERM, so make sure your handler covers both signals or the interactive path will still force-kill. WSL2: if the Docker engine is stopped abruptly (closing the WSL distro, a Windows sleep) containers are killed without a grace period at all, so never rely on interactive shutdown for correctness — the CI assertion is the real guarantee. Clock differences between the Windows host and the VM can also skew any time measurement of the teardown. Apple Silicon (ARM64): an emulated amd64 image runs its signal handler under QEMU and can drain measurably slower, so a stop_grace_period tuned on native arm64 may clip a legitimate drain; leave headroom or pull an arm64-native base image.

Rollback

#!/usr/bin/env bash
set -euo pipefail
git checkout -- docker-compose.yml Dockerfile
docker compose down --timeout 30
docker compose up -d

Frequently Asked Questions

Related

• Sequence multi-service boot order on the way up
• Gate startup on readiness with healthchecks
• Keep the rebuild loop fast without breaking teardown
• Diagnose containers that exit immediately on startup