Multi-Service Orchestration with Compose

A stack of a database, cache, queue, API, and worker has to come up in the right order, share seed data deterministically, and tear back down to zero state — or it rots and stops matching CI. This guide gives platform engineers tactical steps for startup sequencing, shared state, network isolation, resource limits, and teardown. It is part of the broader containerized local environment patterns and builds on local network and port mapping for routing.

The failure mode this guide prevents is subtle: every service starts, docker compose ps reports everything as Up, and the API still throws connection refused on its first query because Postgres accepted a TCP socket before it finished replaying its write-ahead log. Onboarding a new engineer then costs an afternoon of "works on my machine" debugging that never reproduces on the maintainer's warm cache. The remedy is to treat the Compose file as an executable contract — health conditions, explicit networks, resource ceilings, and a one-command reset — so a fresh clone converges to the same running state every time.

Prerequisites

• Docker Compose v2 (the docker compose plugin, not the legacy docker-compose Python binary) with BuildKit enabled (DOCKER_BUILDKIT=1). Confirm with docker compose version and expect v2.20 or later so that --wait and --wait-timeout are available.
• jq for parsing docker compose ps --format json and docker network inspect, and dig (from dnsutils/bind-tools, or the nicolaka/netshoot image) for verifying service discovery.
• A committed seed manifest checksum (.seed-manifest.sha256) for drift detection, plus a .env file that is git-ignored but mirrored by a checked-in .env.example.
• The Compose file below assumes a single project. Set COMPOSE_PROJECT_NAME explicitly so resource names (networks, volumes) are stable across clones and not derived from the working-directory basename.

Everything that follows composes into one docker-compose.yml; the snippets are shown per concern for readability, but they merge into a single file. Each section closes with a drift-diagnostic command you can wire into a pre-commit hook or a CI smoke job. Keep the file under version control and treat any local edit to it the same way you treat a code change — reviewed, not improvised — because an untracked tweak to a port, a subnet, or a healthcheck timing is exactly the kind of divergence that makes one machine behave differently from the rest of the team's.

Before writing a line of YAML, run docker compose config after every change. It resolves variable interpolation, merges override files, and prints the fully materialized configuration, so a typo in an anchor or an unresolved ${VAR} fails at author time rather than at up time. Pair it with docker compose config --services in CI to assert that the set of declared services has not silently changed.

Section 1 - Service Dependency Graph and Startup Sequencing

Implicit depends_on guarantees container start order, not application readiness. Docker starts the dependency's container and immediately proceeds — it never waits for Postgres to accept queries or for Redis to finish loading its RDB snapshot. Replace bare ordering with condition: service_healthy so a dependent service blocks until its target reports healthy, and the dependency's own healthcheck defines what "healthy" means.

# docker-compose.yml
services:
  db:
    image: postgres:16-alpine
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres -d app_db"]
      interval: 3s
      timeout: 5s
      retries: 5
      start_period: 10s
  api:
    image: app/api:latest
    depends_on:
      db:
        condition: service_healthy

The --wait-timeout 90 on the command above is the outer bound: if the aggregate of every service's grace window and retries exceeds it, up gives up even on a stack that would eventually have come healthy, so keep the per-service start_period values honest and the timeout comfortably above their sum. The four healthcheck knobs are not interchangeable. interval is the gap between probes once the container is past its grace window; timeout is how long a single probe may hang before it counts as a failure; retries is the number of consecutive failures that flip the state to unhealthy; and start_period is a grace window during which failing probes do not count against the retry budget. For a database that replays WAL on boot, a too-short start_period marks the service unhealthy before it ever had a chance to open, and --wait aborts the whole up. Size start_period to the slowest cold start you observe, then add headroom.

1. Define explicit healthchecks for every infrastructure dependency (databases, caches, brokers). A container with no healthcheck is always healthy the instant it starts, which silently defeats service_healthy.

2. Replace bare depends_on with the long-form condition: service_healthy for infrastructure, and condition: service_completed_successfully for one-shot init or migration containers that must finish before the app starts.

3. For services lacking a native health endpoint, inject a lightweight readiness probe into the entrypoint — a curl -fsS localhost:PORT/healthz loop or a language-native check — rather than sleeping a fixed number of seconds.

4. Validate boot order immediately after up. The --wait flag makes up exit non-zero if any service fails to reach healthy within the timeout, which is exactly the assertion CI needs:

   #!/usr/bin/env bash
   set -euo pipefail
   docker compose up -d --wait --wait-timeout 90
   docker compose ps --format '{{.Name}} {{.Status}}'

Readiness gating is only half the contract; the other half is what happens when a dependency dies after startup. A restart: unless-stopped policy on infrastructure services means a crashed database is automatically brought back, but the application container that depended on it does not restart with it — depends_on conditions apply only at up, never afterward. Build the application's own retry-with-backoff into its database client so a mid-session restart of Postgres reconnects instead of wedging the whole stack. Reserve fixed sleep calls for nothing: they are either too short (the race still fires on a slow machine) or too long (every developer pays the tax on every boot), whereas a healthcheck adapts to the machine it runs on.

This maps directly onto the devcontainer configuration standards for IDE attachment ordering: an editor that attaches to the API container before the database is healthy will surface a transient connection error to the new hire on their very first launch. When a service still attaches before its dependency is ready — or a healthcheck flaps under load — the deep dive is resolving service startup order and healthcheck races.

Section 2 - Shared State and Seed Data Initialization

Deterministic environments need idempotent seed execution that survives restarts without manual intervention. The Postgres image runs any *.sql, *.sql.gz, or *.sh file mounted into /docker-entrypoint-initdb.d — but only on first boot, when the data directory is empty. Re-running docker compose up against an existing db_data volume does not re-run seeds, which is why a half-seeded volume is a common source of drift.

# docker-compose.yml
services:
  db:
    image: postgres:16-alpine
    environment:
      POSTGRES_DB: app_db
      POSTGRES_PASSWORD: ${DB_PASSWORD:?set DB_PASSWORD in .env}
    volumes:
      - ./db/init:/docker-entrypoint-initdb.d:ro
      - db_data:/var/lib/postgresql/data
volumes:
  db_data:

1. Author idempotent seed scripts (CREATE TABLE IF NOT EXISTS, INSERT ... ON CONFLICT DO NOTHING). Idempotency is what lets you re-run a seed against a warm volume — as a migration container or by hand — without duplicate-key errors.

2. Mount initialization directories read-only (:ro) so a runaway container process cannot rewrite the canonical manifests that every teammate depends on.

3. Run seeds via the entrypoint init directory for a cold volume, or a one-shot init container gated by condition: service_completed_successfully for repeatable migrations against a persistent volume.

4. Block compose up if local seed manifests diverge from the committed baseline. Hash the SQL files and compare against the checked-in checksum so an uncommitted edit to a seed fails fast instead of silently changing everyone's fixtures:

   #!/usr/bin/env bash
   set -euo pipefail
   expected="$(cat .seed-manifest.sha256)"
   actual="$(sha256sum db/init/*.sql | sha256sum | awk '{print $1}')"
   if [[ "$expected" != "$actual" ]]; then
     echo "DRIFT DETECTED: seed manifests diverge from baseline" >&2
     exit 1
   fi
   echo "seed manifests match baseline"

The ${DB_PASSWORD:?...} form is deliberate: the :? operator makes Compose refuse to start and print the message if the variable is unset, converting a silent misconfiguration into a loud, early failure. Keep the value in a git-ignored .env and document the key in .env.example so a fresh clone knows exactly which variables it must define. Named volumes such as db_data persist across docker compose down — they are only removed when you pass -v — which is what makes them the right home for state you want to keep between iterations but reset on demand.

Choose the volume type by intent, not habit. A named volume is managed by Docker, lives outside the project tree, and is the correct choice for database data directories: it avoids the permission and performance problems of bind-mounting a data directory from the host, especially on macOS and Windows where the host filesystem is not native Linux. A bind mount (./src:/app/src) is for source code you edit live, where you want host changes to appear inside the container. Mixing them up — bind-mounting a Postgres data directory, for instance — produces the classic initdb: directory exists but is not empty or Permission denied failures on first boot. When seed volume state itself is the source of drift, resetting is one docker compose down -v away, and the checksum gate above ensures the SQL that repopulates it matches the committed baseline byte for byte.

Section 3 - Network Isolation and Inter-Service Discovery

Default bridge networks allocate IPs unpredictably and risk port collisions when two projects run at once. Declare explicit topology so routing is reproducible and services address each other by stable DNS names rather than volatile IPs. Compose provides an embedded DNS resolver on every user-defined network: any service can reach any other by its service name, and aliases add extra hostnames on top.

# docker-compose.yml
networks:
  dev_overlay:
    driver: bridge
    ipam:
      config:
        - subnet: 172.28.0.0/16
services:
  cache:
    image: redis:7-alpine
    networks:
      dev_overlay:
        aliases:
          - cache.internal
  api:
    image: app/api:latest
    networks:
      - dev_overlay

1. Declare custom bridge networks with explicit IPAM subnets so the range never collides with a corporate VPN or another project on the same host. Pick a block inside 172.16.0.0/12 or 10.0.0.0/8 that your environment does not already route.

2. Prefer DNS aliases over hardcoded IPs for inter-service calls. Application config should read cache.internal:6379, never 172.28.0.5 — the alias survives a container recreation that changes the IP.

3. Split networks by trust boundary when a service should not be reachable from another: put the database on a backend network that the API joins but the public-facing edge does not, so an exposed port cannot become a path to your data tier.

4. Validate cross-service resolution from an ephemeral debug container attached to the same network:

   #!/usr/bin/env bash
   set -euo pipefail
   docker run --rm --network "$(docker compose ps --format '{{.Name}}' | head -1 | sed 's/-.*//')_dev_overlay" \
     nicolaka/netshoot dig +short cache.internal

A non-empty answer proves the alias resolves inside the network; an empty result means either the alias is misspelled or the debug container joined the wrong network. Note that a service's name resolves on the network regardless of container_name, so avoid setting container_name at all — it breaks scaling and adds nothing over the service name. Custom-TLD resolution beyond container names — mapping api.myapp.test to the stack, for instance — is covered in configuring local DNS for microservice routing.

Keep a hard line between internal connectivity and published ports. Two services on the same network reach each other on the container-internal port with no ports: mapping at all — the API talks to db:5432 whether or not 5432 is published to the host. Only publish a port when a process outside the Compose network (your browser, a native client, a debugger) needs it, and publish to an explicit host interface such as 127.0.0.1:5432:5432 so the service is not exposed on every network the laptop is attached to. Every unnecessary ports: entry is both an attack surface and a source of host-side collisions when two projects claim the same number. When you do scale a service with docker compose up --scale worker=3, the embedded DNS returns all three container IPs for the service name in round-robin, so client-side connection pooling spreads load without any external load balancer.

Section 4 - Resource Constraints and Local Performance Tuning

Unconstrained containers starve the host and hide production bottlenecks. A worker with no memory limit will happily consume every gigabyte on a laptop, freezing the editor, while the same code OOM-kills under a 512 MB production cgroup that no one modeled locally. Set quotas so throttling and OOM behavior surface on the developer's machine, where they are cheap to fix, instead of in staging.

# docker-compose.yml
services:
  worker:
    image: app/worker:latest
    deploy:
      resources:
        limits:
          cpus: "1.5"
          memory: 2G
  api:
    image: app/api:latest
    tmpfs:
      - /tmp:size=512m

1. Enforce per-service CPU/memory limits to simulate production cgroup quotas. In Compose v2 the deploy.resources.limits keys are honored by docker compose up on a single host — you do not need Swarm for them to take effect.

2. Use tmpfs for ephemeral logs and build scratch to bypass disk I/O; a memory-backed /tmp shaves seconds off test suites that write and delete thousands of small fixtures.

3. Add BuildKit cache mounts (RUN --mount=type=cache,target=/root/.cache) in Dockerfiles so iterative dependency resolution reuses the package cache instead of re-downloading on every rebuild.

4. Profile the running stack and watch for OOM kills, which show up in the kernel ring buffer even when the container simply "restarted":

   #!/usr/bin/env bash
   set -euo pipefail
   docker stats --no-stream --format 'table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}'
   dmesg 2>/dev/null | grep -i 'oom' || echo "no OOM kills"

Distinguish limits from reservations. A limit is a ceiling the container may never exceed — exceed the memory limit and the kernel OOM-kills the process; exceed the CPU limit and the scheduler throttles it. A reservation is a floor the scheduler tries to guarantee, useful on a busy host to keep a latency-sensitive service from being starved by a noisy neighbor. Model production with limits first, because a limit is what reproduces the failure mode you care about locally. For services that open many files or sockets — databases, brokers, connection-pooling proxies — also raise ulimits.nofile, since the container inherits a conservative default that surfaces as cryptic too many open files errors under load rather than as an obvious resource cap.

The measurable payoff of caching is large and worth quantifying for your own stack. The chart below shows representative cold-versus-warm rebuild times for a mid-sized API image: an uncached rebuild pays for every layer, a layer-cache hit skips unchanged steps, and a BuildKit cache mount additionally reuses the dependency download cache across builds. For file-sync latency that compounds a slow dev loop on macOS and Windows, review volume mounting and hot-reload optimization. To trim the stack to only the services a task needs — and stop paying resource cost for containers you are not using — see Compose profiles and targeted environments.

Section 5 - Automated Teardown and State Reset

Environment rot accumulates silently: an orphaned network from a renamed service, a stale volume with last month's schema, a dangling image that masks a broken build. Automate teardown so a fresh clone reproduces zero state and a single command returns any machine to a known baseline.

# Makefile
.PHONY: reset
reset:
	docker compose down -v --remove-orphans
	docker system prune -f
	docker compose build --no-cache

1. Add pre-push hooks that run docker compose down -v before state-altering operations so no one pushes with a wedged local stack they have grown used to working around.
2. Provide Makefile targets to purge named volumes (-v), dangling images (system prune), and orphaned networks (--remove-orphans) in one step, and document them in the README so the reset is discoverable.
3. Validate zero-state reproducibility by cloning into a fresh directory and running make reset && docker compose up -d --wait. If it converges from an empty checkout, onboarding will too.
4. Add a CI step that runs make reset and asserts exit code 0, so the reset path is exercised on every change rather than rotting until the day someone actually needs it.

Balance clean-state guarantees against cache loss with the tactics in optimizing Docker Compose for fast local rebuilds — --no-cache on every reset is correct for a nightly CI clean but wasteful for a routine local reset, where keeping the layer cache and only dropping volumes is usually enough.

Scope your pruning deliberately. A bare docker system prune -f reaches across every project on the host and will delete another stack's stopped containers and dangling images along with yours; on a machine that runs several projects, prefer the project-scoped docker compose down -v --remove-orphans --rmi local so the blast radius is exactly this stack. Compose stamps every resource it creates with com.docker.compose.project labels, which is how down knows what belongs to the project — you can list a stack's footprint with docker volume ls --filter label=com.docker.compose.project=$COMPOSE_PROJECT_NAME before you delete anything, turning a blind prune into a reviewed one. Finally, make the reset idempotent: running make reset twice in a row on an already-clean checkout must still exit 0, or the CI assertion in step four becomes flaky and teammates learn to ignore it.

Platform caveats

macOS (Docker Desktop): Healthcheck probes route through a Linux VM, adding ~200ms versus native Linux; widen interval/start_period accordingly. Prefer :cached source mounts. dmesg is not available on the host for OOM inspection — read docker inspect --format '{{.State.OOMKilled}}' <container> instead. WSL2: Enable systemd in wsl.conf so pg_isready finds its socket, and run wsl --shutdown before docker system prune if a volume lock hangs. Keep the repo on the Linux filesystem (/home/..., not /mnt/c) to avoid 9p I/O penalties during bulk seeding. Apple Silicon (ARM64): Pull architecture-specific healthcheck binaries or use CMD-SHELL wrappers to avoid exec format error; only set platform: linux/amd64 for images that lack an arm64 manifest, and expect the emulation penalty to inflate every healthcheck timing.

Rollback and recovery

If a stack wedges or volumes hold stale state, tear everything down, prune the project network, restore the committed configuration, and bring it back up with the readiness gate so recovery is verified, not assumed:

#!/usr/bin/env bash
set -euo pipefail
docker compose down -v --remove-orphans
docker network prune -f
git checkout HEAD -- docker-compose.yml
docker compose up -d --wait --wait-timeout 90

If up --wait still exits non-zero after this, the problem is not local state — it is a genuine healthcheck failure, and the next stop is the startup-order deep dive linked below rather than another prune.

Frequently Asked Questions

Related

• Resolve service startup order and healthcheck races
• Speed up local rebuilds with BuildKit caching
• Run only the services a task needs with profiles
• Diagnose containers that exit immediately on startup