Using GCP Pub/Sub and Firestore Emulators in Compose

Run Google's official Pub/Sub and Firestore emulators as Compose services, create topics on startup, and fix clients that ignore EMULATOR_HOST and hit real GCP.

The service starts locally and immediately fails with google.auth.exceptions.DefaultCredentialsError: Your default credentials were not found, or 7 PERMISSION_DENIED: User not authorized to perform this action from Pub/Sub — both signs that the client library is trying to reach real Google Cloud instead of the emulator you started. Google ships official emulators for Pub/Sub, Firestore, Datastore, Bigtable and Spanner; this page runs the two most common ones as Compose services, creates topics and subscriptions at startup, and makes sure every client actually uses them. It belongs to emulating cloud services locally.

Unlike LocalStack for AWS, the Google emulators are separate processes, one per service, each shipped in the google-cloud-cli emulators image. They are official, so their API behaviour tracks the real services closely, but they are also minimal: no IAM, no console, and no automatic resource creation.

Diagnostic

Check the emulator variables inside the application container and whether a client call reaches the emulator:

#!/usr/bin/env bash
set -euo pipefail
docker compose exec -T api env | grep -E 'EMULATOR_HOST|GOOGLE_CLOUD_PROJECT|GOOGLE_APPLICATION_CREDENTIALS' || echo "no emulator variables set"
docker compose exec -T api python -c "
from google.cloud import pubsub_v1
p = pubsub_v1.PublisherClient()
print([t.name for t in p.list_topics(request={'project': 'projects/local-dev'})])" 2>&1 | tail -2

Expected bad output:

no emulator variables set
google.auth.exceptions.DefaultCredentialsError: Your default credentials were not found.

With no PUBSUB_EMULATOR_HOST, the client looks for Application Default Credentials to call the real API and fails because none exist in the container — which is the good outcome. On a laptop where ADC is configured and mounted, the same call would reach a real project.

How a Client Chooses Emulator or Cloud Flow showing the client library checking for the emulator host variable before falling back to real credentials. How a Client Chooses Emulator or Cloud client starts reads env host variable? yes: plaintext emulator no auth, local else ADC, real project
The emulator variable is the switch; when it is missing, the client goes to real GCP.

Root cause

Google Cloud client libraries switch to an emulator only when a service-specific variable is present: PUBSUB_EMULATOR_HOST, FIRESTORE_EMULATOR_HOST, DATASTORE_EMULATOR_HOST, and so on. When the variable is set, the client connects over plaintext gRPC and skips authentication. When it is missing, the client uses Application Default Credentials and the real endpoint. The variables are per service, so setting the Pub/Sub one does nothing for Firestore. And the emulators start empty: topics and subscriptions must be created by something before the application publishes, and the project ID the application uses must match the one used to create them, or the application sees NOT_FOUND.

A second trap is the host format. The variables take host:port without a scheme — pubsub:8085, not http://pubsub:8085. Some libraries tolerate the scheme; others fail with a confusing DNS error for a host literally named http. Keep the value scheme-less everywhere.

Resolution

  1. Run each emulator as a service from the official image, binding to all interfaces so other containers can connect:
services:
  pubsub:
    image: gcr.io/google.com/cloudsdktool/google-cloud-cli:492.0.0-emulators
    command: gcloud beta emulators pubsub start --project=local-dev --host-port=0.0.0.0:8085
    healthcheck:
      test: ["CMD-SHELL", "curl -fsS http://localhost:8085 || exit 1"]
      interval: 3s
      retries: 30

  firestore:
    image: gcr.io/google.com/cloudsdktool/google-cloud-cli:492.0.0-emulators
    command: gcloud emulators firestore start --host-port=0.0.0.0:8086
    healthcheck:
      test: ["CMD-SHELL", "curl -fsS http://localhost:8086 || exit 1"]
      interval: 3s
      retries: 30
  1. Set the emulator variables and a fixed project ID on every service that uses the clients:
x-gcp-local: &gcp-local
  GOOGLE_CLOUD_PROJECT: local-dev
  PUBSUB_EMULATOR_HOST: pubsub:8085
  FIRESTORE_EMULATOR_HOST: firestore:8086

services:
  api:
    environment:
      <<: *gcp-local
    depends_on:
      pubsub-init:
        condition: service_completed_successfully
      firestore:
        condition: service_healthy
  1. Create topics and subscriptions in a one-shot init service. The emulator's REST interface accepts plain PUT requests, so curl is enough — no SDK needed:
services:
  pubsub-init:
    image: curlimages/curl:8.9.1
    depends_on:
      pubsub:
        condition: service_healthy
    entrypoint: ["/bin/sh", "-c"]
    command:
      - |
        set -e
        base=http://pubsub:8085/v1/projects/local-dev
        curl -fsS -X PUT "$$base/topics/orders" -o /dev/null || true
        curl -fsS -X PUT "$$base/subscriptions/orders-worker" -H 'content-type: application/json' \
          -d '{"topic":"projects/local-dev/topics/orders","ackDeadlineSeconds":30}' -o /dev/null || true
        echo "pubsub resources ready"

service_completed_successfully makes the API wait until the init container exits 0, so topics exist before the first publish.

  1. Keep ADC out of containers. Do not mount ~/.config/gcloud into application services; with emulator variables set it is unnecessary, and without them it turns a misconfiguration into writes against a real project.
Startup Order With Emulators Ordered startup from emulator containers to init job to application. Startup Order With Emulators 1 — pubsub and firestore start 2 — healthchecks pass 3 — pubsub-init creates topics exits 0 4 — api starts with EMULATOR_HOST set 5 — first publish succeeds
The init job's successful exit is the signal the application waits on.

Expected output

$ docker compose exec -T api python -c "from google.cloud import pubsub_v1; p=pubsub_v1.PublisherClient(); print([t.name for t in p.list_topics(request={'project':'projects/local-dev'})])"
['projects/local-dev/topics/orders']
$ docker compose logs pubsub --since 1m | grep -i publish | tail -1
pubsub-1  | [pubsub] INFO: Detected HTTP/2 connection. ... Publish request, 1 messages

The topic created by the init job is visible to the application, and the emulator logs the publish — no credentials were needed and nothing touched a real project.

For push subscriptions, where Pub/Sub calls an HTTP endpoint instead of the worker pulling, set pushConfig.pushEndpoint in the subscription body to the worker's service name, for example http://worker:8080/pubsub. The emulator delivers from inside the Compose network, so the endpoint must use a name the emulator container can resolve, never localhost. Delivery attempts and failures appear in the emulator's log, which is the quickest way to see whether a push handler returned a non-2xx status and is being retried.

Prevention

  1. Fail fast without emulator variables in local mode. At startup, if APP_ENV=local and any Google client is configured, require the matching *_EMULATOR_HOST variable. A missing variable is a configuration bug, not a reason to fall back to real credentials.

  2. Generate the init requests from infrastructure code. Keep topic and subscription names, ack deadlines and dead-letter policies in the same source as production (Terraform google_pubsub_topic resources), and generate the curl calls from it so they cannot drift.

  3. Pin the emulator image version and update it deliberately; emulator behaviour changes between SDK releases.

Emulator Variables by Service Table listing the Google Cloud services, their emulator variable and the default port used here. Emulator Variables by Service Service Variable Port here Pub/Sub PUBSUB_EMULATOR_HOST 8085 Firestore FIRESTORE_EMULATOR_HOST 8086 Datastore DATASTORE_EMULATOR_HOST 8081 Bigtable BIGTABLE_EMULATOR_HOST 8087
Each service has its own variable; setting one does not redirect the others.

Platform caveats

Apple Silicon (ARM64): recent google-cloud-cli emulator images are multi-arch. The Firestore and Datastore emulators are Java processes; give them 512 MB each and expect a slower first start than Pub/Sub.

macOS (Docker Desktop): the emulators keep all data in memory. Restarting the container empties them, which is useful for tests but means init jobs must run on every start — the service_completed_successfully dependency handles that.

WSL2: host-side tools using the emulators need localhost ports published and the variable set to localhost:8085 in the host shell, while containers keep pubsub:8085.

Rollback

#!/usr/bin/env bash
set -euo pipefail
docker compose rm -sf pubsub firestore pubsub-init
git restore compose.yaml

Without the emulator variables, clients revert to their normal behaviour, which in a container without credentials is a clear authentication error.

Frequently Asked Questions

Why does the client still ask for credentials with the emulator running?

The client does not detect a running emulator; it only checks the service-specific variable. Set PUBSUB_EMULATOR_HOST or FIRESTORE_EMULATOR_HOST in the container's environment, without an http:// prefix, and restart the process.

Does the project ID matter when using emulators?

Yes. Resources are namespaced by project inside the emulator. Create topics under the same project ID the application uses, set through GOOGLE_CLOUD_PROJECT, or the application sees NOT_FOUND for topics that exist under another ID.

Can the Firestore emulator import production data?

It can import an export made with gcloud firestore export using --import-data on start. Treat any such data as sensitive and anonymise it first; for most development, seeded fixtures are safer and more reproducible.

Is there a UI for the emulators?

Google's standalone emulators have none. The Firebase Local Emulator Suite provides a UI for Firestore and related services if the project uses Firebase tooling; otherwise use the client libraries or REST calls to inspect state.