Capstone stage 3 — data plane, edge, and the first request · Docker & Containers

Stage 3 is where the capstone becomes a running system. The order matters: data plane first, then the application, then the edge. Build it in the other direction and you spend the session debugging a reverse proxy that has nothing to route to.

Networks and secrets first

# compose.yml — the parts stage 3 adds
networks:
  edge-net:
  app-net:
  db-net:
    internal: true
  obs-net:

volumes:
  caddy-data:
  caddy-config:
  db-data:
  redis-data:

secrets:
  db_password:
    file: ./secrets/db_password
  grafana_admin:
    file: ./secrets/grafana_admin

db-net is declared internal: true, which is the single most valuable line in the file. An internal network gets no gateway and no masquerade rule, so the database cannot reach the internet and nothing outside the host can reach it — a stronger guarantee than a firewall rule, because there is no route to filter.

Configuration changecreate the secret files with correct permissions— Generated, never chosen by a human, and readable only by the operator account that runs Compose.

mkdir -p ./secrets
umask 077
openssl rand -base64 32 | tr -d '\n' > ./secrets/db_password
openssl rand -base64 32 | tr -d '\n' > ./secrets/grafana_admin
ls -l ./secrets/

total 8
-rw------- 1 ops ops 44 Aug 11 09:02 db_password
-rw------- 1 ops ops 44 Aug 11 09:02 grafana_admin

Illustrative output

tr -d '\n' matters more than it looks. openssl rand -base64 appends a newline, and Docker passes the file byte-for-byte into /run/secrets/. Postgres reads the trailing newline as part of the password, so the value in the file and the value the application sends differ by one byte, and the resulting authentication failure looks like a wrong password rather than a formatting bug. This costs people an hour roughly once per career.

`./secrets/` must be in .gitignore before it exists

Add the ignore rule in the same commit that adds the directory, not after. A secret committed once is in the history forever, and the remediation is a credential rotation plus a history rewrite that everyone with a clone must act on.

# .gitignore
secrets/
!secrets/.gitkeep

For anything beyond a capstone, a file-based Compose secret is a starting point rather than a destination — the file still sits on the host, unencrypted, with a copy in whatever provisioned it.

The data plane

services:
  db:
    image: postgres:16.4-bookworm
    environment:
      POSTGRES_DB: app
      POSTGRES_USER: app
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password
      PGDATA: /var/lib/postgresql/data/pgdata
    secrets: [db_password]
    volumes:
      - db-data:/var/lib/postgresql/data
    networks: [db-net]
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U app -d app"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 60s
    user: "999:999"
    security_opt: [ "no-new-privileges:true" ]
    cap_drop: [ALL]
    cap_add: [CHOWN, SETGID, SETUID, DAC_OVERRIDE, FOWNER]
    deploy:
      resources:
        limits: { memory: 2G, cpus: "2.0" }
    restart: unless-stopped

  cache:
    image: redis:7.4-alpine
    command: ["redis-server", "--maxmemory", "384mb", "--maxmemory-policy", "allkeys-lru", "--save", ""]
    volumes:
      - redis-data:/data
    networks: [db-net]
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 3s
      retries: 5
    user: "999:1000"
    read_only: true
    security_opt: [ "no-new-privileges:true" ]
    cap_drop: [ALL]
    deploy:
      resources:
        limits: { memory: 512M, cpus: "0.5" }
    restart: unless-stopped

Two things here are deliberate and often got wrong:

Postgres keeps five capabilities. The entrypoint runs as root briefly to fix ownership on PGDATA and then drops to the postgres user. cap_drop: [ALL] with nothing added makes the container fail at start with a chown error. Adding back the five it needs is the hardened position; pretending it can run with none is a container that does not start.
Redis maxmemory is set below the container limit. 384 MB against a 512 MB limit. Redis evicts when it reaches maxmemory; if maxmemory were above the container limit, the OOM killer would reach the process first and Redis would be killed rather than evicting. The eviction policy is the graceful degradation, and the gap between the two numbers is what lets it happen.

The application services

  api:
    image: ${api_DIGEST}
    environment:
      DATABASE_URL: "postgres://app@db:5432/app?sslmode=disable"
      DATABASE_PASSWORD_FILE: /run/secrets/db_password
      REDIS_URL: "redis://cache:6379/0"
      OTEL_EXPORTER_OTLP_ENDPOINT: "http://collector:4317"
    secrets: [db_password]
    networks: [app-net, db-net]
    depends_on:
      db: { condition: service_healthy }
      cache: { condition: service_healthy }
    healthcheck:
      test: ["CMD", "/app/api", "healthcheck"]
      interval: 15s
      timeout: 5s
      retries: 3
      start_period: 30s
    user: "65532:65532"
    read_only: true
    tmpfs: [ "/tmp:rw,noexec,nosuid,size=64m" ]
    security_opt: [ "no-new-privileges:true" ]
    cap_drop: [ALL]
    deploy:
      resources:
        limits: { memory: 512M, cpus: "1.0" }
    restart: unless-stopped

  worker:
    image: ${worker_DIGEST}
    environment:
      DATABASE_URL: "postgres://app@db:5432/app?sslmode=disable"
      DATABASE_PASSWORD_FILE: /run/secrets/db_password
    secrets: [db_password]
    networks: [db-net]
    depends_on:
      db: { condition: service_healthy }
    user: "65532:65532"
    read_only: true
    tmpfs: [ "/tmp:rw,noexec,nosuid,size=64m" ]
    security_opt: [ "no-new-privileges:true" ]
    cap_drop: [ALL]
    deploy:
      replicas: 2
      resources:
        limits: { memory: 256M, cpus: "0.5" }
    restart: unless-stopped

api is on app-net and db-net; worker is on db-net only, because nothing routes HTTP to it. web is on app-net only. Each service is on the minimum set of networks its role requires, and stage 3’s gate tests that claim rather than trusting it.

The edge

# Caddyfile
{
    email ops@example.com
}

app.example.com {
    encode zstd gzip
    reverse_proxy web:8080 {
        health_uri /health
        health_interval 10s
    }
    header {
        Strict-Transport-Security "max-age=31536000; includeSubDomains"
        X-Content-Type-Options "nosniff"
        -Server
    }
}

api.example.com {
    encode zstd gzip
    reverse_proxy api:8080 {
        health_uri /health
        health_interval 10s
    }
}

  caddy:
    image: caddy:2.8-alpine
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - caddy-data:/data
      - caddy-config:/config
    networks: [edge-net, app-net]
    depends_on:
      web: { condition: service_healthy }
      api: { condition: service_healthy }
    security_opt: [ "no-new-privileges:true" ]
    cap_drop: [ALL]
    cap_add: [NET_BIND_SERVICE]
    deploy:
      resources:
        limits: { memory: 128M, cpus: "0.25" }
    restart: unless-stopped

caddy-data must be a named volume, not a bind mount and not ephemeral. It holds the ACME account key and the issued certificates. Losing it means re-issuing every certificate on the next start, and Let’s Encrypt rate limits will stop you doing that more than a few times a week — which is how a routine docker compose down -v turns into a multi-hour outage.

Bringing it up

Configuration changevalidate, then start in order— `config --quiet` catches YAML and interpolation errors before anything runs.

docker compose --env-file digests.env config --quiet && echo 'compose valid'
docker compose --env-file digests.env up -d db cache
docker compose ps --format '{{.Service}}\t{{.State}}\t{{.Health}}'
docker compose --env-file digests.env up -d api worker web
docker compose --env-file digests.env up -d caddy

compose valid
db	running	healthy
cache	running	healthy

Illustrative output

Bringing the stack up in three explicit steps rather than one up -d is not superstition: it means a failure names its own stage. A failure after step one is a data-plane problem, after step two an application problem, after step three an edge or DNS problem. One combined command gives you eleven containers and a wall of output.

The stage 3 gate

Read-only / Safeeverything healthy— Compose reports Health per service. Any service not healthy fails the gate.

docker compose ps --format json \
  | jq -r '"\(.Service)\t\(.State)\t\(.Health)"'
docker compose ps --format json \
  | jq -se '[.[] | select(.Health != "" and .Health != "healthy")] | length == 0' \
  && echo 'OK: all health-checked services healthy'

caddy	running	
db	running	healthy
cache	running	healthy
api	running	healthy
web	running	healthy
worker	running	
OK: all health-checked services healthy

Illustrative output

Two details in that gate are worth stealing:

docker compose ps --format json emits one JSON object per line, not an array. jq '.[] | ...' fails with Cannot index string with string. Either address the fields directly, as the first command does, or slurp with jq -s when you need to reason about the set, as the second does.
A service with no healthcheck reports an empty Health string, not unhealthy. A naive == "healthy" test fails caddy and worker forever, so the filter excludes the empty case explicitly. Deciding whether that is acceptable is part of the gate — for the capstone, worker should gain a health check in stage 4, once its queue depth is observable.

Read-only / Safethe first end-to-end request— Through the public name, over TLS, with a certificate the client validates.

curl -fsS -o /dev/null -w 'status=%{http_code} tls=%{ssl_verify_result} time=%{time_total}s\n' \
  https://app.example.com/health
curl -fsS https://api.example.com/health

status=200 tls=0 time=0.184s
{"status":"ok","db":"ok","cache":"ok"}

Illustrative output

ssl_verify_result=0 is the part that matters. A 200 over a certificate curl did not validate is not a passing edge.

Read-only / Safecertificate identity and expiry— Subject, issuer and dates, read from the served certificate rather than from the ACME logs.

echo | openssl s_client -connect app.example.com:443 -servername app.example.com 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates

subject=CN=app.example.com
issuer=C=US, O=Let's Encrypt, CN=R11
notBefore=Aug 11 08:02:11 2026 GMT
notAfter=Nov  9 08:02:10 2026 GMT

Illustrative output

The negative tests

A gate made only of things that should work proves half the design. These four must fail:

# 1. The edge must not reach the database.
docker compose exec caddy sh -c 'nc -z -w2 db 5432; echo "exit=$?"'
# Expected: exit=1

# 2. The database must not reach the internet.
docker compose exec db sh -c 'timeout 3 getent hosts example.com; echo "exit=$?"'
# Expected: non-zero — db-net is internal, so there is no resolver route out

# 3. The database port must not be published on the host.
ss -tlpn | grep ':5432' || echo 'OK: postgres not listening on the host'

# 4. No secret in any container environment.
docker compose ps -q | xargs -r -I{} docker inspect --format \
  '{{.Name}} {{range .Config.Env}}{{println .}}{{end}}' {} \
  | grep -iE '(password|secret|token)=..' && echo 'FAIL' || echo 'OK: no secrets in env'

Run these in the deploy pipeline, not once. Test 1 in particular reverses silently the first time somebody runs docker network connect db-net caddy to debug something.

How Compose secrets actually reach the container

A Compose secrets: entry with a file: source is not an environment variable and not a bind mount of the original file. The daemon creates a tmpfs at /run/secrets in the container and writes each secret into it as a file mode 0444, owned by the container’s user.

Three consequences:

The secret never touches the container’s writable layer, so it cannot end up in a docker commit or a diff.
It lives in memory, so it is not on disk inside the container and disappears with the container.
Rotation requires recreating the container. The tmpfs is populated at creation; changing the source file changes nothing for a running container. An application that re-reads /run/secrets/db_password on each connection still sees the old value until it is recreated.

That last point is why the capstone’s rotation procedure in stage 5 is “update the file, then docker compose up -d --force-recreate api worker”, and why a rotation that only rewrites the file appears to succeed and changes nothing.

What happens if Caddy cannot obtain a certificate?

Caddy starts, serves HTTP, and retries ACME with backoff. The service is up and broken, which is the worst state for a health check that only looks at the container.

The three causes, in the order they occur in practice:

DNS does not resolve to this host yet. The HTTP-01 challenge requires the name to point at the machine answering port 80. Check with getent hosts app.example.com from outside the host, not from the host itself, where /etc/hosts may be lying to you.
Port 80 is not reachable from the internet. The challenge cannot use 443 alone. A firewall permitting only 443 fails ACME while looking correct.
Rate limits. Repeated failed attempts against the production endpoint exhaust the limit for the name, and the wait is measured in days. Use the staging endpoint while iterating: acme_ca https://acme-staging-v02.api.letsencrypt.org/directory in the Caddyfile global block, then remove it for the real issuance.

Read Caddy’s own logs first: docker compose logs caddy | grep -i acme names the failing challenge directly.

Generate the secret files with umask 077 and no trailing newline, and confirm secrets/ is ignored by git.
Start the data plane and wait for both health checks before continuing.
Start the application services, confirming api reports healthy — which for the capstone means its database and cache checks pass.
Start the edge and watch the ACME exchange in the Caddy logs.
Run the positive gate: all health-checked services healthy, both public endpoints returning 200 with ssl_verify_result=0, certificate expiry more than 30 days out.
Run the negative gate: edge cannot reach the database, database cannot reach the internet, 5432 not published, no credentials in any container environment.

Networks and secrets first

The data plane

The application services

The edge

Bringing it up

The stage 3 gate

The negative tests

Sanity check