Capstone — the production Docker environment · Docker & Containers

The capstone is a single environment that exercises every lesson in the course. It is not a complete production system; it is a template that demonstrates the patterns.

The architecture

flowchart TB
  subgraph Internet
    User[User]
  end
  subgraph Edge[Edge]
    RP[Caddy / Traefik<br/>TLS termination]
  end
  subgraph App[Application]
    Web[Web service]
    API[API service]
    Worker[Background worker]
  end
  subgraph Data[Data plane]
    DB[(Postgres)]
    Cache[(Redis)]
  end
  subgraph Obs[Observability]
    Prom[Prometheus]
    Grafana
    Loki
    Tempo
  end
  Internet --> RP
  RP --> Web
  RP --> API
  API --> DB
  API --> Cache
  Worker --> DB
  Prom --> Web
  Prom --> API
  Prom --> Worker
  Prom --> DB
  Prom --> Cache
  Prom --> RP
  Grafana --> Prom
  Grafana --> Loki
  Grafana --> Tempo

What’s in the environment

Reverse proxy (Caddy or Traefik). TLS termination, ACME, routing by hostname.
Web + API + worker services. Each runs as a Compose service with appropriate limits.
Postgres + Redis. Stateful services with named volumes and backups.
Prometheus + Grafana + Loki + Tempo. The standard observability stack.

The compose file

# compose.yml
services:
  caddy:
    image: caddy:2
    ports: ["80:80", "443:443"]
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - caddy_data:/data
      - caddy_config:/config
    restart: unless-stopped
    user: "0:0"  # Caddy needs root for :80/:443; reviewed acceptable
    read_only: true
    # /data holds the ACME account key and issued certificates and MUST be
    # persistent - see the callout below. Only genuinely scratch paths go
    # on tmpfs.
    tmpfs: ["/tmp"]
    security_opt:
      - no-new-privileges:true
    cap_drop: [ALL]
    cap_add: [NET_BIND_SERVICE]

  web:
    image: myorg/web:1.0.0
    networks: [app-net]
    depends_on:
      api:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "wget", "-qO-", "http://localhost:8080/health"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 30s
    user: "10001:10001"
    read_only: true
    tmpfs: ["/tmp", "/var/cache/myapp"]
    security_opt:
      - no-new-privileges:true
    cap_drop: [ALL]
    cap_add: [NET_BIND_SERVICE]
    deploy:
      resources:
        limits:
          memory: 256M
          cpus: "0.5"

  api:
    image: myorg/api:1.0.0
    environment:
      DATABASE_URL_FILE: /run/secrets/db_password
    secrets: [db_password]
    networks: [app-net, db-net]
    depends_on:
      db:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "wget", "-qO-", "http://localhost:8080/live"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 60s
    user: "10001:10001"
    read_only: true
    tmpfs: ["/tmp", "/var/cache/myapp"]
    security_opt:
      - no-new-privileges:true
    cap_drop: [ALL]
    cap_add: [NET_BIND_SERVICE]
    deploy:
      resources:
        limits:
          memory: 512M
          cpus: "1.0"

  worker:
    image: myorg/worker:1.0.0
    networks: [db-net]
    depends_on:
      api:
        condition: service_healthy
    user: "10001:10001"
    security_opt:
      - no-new-privileges:true
    cap_drop: [ALL]
    deploy:
      replicas: 2
      resources:
        limits:
          memory: 256M
          cpus: "0.5"

  db:
    image: postgres:16
    environment:
      POSTGRES_DB: app
      POSTGRES_USER: app
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password
    secrets: [db_password]
    volumes:
      - db-data:/var/lib/postgresql/data
    networks: [db-net]
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U app"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 30s
    user: "999:999"  # postgres user in the image
    security_opt:
      - no-new-privileges:true

  cache:
    image: redis:7-alpine
    networks: [app-net, db-net]
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 30s
      timeout: 5s
      retries: 3
    user: "999:999"
    security_opt:
      - no-new-privileges:true

  prometheus:
    image: prom/prometheus:v2.55.0
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prom-data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.enable-lifecycle'
    ports: ["9090:9090"]
    networks: [obs-net]
    user: "65534:65534"  # nobody
    security_opt:
      - no-new-privileges:true

  grafana:
    image: grafana/grafana:11.3.0
    environment:
      GF_SECURITY_ADMIN_PASSWORD_FILE: /run/secrets/grafana_admin
    secrets: [grafana_admin]
    ports: ["3000:3000"]
    networks: [obs-net]

  loki:
    image: grafana/loki:3.3.0
    command: -config.file=/etc/loki/local-config.yaml
    volumes:
      - loki-data:/loki
    networks: [obs-net]

secrets:
  db_password:
    file: ./secrets/db_password.txt
  grafana_admin:
    file: ./secrets/grafana_admin.txt

volumes:
  caddy_data:
  caddy_config:
  db-data:
  prom-data:
  loki-data:

networks:
  app-net:
  db-net:
  obs-net:

Why every service drops ALL capabilities

The Compose file shows --cap-drop=ALL on every service. The production minimum is the smallest set that lets the workload run. Most services need only NET_BIND_SERVICE (to listen on ports below 1024) or nothing at all.

A misbehaving process inside a container with the default capability set can do substantial damage even without escaping the container. Drop the capabilities, add back only what is needed.

Note the interaction with user:. cap_add raises the bounding set; a container whose entrypoint runs as a non-root UID starts with an empty effective set regardless, so NET_BIND_SERVICE on the web and api services — both user: "10001:10001" — does nothing for them. Those services listen on 8080 and are published by the proxy, so they never needed it. It is harmless here and it is the kind of copied line that becomes a misleading answer in a security review; the lesson “Linux capabilities” has the full mechanism.

tmpfs over the ACME storage: TLS fails four days after a deploy that went fine

An earlier revision of the Compose file above declared both a named volume and a tmpfs at the same path on the reverse proxy:

volumes:
  - caddy_data:/data
tmpfs: ["/data", "/config"]

The deploy is clean. TLS works. Certificates are valid. Nothing is wrong for days — and that delay is the whole problem, because it puts the cause outside the window anyone is looking at.

/data is where Caddy keeps its ACME account key and its issued certificates. On tmpfs, that directory is empty on every container start. So every restart, every compose up -d after a config change, every host reboot, throws away the account state and issues a brand-new certificate for the same names.

Let’s Encrypt’s duplicate-certificate limit is 5 per identical name set per 168 hours. A service that restarts a few times a day consumes that in under two days and then starts failing issuance. Now the container starts with no certificate, cannot get one, and the site is down on TLS — days after the change that caused it, during a restart that had nothing to do with certificates.

Why the first diagnosis goes wrong. The failure appears during an unrelated restart, so it is attributed to that restart. The error is a rate-limit response from the CA, which reads as “the CA is having problems” or “our ACME client is misconfigured”. Nobody looks at a tmpfs: line that was reviewed, approved and working a week ago.

And the rate limit is a wall, not a warning. You cannot fix it by redeploying. Once you are limited, the only options are to wait out the window or move to a different name set. Correct the volume first, then wait.

How to tell. Persistent ACME storage survives a restart. Check that the certificate’s notBefore is older than the container’s StartedAt — if it is newer on every restart, you are re-issuing every time.

Read-only / Safeis the certificate surviving restarts, or being re-issued?— Compare when the certificate was issued with when the container last started. A certificate consistently younger than the container means storage is not persisting and you are burning CA rate limit on every restart.

DOMAIN=app.example.com
docker inspect --format 'container started: {{.State.StartedAt}}' caddy
echo | openssl s_client -connect "$DOMAIN:443" -servername "$DOMAIN" 2>/dev/null \
  | openssl x509 -noout -dates
docker compose exec caddy sh -c 'ls -la /data/caddy/certificates 2>/dev/null || echo "NO PERSISTENT ACME STORAGE"'

container started: 2026-08-12T03:04:11Z
notBefore=Jul 28 09:41:00 2026 GMT
notAfter=Oct 26 09:40:59 2026 GMT
drwx------ 3 root root 4096 Jul 28 09:41 acme-v02.api.letsencrypt.org-directory

Illustrative output

notBefore two weeks before the container started is the pass condition: the certificate outlived the restart, so storage is persistent. A notBefore within seconds of StartedAt, or the NO PERSISTENT ACME STORAGE branch, is the failure — and it is worth checking on any proxy you inherit rather than waiting for the rate limit to find it for you.

Definition of Done

The capstone is done when every item below is verified with a running command and a positive result. No item is “should be”; each is “this command exits 0 and shows the expected output”.

A green check that cannot go red is worse than no check at all

Before you trust any of the checks below, make each one fail on purpose. A Definition of Done is a set of claims about production, and a claim that cannot be falsified is not evidence — it is decoration that has been audited, which is worse than a gap somebody knows about.

The failure mode is specific and it is easy to ship. Consider:

docker compose ps --format json | jq -e '[.[] | select(.Privileged == true)] | length == 0' \
  && echo "OK: no privileged"

That reads like a check. It is not, for two independent reasons, either of which is enough on its own:

docker compose ps --format json emits newline-delimited JSON objects, not a JSON array. .[] applied to an object iterates its values, which are strings, so select(.Privileged == true) tries to index a string and jq aborts with Cannot index string with string "Privileged" and exit 5.
There is no Privileged field anyway. The available keys are Command, CreatedAt, ExitCode, Health, ID, Image, Labels, LocalVolumes, Mounts, Name, Names, Networks, Ports, Project, Publishers, RunningFor, Service, Size, State, Status. Privileged mode, memory limits, CPU limits, read-only rootfs and capability sets are all container configuration, and they live in docker inspect, not in compose ps.

What that looks like when you run it. jq exits non-zero, && short-circuits, and echo "OK: no privileged" never runs. The script prints a jq error and moves on. If someone pipes the output through grep -c OK, or reads only the summary, or runs it under a CI step that does not use set -o pipefail, the missing line looks like ordinary noise. Nobody notices that the security section of the Definition of Done has been reporting nothing for months.

Why it survives review. The command is plausible, it is in the right section, it uses jq -e which looks rigorous, and it has a sensible success message. Reviewers read what the check intends. The only way to know what it does is to run it against a violation and watch it complain.

The rule. Every check in this section must be tested twice: once against a compliant environment, where it must print its OK line, and once against a deliberately non-compliant one — start one container with --privileged, drop one memory limit, unset one read-only flag — where it must fail. A check that prints OK in both runs is removed or rewritten. There is no third option.

Two conventions make the rest of this section trustworthy. First, every check names the offenders when it fails rather than only saying “FAIL” — a check you cannot act on gets ignored on the second occurrence. Second, container configuration is read from docker inspect, never from docker compose ps: compose ps reports runtime facts (state, health, ports, mounts) and does not carry privilege, limits, capabilities or read-only status at all.

Functional

set -o pipefail

# Compose config validates
docker compose -f compose.yml config --quiet && echo "OK: compose valid"

# Every service running, and healthy where a healthcheck is declared.
# jq -s is required: compose ps emits newline-delimited objects, not an array.
BAD=$(docker compose ps --format json | jq -s -r '
  .[] | select(.State != "running" or (.Health != "" and .Health != "healthy"))
      | "\(.Service) state=\(.State) health=\(.Health)"')
[ -z "$BAD" ] && echo "OK: all services running and healthy" \
  || { echo "FAIL: not running or unhealthy:"; echo "$BAD"; }

# Every service that should have a healthcheck has one. An empty Health
# field is not a pass - it means nothing is being checked.
docker compose ps --format json | jq -s -r '
  .[] | select(.Health == "") | "WARN: \(.Service) has no healthcheck"'

# Public endpoint reachable
curl -fsS https://app.example.com/health
# Expected: 200 "ok"

# API endpoint reachable
curl -fsS https://api.example.com/v1/orders
# Expected: 200 application/json

Security

set -o pipefail
CIDS=$(docker compose ps -q)

# No privileged containers
BAD=$(docker inspect $CIDS --format '{{.Name}} {{.HostConfig.Privileged}}' | grep ' true$' || true)
[ -z "$BAD" ] && echo "OK: no privileged" || { echo "FAIL: privileged:"; echo "$BAD"; }

# Capability sets, per container. cap_drop must be ALL; cap_add must be
# empty or exactly the documented exception.
docker inspect $CIDS --format \
  '{{.Name}} user={{.Config.User}} drop={{json .HostConfig.CapDrop}} add={{json .HostConfig.CapAdd}}'
# Expected: drop=["ALL"] on every line; add=["NET_BIND_SERVICE"] only on caddy

# no-new-privileges on every container
BAD=$(docker inspect $CIDS --format \
  '{{.Name}} {{range .HostConfig.SecurityOpt}}{{.}} {{end}}' \
  | grep -v 'no-new-privileges' || true)
[ -z "$BAD" ] && echo "OK: no-new-privileges everywhere" \
  || { echo "FAIL: missing no-new-privileges:"; echo "$BAD"; }

# Nothing runs as root
BAD=$(docker inspect $CIDS --format '{{.Name}} user={{.Config.User}}' \
  | grep -E 'user=$|user=0(:|$)' || true)
[ -z "$BAD" ] && echo "OK: no root containers" \
  || { echo "REVIEW: running as root (caddy is the documented exception):"; echo "$BAD"; }

# No Docker socket mounts
BAD=$(docker inspect $CIDS --format '{{.Name}} {{range .Mounts}}{{.Source}} {{end}}' \
  | grep 'docker.sock' || true)
[ -z "$BAD" ] && echo "OK: no socket mounts" || { echo "FAIL: socket mounted:"; echo "$BAD"; }

# Read-only root filesystem where declared
BAD=$(docker inspect $CIDS --format '{{.Name}} ro={{.HostConfig.ReadonlyRootfs}}' \
  | grep -E '/(web|api)\s.*ro=false' || true)
[ -z "$BAD" ] && echo "OK: read-only enforced on web and api" \
  || { echo "FAIL: writable rootfs:"; echo "$BAD"; }

# No secrets in env
docker compose exec -T api printenv | grep -iE 'password|secret|token' \
  && echo "FAIL: secret in env" || echo "OK: no secrets in env"

# Secrets only in /run/secrets
docker compose exec -T api ls /run/secrets/
# Expected: db_password only

# No critical CVEs
trivy image --severity CRITICAL --exit-code 1 myorg/web:1.0.0 myorg/api:1.0.0 myorg/worker:1.0.0
# Expected: exit 0, no CRITICAL findings

Prove the security block can fail before you sign it

Run this once, on a scratch host, and confirm each check names the container it should:

docker run -d --name dod-canary --privileged -v /var/run/docker.sock:/var/run/docker.sock alpine sleep 600

Point the security checks at it — a CIDS that includes dod-canary — and every relevant one must report FAIL with dod-canary in the output: privileged, socket mount, no no-new-privileges, running as root, no memory limit, no CPU limit. If any of them stays silent or prints OK, that check is broken and the fix is the check, not the canary. Then docker rm -f dod-canary.

This takes two minutes and it is the only thing that turns the Definition of Done from a document into a control.

Networking

# SSL chain valid
echo | openssl s_client -connect app.example.com:443 -servername app.example.com 2>/dev/null | \
  openssl x509 -noout -subject -dates -issuer
# Expected: subject CN=app.example.com; notAfter > 30 days; issuer = Let's Encrypt (or your CA)

# TLS grade
# Use https://www.ssllabs.com/ssltest/ or testssl.sh
testssl --quiet app.example.com | grep -E "Rating|Grade"
# Expected: A or A+

# External reachability from a second host
curl -fsS https://app.example.com/health

Resource controls

set -o pipefail
CIDS=$(docker compose ps -q)

# Memory and CPU limits on every container. A zero in either column is an
# unlimited container, which can take the host down on its own.
docker inspect $CIDS --format \
  '{{.Name}} mem={{.HostConfig.Memory}} cpu={{.HostConfig.NanoCpus}} pids={{.HostConfig.PidsLimit}}'

BAD=$(docker inspect $CIDS --format \
  '{{.Name}} mem={{.HostConfig.Memory}} cpu={{.HostConfig.NanoCpus}}' \
  | grep -E 'mem=0|cpu=0' || true)
[ -z "$BAD" ] && echo "OK: every container has memory and CPU limits" \
  || { echo "FAIL: unlimited containers:"; echo "$BAD"; }

# The limits must also fit the host. Sum of memory limits vs host RAM.
TOTAL=$(docker inspect $CIDS --format '{{.HostConfig.Memory}}' | paste -sd+ | bc)
HOSTRAM=$(( $(awk '/^MemTotal:/{print $2}' /proc/meminfo) * 1024 ))
echo "sum of limits: $(numfmt --to=iec "$TOTAL")  host RAM: $(numfmt --to=iec "$HOSTRAM")"
[ "$TOTAL" -lt "$HOSTRAM" ] && echo "OK: limits fit within host RAM" \
  || echo "FAIL: overcommitted - the host OOM-killer is your only limit"

# Nothing is currently being OOM-killed or CPU-throttled. These counters are
# per-cgroup and reset on restart, so read them as a trend, not a one-off.
for cid in $CIDS; do
  NAME=$(docker inspect --format '{{.Name}}' "$cid")
  PID=$(docker inspect --format '{{.State.Pid}}' "$cid")
  CG=$(awk -F: '{print $3}' "/proc/$PID/cgroup" | head -1)
  OOM=$(awk '/^oom_kill/{print $2}' "/sys/fs/cgroup$CG/memory.events" 2>/dev/null)
  THR=$(awk '/^nr_throttled/{print $2}' "/sys/fs/cgroup$CG/cpu.stat" 2>/dev/null)
  echo "$NAME oom_kill=$OOM nr_throttled=$THR"
done
# Expected: oom_kill=0 everywhere. A non-zero value on a container that has
# not restarted is a partial OOM - see "cgroups v2".

Observability

# Prometheus targets up
curl -s http://prometheus:9090/api/v1/targets | jq '.data.activeTargets[] | select(.health != "up")'
# Expected: empty

# Logs flowing to Loki
curl -s 'http://loki:3100/loki/api/v1/query?query={service="api"}' | jq '.data.result | length'
# Expected: > 0

# Traces emitted (sample one)
curl -s 'http://tempo:3200/api/search?tags=service.name=api&limit=1' | jq .
# Expected: at least one trace

Backup and restore

# Backup ran today
test -f /srv/backups/postgres-$(date +%F).dump && echo "OK: backup exists"

# Restore test (run quarterly; documented last successful date)
test "$(stat -c %y /var/log/restore-tests/last-success.log | cut -d' ' -f1)" && echo "OK: restore test recorded"

# RPO documented
grep -q "RPO:" runbooks/docker-runbook-disk-full.mdx && echo "OK: RPO documented"

Documentation

# Runbook for every critical scenario exists
ls docs/audits/runbooks/ 2>/dev/null || ls src/content/courses/docker/runbooks/ | wc -l
# Expected: >= 5

# Final assessment reachable
test -f dist/courses/docker/assessments/docker-final-assessment/index.html && echo "OK: assessment reachable"

Architectural boundaries

set -o pipefail

# Which network is each service actually on? Read it from the running
# containers, not from compose.yml - the file is intent, this is fact.
docker compose ps --format json | jq -s -r '.[] | "\(.Service)\t\(.Networks)"'

# The data plane must not be reachable from the edge network. Nothing except
# api and worker belongs on db-net.
BAD=$(docker compose ps --format json | jq -s -r '
  .[] | select(.Networks | test("db-net"))
      | select(.Service | test("^(api|worker|db)$") | not)
      | "\(.Service) is on db-net"')
[ -z "$BAD" ] && echo "OK: db-net membership is minimal" \
  || { echo "FAIL: unexpected db-net members:"; echo "$BAD"; }

# The observability plane must not be published to the internet. Prometheus
# and Grafana bind loopback only; the proxy fronts them if they need access.
docker compose ps --format json | jq -s -r '
  .[] | select(.Service | test("^(prometheus|grafana|loki)$"))
      | "\(.Service)\t\(.Publishers // [] | map("\(.URL):\(.PublishedPort)") | join(","))"'
# Expected: 127.0.0.1 in every published address, never 0.0.0.0

A published address of 0.0.0.0 on Prometheus or Grafana is the one line in this section that has caused real incidents: an unauthenticated metrics endpoint and a Grafana login page, both on the public internet, on a host whose firewall lists only 80 and 443 — because a published port is DNAT’d before the firewall’s INPUT chain ever sees it. The lesson “Docker networking primitives” has the mechanism.

Acceptance

The capstone is passed when:

Every command in the Definition of Done exits 0 with the expected output.
The final assessment is reachable and a test learner passes.
The DR drill (Lab 21) has been run successfully in the last 90 days.
The restore test has been run successfully in the last 90 days.