Capstone stage 1 — the host under the environment

The capstone overview showed the finished environment: Caddy at the edge, web, api and worker on app-net, Postgres and Redis on db-net, and the observability stack on obs-net. That lesson stated what the environment is and how it is accepted.

These five stages build it. Each one ends in a gate — a short list of commands with expected output — and you do not proceed to the next stage until the gate passes. That discipline is the point: the reason production environments end up undocumented is that they are built in one long session and the intermediate states are never recorded.

Stage 1 is the host. Nothing in the stack is more reliable than the machine underneath it, and two of the four capstone failure modes we will drill in stage 5 are host-level.

Sizing, from the capacity model

The stack has eleven containers. Apply the memory model to each and sum the result rather than picking a host size and hoping:

sum of limits          = 128 + 256 + 512 + (2 x 256) + 2048 + 512
                       + 1024 + 512 + 512 + 512               = 6784 MB
host reserve           = 1 GiB + 1% of RAM
required RAM (at 70%)  = 6784 / 0.70                           = 9691 MB
                                                          -> 12 GB minimum,
                                                             16 GB comfortable

CPU, using the same approach: the quota sum should land near the core count on a single-host capstone, because there is no second host for the load to fail over to.

sum of quotas = 0.25 + 0.5 + 1.0 + (2 x 0.5) + 2.0 + 0.5
              + 1.0 + 0.5 + 0.5 + 0.5                        = 8.25 cores
                                                        -> 8 cores, ratio 1.03

Disk, from the four growth terms:

base images (11 services, much shared)                    ~  4 GB
image churn: 2 deploys/day x 80 MB x 14 days retention    =  2.2 GB
volumes: db 1.5 GB/month x 12 + prometheus 15d retention  = 24 GB
logs: 11 containers x 50 MB x 3 files                     =  1.7 GB
build cache ceiling                                       = 10 GB
                                                            -------
subtotal                                                    41.9 GB
with 20% headroom                                           52.4 GB
                                                        -> 64 GB for /var/lib/docker

Storage layout

lsblk -o NAME,SIZE,FSTYPE,MOUNTPOINT
findmnt -no SOURCE,FSTYPE,OPTIONS /var/lib/docker

NAME   SIZE FSTYPE MOUNTPOINT
sda     256G
├─sda1   1G vfat   /boot/efi
├─sda2  32G ext4   /
├─sda3  64G ext4   /var/lib/docker
└─sda4  32G ext4   /srv/backups
/dev/sda3 ext4 rw,relatime

Four filesystems, each with a job:

• / — the OS. Never grows unexpectedly once Docker is elsewhere.
• /var/lib/docker — images, volumes, logs, build cache. Sized above.
• /srv/backups — the local staging area for backups, which stage 5 copies off-host. Separate so that a runaway backup cannot fill the Docker filesystem.
• /boot/efi — as installed.

The daemon configuration

{
  "icc": false,
  "live-restore": true,
  "userland-proxy": false,
  "no-new-privileges": true,
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "50m",
    "max-file": "3",
    "labels": "com.docker.compose.service"
  },
  "default-ulimits": {
    "nofile": { "Name": "nofile", "Hard": 8192, "Soft": 4096 }
  },
  "metrics-addr": "127.0.0.1:9323"
}

Each line is a decision the capstone will be graded on:

• live-restore keeps the eleven containers running across a daemon restart, which stage 5 tests deliberately.
• userland-proxy: false removes one docker-proxy process per published port. The capstone publishes only 80 and 443, so this is a small win — take it anyway, and verify the hairpin case in stage 3.
• log-opts.labels attaches the Compose service name to every log entry, which is what makes the Loki labels in stage 4 work.
• metrics-addr on loopback exposes the daemon’s own Prometheus metrics for stage 4 to scrape, without exposing them to the network.
• no-new-privileges sets the default so that a service someone adds later without the flag still gets it.

sudo python3 -c 'import json; json.load(open("/etc/docker/daemon.json")); print("valid")'
sudo systemctl restart docker
systemctl is-active docker
docker info --format 'live-restore={{.LiveRestoreEnabled}} logging={{.LoggingDriver}}'

valid
active
live-restore=true logging=json-file

Host-level prerequisites

1. Time synchronisation. TLS certificate validation, Prometheus sample ordering and log correlation all depend on it. timedatectl show -p NTPSynchronized --value must be yes.
2. DNS for the two hostnames. app.example.com and api.example.com must resolve to this host from the internet before Caddy can obtain a certificate in stage 3.
3. Firewall. Inbound 80 and 443 from anywhere, 22 from the management range only. Everything else denied, and remember that published ports bypass the INPUT chain.
4. Unattended security updates enabled, with a reboot window that stage 5 assumes the stack survives.
5. **The docker group reviewed.** Its membership is a root grant; the capstone host should have the operators and the CI runner in it, and nothing else.
6. A non-root operator account with sudo, and SSH key authentication only.

echo '--- filesystem'
findmnt -T /var/lib/docker -no TARGET,SOURCE,SIZE,USE%
echo '--- daemon'
docker info --format 'live-restore={{.LiveRestoreEnabled}} logging={{.LoggingDriver}}'
docker info --format '{{json .SecurityOptions}}' | grep -q unconfined && echo 'FAIL seccomp' || echo 'OK seccomp'
echo "--- userland proxy processes: $(pgrep -c docker-proxy || echo 0)"
echo '--- time'
timedatectl show -p NTPSynchronized --value
echo '--- dns'
getent hosts app.example.com api.example.com
echo '--- access'
getent group docker

--- filesystem
/var/lib/docker /dev/sda3 64G 2%
--- daemon
live-restore=true logging=json-file
OK seccomp
--- userland proxy processes: 0
--- time
yes
--- dns
203.0.113.24    app.example.com
203.0.113.24    api.example.com
--- access
docker:x:988:ops,ci-runner

The findmnt -T form is important: it reports the filesystem containing the path, so a host where /var/lib/docker was never split reports / and /dev/sda2 instead of failing silently. A bare findmnt /var/lib/docker exits non-zero with no output when the path is not itself a mount point, which is exactly the case you are trying to detect.

docker-proxy process count is the evidence for userland-proxy: false — there is no docker info field for it. Zero processes while ports are published means the setting took effect; one per published port means it did not.

Sanity check