Docker Compose Installation

This course’s labs run on Compose, and so do a surprising number of real deployments: a single monitoring VM at a small site, an edge box, a proof of concept that quietly became permanent. Compose earns that place honestly — one declarative file, one command to converge, restart policy included. It also has sharp edges that the docker run tutorials glide over: data that vanishes with the container, host networking that does not mean what you think, and upgrades that happen to you instead of by you.

This lesson writes the service properly and names each edge.

What Compose gives you — and what it does not

docker compose (the v2 plugin, standard on Docker 28.x) reads compose.yaml, and creates or updates the containers, networks and volumes it declares. Resources are prefixed with the project name (the directory name, by default), so one host can run several stacks without collision. Compose gives you declarative configuration, dependency ordering, and a restart policy. It does not give you high availability, multi-host scheduling, secrets management, or upgrade orchestration. Keep that list in mind; the final section turns it into a decision.

The service, line by line

# compose.yaml
services:
  prometheus:
    image: prom/prometheus:v2.55.1        # pinned: upgrades are a decision
    container_name: prometheus
    command:                               # entrypoint is the binary; these are its flags
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=30d'
      - '--web.enable-lifecycle'
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./prometheus/rules:/etc/prometheus/rules:ro
      - prometheus_data:/prometheus        # the TSDB survives container replacement
    ports:
      - '127.0.0.1:9090:9090'              # localhost only; the API has no auth by default
    restart: unless-stopped
    healthcheck:
      test: ['CMD', 'wget', '--spider', '-q', 'http://localhost:9090/-/healthy']
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 60s

volumes:
  prometheus_data:

Lines with non-obvious consequences:

• command — the image entrypoint is the prometheus binary, so every list item is a server flag. Flags here behave exactly like ExecStart flags in lesson 03: changing them requires recreating the container, and a reload will not apply them.
• prometheus_data:/prometheus — the single most important line. Without it the TSDB lives in the container’s writable layer: slower, and deleted with the container.
• 127.0.0.1:9090:9090 — publishing as 9090:9090 binds every interface, exposing the unauthenticated API to the LAN. Bind localhost and front it with a reverse proxy, or publish on a management interface deliberately.
• healthcheck against /-/healthy, not /-/ready — readiness returns 503 during WAL replay after every restart. A ready-based check marks the container unhealthy during a slow replay, and any automation that restarts unhealthy containers turns one restart into a loop. Health answers “is the process serving”; that is what a supervisor should watch.

Networking: reaching the host from the container

The default bridge network gives each container its own interface and DNS by service name. Scraping other containers is trivial — targets: ['grafana:3000'] resolves. Scraping the host is the interesting case: localhost inside the container is the container.

Two supported answers on Linux with Docker 28.x:

    # Option A: stay on the bridge, name the host explicitly
    extra_hosts:
      - 'host.docker.internal:host-gateway'

host-gateway resolves to the host’s address on the bridge. Target host.docker.internal:9100 for a node_exporter on the host — provided the host firewall lets the bridge subnet reach it, which is the part everyone forgets.

    # Option B: share the host network namespace
    network_mode: host

With host networking, localhost:9100 from Prometheus reaches the host’s node_exporter, and ports: becomes meaningless (the container binds the host’s interfaces directly). The trade-offs: you lose service-name DNS to sibling containers, port collisions with host services become possible, and the unauthenticated API is now on whatever interfaces the host has. Option A isolates better; option B is simpler on a dedicated monitoring host. Pick per deployment, and write the choice down.

Operating the stack

# READ-ONLY -- render and validate the compose file before applying
docker compose config --quiet && echo config-ok

# CONFIGURATION -- converge: create or recreate what changed
docker compose up -d

# READ-ONLY -- state, including health
docker compose ps
# NAME         STATUS                    PORTS
# prometheus   Up 2 hours (healthy)      127.0.0.1:9090->9090/tcp

# READ-ONLY -- daemon logs through the container runtime
docker compose logs --tail 20 prometheus

# READ-ONLY -- validate config inside the running container
docker compose exec prometheus \
  promtool check config /etc/prometheus/prometheus.yml

# CONFIGURATION -- reload without recreating (config/rules only)
curl -X POST http://localhost:9090/-/reload
curl -s http://localhost:9090/-/ready

Editing compose.yaml does nothing to a running container until docker compose up -d converges it — and convergence recreates the container when the definition changed, which is exactly why the named volume matters.

Backup and restore

The TSDB lives in the named volume. A consistent backup needs the same discipline as on bare metal — snapshot or stop first:

# SERVICE-IMPACT -- stop, archive the volume, start
docker compose stop prometheus
sudo tar -C "/var/lib/docker/volumes/$(docker volume ls -q | grep prometheus_data)/_data" \
  -czf "prometheus-data-$(date +%F).tar.gz" .
docker compose start prometheus

Restore is the reverse: create the volume, unpack into it, up -d. Test the restore once before you need it; an untested backup is a hypothesis.

How it can fail

1. No volume on /prometheus. Data in the container layer: lost on the next recreate, and written through the overlay filesystem meanwhile. Symptom: history resets after every upgrade; docker volume ls shows nothing.
2. Root-owned bind mount for data. The nobody process cannot write; the container restart-loops with opening storage failed: ... permission denied. Fix: chown -R 65534:65534 the host directory, or use a named volume.
3. Unpinned tag. docker compose pull moves latest to a new major; the next up -d runs it against a config written for the old one.
4. Port collision. Another service already holds 9090; the container fails to start with bind: address already in use from the docker-proxy layer.
5. Host firewall vs host-gateway. The container reaches host.docker.internal:9100 only if nftables lets the bridge in. Symptom: the target is down in /targets with connection refused while curl from the host works fine.
6. Daemon maintenance. Upgrading or restarting dockerd stops every container unless live-restore is enabled in /etc/docker/daemon.json. Monitoring gaps that coincide exactly with Docker maintenance windows are this.
7. Reload expected to apply flags. /-/reload re-reads prometheus.yml only; a changed command: needs up -d to recreate. Symptom: the reload succeeds and the new retention never appears in /api/v1/status/flags.

How to troubleshoot it

1. docker compose ps — is the container up, restarting, or unhealthy? docker inspect -f '\{\{.State.Health.Status\}\}' prometheus for the health verdict and its log.
2. docker compose logs --tail 100 prometheus — the startup error is in there: permission, flag parse, lock.
3. docker compose exec prometheus wget -qO- http://localhost:9090/-/ready — proves the server inside the container independent of port publishing, firewall, and proxies.
4. docker volume inspect <project>_prometheus_data — confirm the TSDB is really on the volume, and find the host path for backup or forensics.
5. docker events --since 1h — restarts, OOM kills, and recreates in order; the fastest way to see a flap loop.

Compose or systemd?

Compose is a good home for Prometheus when: one host, a small team, no compliance-driven change control, and a tested volume backup. It is the wrong home when you need HA pairs on separate machines, when the estate is Kubernetes (use the operator and forget this file), or when the monitoring host must be provably rebuildable from configuration management — that points at the tarball-plus-systemd install of lessons 01–03, which is this course’s default for production VMs. The trade-off is real: Compose is faster to stand up and easier to read; systemd on a dedicated host has fewer moving parts between you and the process and does not share fate with a container runtime’s upgrade cycle.

Security implications

The image already runs as an unprivileged user — keep it that way and never add user: root to silence a permission problem. Bind published ports to localhost or a management interface; the API has no authentication by default, and 9090:9090 on all interfaces is an open door. Mount config read-only (:ro), as in the example. Never mount the Docker socket into the Prometheus container — it is root-equivalent on the host. If the config carries credentials, the host-side file should be 0640 and its directory 0750; remember the in-container reader is uid 65534 when you choose ownership.

Performance implications

A named volume is plain host storage under /var/lib/docker, so the TSDB performs as the underlying filesystem performs — size and place that filesystem deliberately. Writing data into the container layer instead forces every block through overlayfs: a real and pointless penalty. Bridge networking adds NAT traversal per connection; at Prometheus scrape rates it is noise, and host networking removes it where it matters. The healthcheck costs nothing worth measuring.

Verification

You should now be able to answer:

• Why does the TSDB need a named volume, and what two failures does it prevent?
• Why does the healthcheck target /-/healthy and not /-/ready?
• What are the two supported ways to scrape a host-level exporter from a container on Linux, and what does each cost?
• What does docker compose down -v do that down does not?
• When is Compose the wrong tool for a production Prometheus?

Quiz