Container Discovery

A Docker host running Compose services has a property no VM estate has: the list of things worth scraping changes every deploy, and only one system already knows the truth at every instant — the Docker daemon. The containers have names, labels, networks and ports, all queryable through one API. docker_sd_configs turns that API into a target list: instead of telling Prometheus what exists, you teach it to ask Docker.

Done well, deploying a new service with the right labels is the entire monitoring onboarding process. Done carelessly, Prometheus scrapes every exposed port of every container, half of them time out, and the target list churns with every container restart. The difference is a filtering convention, and most of this lesson is that convention.

What docker_sd_configs is

docker_sd_configs discovers the running containers on one Docker host (or, with a TCP socket, several) by polling the Docker Engine API. Each refresh lists containers, follows their networks, and produces targets annotated with __meta_docker_* labels describing the container, its networks and its ports. Relabeling — lesson 05, used here in its standard recipes — decides which containers become scrape targets and on which address.

Two siblings exist and this lesson touches both only to place them: dockerswarm_sd_configs does the same job against a Swarm manager (roles tasks, services, nodes), and kubernetes_sd_configs is the full Kubernetes mechanism. This course targets Docker 28.x hosts running Compose, not Kubernetes; the concepts below — meta labels, label-based opt-in, relabel filtering — transfer directly if you meet kubernetes_sd_configs later.

Why a sysadmin cares

Containers break the assumptions behind every previous lesson in this part. Static lists fossilise on the first redeploy. File SD needs a generator that knows about containers. DNS does not know about them at all. Meanwhile the deploy tool already speaks to the Docker API, so discovery from the same API means deploying is registering — no second step to forget, no inventory to drift.

The operational risk is the inverse: discovery now tracks the container lifecycle, including its pathologies. Crash-looping containers flap targets; one-shot migration containers appear for ninety seconds; every exposed port of every container is a candidate scrape. Without an opt-in convention you monitor noise and pay for churn.

How it works

scrape job with docker_sd_configs
        |
        |  every refresh_interval (default 60s)
        v
Docker Engine API: list running containers (+ filters)
                   list networks
        |
        v
one target per container, per attached network,
per exposed TCP port:
  __address__ = <container-ip>:<private-port>
  (no exposed TCP port -> <container-ip>:<port from config, default 80>)
  (host networking     -> <host_networking_host>, NO port)
        |
        v
meta labels: container id / name / labels,
             network name / ip, port numbers
        |
        v
relabel_configs: keep the opted-in, rewrite the rest
        |
        v
scrape loops

The address rule deserves a second read. Docker SD guesses the scrape port from exposed ports, and a container that exposes both 8080 (application) and 9100 (metrics) produces a target for each — Prometheus cannot know which one serves /metrics. The opt-in convention below exists precisely to override that guess.

Configuring it

The job, with a label-based opt-in. Only containers carrying com.prometheus.scrape=true are scraped; com.prometheus.port and com.prometheus.path override the address and metrics path:

scrape_configs:
  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        # refresh_interval: 60s   # default; how often the API is polled
        filters:
          # Server-side pre-filter: only containers that opted in
          # are ever returned by the API call.
          - name: label
            values: ['com.prometheus.scrape=true']
    relabel_configs:
      # 1. Strip the leading slash for a clean instance label.
      - source_labels: [__meta_docker_container_name]
        regex: '/(.*)'
        target_label: instance
      # 2. If a port override exists, rebuild __address__ from it.
      #    Non-matching containers keep their discovered address.
      - source_labels: [__meta_docker_network_ip, __meta_docker_container_label_com_prometheus_port]
        separator: ':'
        regex: '(.+):(\d+)'
        target_label: __address__
        replacement: '${1}:${2}'
      # 3. If a path override exists, use it for __metrics_path__.
      - source_labels: [__meta_docker_container_label_com_prometheus_path]
        regex: '(.+)'
        target_label: __metrics_path__
        replacement: '${1}'
      # 4. Persist a couple of useful container facts as labels.
      - source_labels: [__meta_docker_container_label_com_example_team]
        target_label: team

The Compose side of the contract:

# docker-compose.yml (Docker 28.x, Compose v2)
services:
  payments:
    image: registry.example.com/payments:1.42
    networks: [app, monitoring]
    labels:
      com.prometheus.scrape: 'true'
      com.prometheus.port: '9100'
      com.prometheus.path: '/metrics'
      com.example.team: 'checkout'

And the network reachability, which is the piece everyone forgets — Prometheus scrapes container IPs, so it must share a network with them:

networks:
  app:
  monitoring:
    name: monitoring   # the prometheus container joins this too

With match_first_network at its default, the alphabetically first attached network supplies the IP — here app, which may not be where Prometheus lives. If containers sit on several networks, either name the shared one to sort first, or set match_first_network: false and select the network by name in relabeling.

Validating it

# READ-ONLY: config syntax, then a full discovery dry-run
promtool check config /etc/prometheus/prometheus.yml
promtool check service-discovery /etc/prometheus/prometheus.yml docker

The dry-run prints every discovered target with its full meta label set and its post-relabeling labels — inspect this before the config ever reaches the server. Live:

# READ-ONLY: is the Docker refresh succeeding?
curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode \
  'query=prometheus_sd_refresh_failures_total{mechanism="docker"}' \
  | jq '.data.result[].value[1]'

# READ-ONLY: which containers became targets, and are they up?
curl -s 'http://localhost:9090/api/v1/targets?state=active' \
  | jq -r '.data.activeTargets[] | select(.scrapePool=="docker")
      | [.labels.instance, .scrapeUrl, .health] | @tsv'

On the Docker host, verify the labels and networks the way the daemon sees them:

# READ-ONLY: the opt-in label set as the API reports it
docker ps --filter label=com.prometheus.scrape=true \
  --format '{{.Names}}\t{{.Ports}}'
docker inspect payments --format '{{json .NetworkSettings.Networks}}' | jq 'keys'

/service-discovery remains the master view: discovered labels on the left, final labels on the right, per container.

How it fails

1. The socket is unreachable or unreadable. Prometheus runs as a container without the socket mounted, or as a user outside the docker group. Every refresh errors, the previous targets persist, and new containers never appear. Symptom: prometheus_sd_refresh_failures_total{mechanism="docker"} climbing, targets frozen in time.
2. Prometheus cannot route to container IPs. It sits on a different Docker network (or on the host while containers use private bridge IPs). Discovery works perfectly; every scrape times out. Symptom: targets exist, all down, context deadline exceeded.
3. The alphabet picked the network. match_first_network chose the wrong attached network and the IP is unreachable from Prometheus. Symptom identical to failure 2, for one subset of containers — the multi-homed ones.
4. Port ambiguity. A container exposes an application port and a metrics port; the opt-in port label is missing, so both become targets and one 404s every scrape. Symptom: up is 0 for half the targets with server returned HTTP status 404 as the last error.
5. Host-networked containers scrape the wrong thing. Their discovered address is host_networking_host (default localhost) with no port, so the scrape defaults to port 80 — almost never the exporter. Symptom: connection refused or an unexpected 200 from whatever does listen there. These containers need the port-override label as a matter of course.
6. Churn. Crash-looping and one-shot containers appear and vanish between refreshes, flapping up, flooding the TSDB with short-lived series, and firing target-down alerts for things that were never services. Symptom: target count oscillates; series churn metrics climb. The fix is the filter, not faster refreshes.

Troubleshooting it

1. Is discovery itself alive? The refresh-failure metric and the Prometheus log answer this in one step. If refreshes fail, nothing downstream matters.
2. What did the daemon return? promtool check service-discovery or /service-discovery shows discovered labels per container. If the container is absent there, the problem is filters, labels, or the daemon — not relabeling.
3. What did relabeling produce? Compare discovered labels against final labels on the same page. A missing target with correct discovery means a keep dropped it; a wrong address means the override rules did not match.
4. Can Prometheus reach the address? From the Prometheus container or host, curl -sv http://<container-ip>:<port>/metrics. This single command separates “discovery wrong” from “network wrong.”
5. Is the container even running? Docker SD lists running containers only; docker ps -a shows the ones you will never discover.

Security implications

Read access to the Docker socket is, in effect, root on the host: the API that lists containers also runs them. Mounting /var/run/docker.sock into the Prometheus container therefore hands the monitoring stack the keys to every container on the host. The standard mitigation is a socket proxy (for example tecnativa/docker-socket-proxy) that exposes only the read-only CONTAINERS and NETWORKS endpoints, with Prometheus pointed at the proxy over TCP. Remote Docker hosts must use TLS (tls_config in the SD entry); an unauthenticated TCP socket on the network is an unauthenticated root shell.

The second boundary is labels: anyone who can start a container can influence discovery. A container labelled com.prometheus.scrape: 'true' with a port pointing at an internal service turns Prometheus into a fetch proxy whose results are stored as metrics. On a shared host, scope the job with filters (for example, a dedicated network or a project label) so opt-in alone is not sufficient.

Performance implications

Each refresh is one container list plus one network list — trivial against Docker 28.x even with hundreds of containers. The real cost is target churn: every container restart with a changed IP or name is a target deletion plus creation, writing staleness markers and starting new series. Fleets of short-lived containers can churn the head block far harder than their metrics justify. Mitigate by filtering to long-lived services (the opt-in convention does this naturally), keeping relabeled identities stable across restarts (container name, not ID, as instance), and resisting the urge to lower refresh_interval below the 60s default — discovery lag is rarely the outage you are having.

Production guidance

• Publish the label convention (scrape, port, path, one owner label) in the service template repository, so opting in is the default, not a favour.
• Run Prometheus (or Alloy) on a shared monitoring network that every scrapeable Compose project joins. One network, one routing story.
• Use filters to scope discovery to what the convention covers; treat unlabelled containers as out of scope, not as errors.
• Alert on prometheus_sd_refresh_failures_total{mechanism="docker"} and on the ratio up{job="docker"} falling — the first catches a dead daemon, the second catches dead networking.
• On Swarm, switch to dockerswarm_sd_configs with role tasks and the same label discipline; the meta labels change prefix (__meta_dockerswarm_*) but the recipes carry over.
• Rollback: discovery changes roll back like any config — revert, reload, verify on /service-discovery. Container-side label changes roll back on the next deploy; a bad keep rule fixed within the hour leaves only a gap, not data loss.

Verification

You should now be able to answer:

• How does docker_sd build __address__ for a container with two exposed TCP ports, with no exposed ports, and in host networking mode?
• What does the com.prometheus.port relabel rule actually rewrite, and why is the regex written to no-op on containers without the label?
• What does Prometheus do with its target list when the Docker daemon is unreachable?
• Why is mounting the Docker socket into the Prometheus container a privileged action, and what is the standard mitigation?
• Which metric tells you container discovery is failing, as opposed to the containers failing?

Quiz