Lab: Secure the Observability Stack

Objective

By the end of this lab you will have taken a working observability stack apart from the outside, using nothing but curl and the ports it publishes, and then closed it in the order that matters.

The order is the lesson. Most hardening guides start with TLS, because TLS is the part that looks like security. In this stack the first finding is that an unauthenticated HTTP POST deletes metrics, the second is that the scrape configuration is readable by anyone who can reach the port, and the cheapest fix for both is a line removed from a Compose file. You will do the certificates too — but after you have measured how much of the problem was reachability.

Architecture

One host, two containers, and a deliberately careless starting configuration.

   you, on the host                     the compose network
        |                          +--------------------------+
        |  :9090 published         |                          |
        +------------------------> | prometheus 2.55          |
        |                          |  --web.enable-admin-api  |
        |                          |  --web.enable-lifecycle  |
        |                          +------------+-------------+
        |                          |            | scrape
        |  :9100 published         |            v
        +------------------------> | node_exporter 1.8        |
                                   |  plain HTTP, no auth     |
                                   |  textfile collector      |
                                   +--------------------------+

Publishing both ports to the host is what puts you in the attacker’s position without needing a second machine: anything you can do from the host shell, a process on any host that can route to this one can do too. That is the whole point of the first half of the lab, and the reason the second half starts by taking one of those ports away.

Requirements

• A Linux host with Docker Engine 28.x and Docker Compose v2.
• curl, jq and openssl on the host. openssl is used for the lab CA; everything else runs in a container.
• Free TCP ports 9090 and 9100.
• About 1 GiB of free disk for two images.
• 90 minutes. No step here takes long; the deliberate failure in Task 3 is worth the five minutes it costs.
• Nothing outside the lab directory and the Compose project is modified. The certificates are self-signed and never leave the directory.

Scenario

The stack was stood up in an afternoon during an incident, when nobody had time for certificates. It worked, it was useful, and it is still running eight months later. Nothing about it looks wrong: the dashboards are good, the alerts fire, the targets are green.

Then someone asks the question that starts every security review of a monitoring platform: who else can reach this? The honest answer is unknown, because nobody wrote down what was exposed. Your job is to produce that list first, with evidence, and only then to start fixing things — because a hardening plan written before the exposure is measured usually protects the wrong boundary.

Tasks

Task 1: Stand up the stack the way it ships

LABDIR="$HOME/rb-obs-security"
mkdir -p "$LABDIR"/textfile
cd "$LABDIR"

A textfile collector drop, so that the exporter is publishing something an application team added rather than only kernel counters. This one is real in the worst way: a build-info metric whose labels carry a database connection string.

cat > textfile/app.prom <<'EOF'
# HELP app_build_info Build and runtime metadata for the checkout service.
# TYPE app_build_info gauge
app_build_info{version="1.4.2",commit="9f2c1a",db_dsn="postgres://app:hunter2@db.example.com:5432/app"} 1
EOF

prometheus.yml — two jobs, no credentials anywhere:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']

  - job_name: node
    static_configs:
      - targets: ['exporter:9100']

compose.yaml:

name: rb-obs-security

services:
  prometheus:
    image: prom/prometheus:v2.55.1
    container_name: rb-sec-prom
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.enable-admin-api'
      - '--web.enable-lifecycle'
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./certs:/etc/prometheus/certs:ro
      - ./secrets:/etc/prometheus/secrets:ro
      - prom-data:/prometheus
    ports:
      - '9090:9090'

  exporter:
    image: prom/node-exporter:v1.8.2
    container_name: rb-sec-exporter
    command:
      - '--collector.textfile.directory=/textfile'
    volumes:
      - ./textfile:/textfile:ro
      - ./certs:/etc/exporter/certs:ro
      - ./web-config.yml:/etc/exporter/web-config.yml:ro
    ports:
      - '9100:9100'

volumes:
  prom-data:

The certs, secrets and web-config.yml mounts are declared now and filled in Task 3, so that later tasks are one restart rather than a Compose edit. Create the placeholders and start:

mkdir -p certs secrets
touch web-config.yml
docker compose up -d
sleep 20
curl -s http://localhost:9090/api/v1/targets | jq -r '.data.activeTargets[] | "\(.labels.job) \(.health)"'

Both jobs report up. This is the state the scenario describes: working, useful, and unmeasured.

Task 2: Take the outsider’s position and write down what you find

Everything in this task is an unauthenticated request from the host shell. Keep a findings file as you go; it is a deliverable, and it is the document that makes the fixes arguable later.

Finding one — the exporter identifies the host. No credentials, no TLS, one GET:

$ curl -s http://localhost:9100/metrics | grep '^node_uname_info'

node_uname_info{domainname="(none)",machine="x86_64",nodename="rb-sec-exporter",release="6.8.0-51-generic",sysname="Linux",version="#52-Ubuntu SMP"} 1

Host identity and kernel version in one line. Add the filesystem and network collectors and an anonymous caller has your mount layout and your interface list — which is not a breach on its own, and is an excellent start on one.

Finding two — the scrape configuration is public. Prometheus serves its own configuration to anyone who asks:

curl -s http://localhost:9090/api/v1/status/config | jq -r '.data.yaml'
curl -s http://localhost:9090/api/v1/status/flags | jq '{"web.enable-admin-api": ."web.enable-admin-api", "web.enable-lifecycle": ."web.enable-lifecycle"}'

Right now that configuration holds no credentials, so it costs you a target list and a topology map. After Task 3 it will hold a credential reference, which is exactly why the endpoint matters more than it looks.

Finding three — an anonymous POST deletes data. Confirm the series exists, delete it, and confirm it is gone:

$ curl -s -X POST 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match%5B%5D=node_uname_info' -o /dev/null -w 'delete: %{http_code}\n'

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=node_uname_info' | jq '.data.result | length'

Zero. The exporter is still running and still publishing the metric, so the series comes back on the next scrape — which is the good case. A delete against a series nobody is currently scraping, or issued with a time range covering last quarter, is not recoverable from anywhere except a backup.

Your findings file should now have four lines: host identity is public, the configuration is public, the flags are public, and an unauthenticated caller can delete and snapshot. Note what none of those needed: a vulnerability, a stolen credential, or anything cleverer than curl.

Task 3: Authenticate and encrypt the scrape path

Issue a lab CA and a certificate for the exporter. The subjectAltName is not optional — Go has not honoured the Common Name field for hostname verification for years, and a CN-only certificate fails the scrape with a message about the legacy field:

cd "$LABDIR"/certs

openssl req -x509 -newkey rsa:2048 -sha256 -days 30 -nodes \
  -keyout ca.key -out ca.crt -subj "/CN=rb-obs-lab-ca"

openssl req -newkey rsa:2048 -nodes \
  -keyout exporter.key -out exporter.csr -subj "/CN=exporter"

printf 'subjectAltName=DNS:exporter\nextendedKeyUsage=serverAuth\n' > exporter.ext

openssl x509 -req -in exporter.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out exporter.crt -days 30 -sha256 -extfile exporter.ext

openssl x509 -in exporter.crt -noout -text | grep -A1 'Subject Alternative Name'
cd "$LABDIR"

DNS:exporter matches the Compose service name, which is the name Prometheus connects to. A certificate for localhost would be correct for your curl and wrong for the scrape.

chmod 0644 certs/exporter.key certs/exporter.crt certs/ca.crt

Now the credential. Generate a bcrypt hash without installing anything:

PASSWORD='lab-exporter-password'
HASH=$(docker run --rm httpd:2.4-alpine htpasswd -nbB prometheus "$PASSWORD" | cut -d: -f2)

printf 'tls_server_config:\n  cert_file: /etc/exporter/certs/exporter.crt\n  key_file: /etc/exporter/certs/exporter.key\nbasic_auth_users:\n  prometheus: %s\n' "$HASH" > web-config.yml

printf '%s' "$PASSWORD" > secrets/exporter.pass
chmod 0644 secrets/exporter.pass
cat web-config.yml

Validate the web configuration before anything restarts. Mount the lab directory at the path the exporter will see, so the certificate paths inside the file resolve:

$ docker run --rm -v "$PWD:/etc/exporter" -w /etc/exporter --entrypoint /bin/promtool prom/prometheus:v2.55.1 check web-config web-config.yml

web-config.yml SUCCESS

Point the exporter at it by adding one line to its command: list in compose.yaml:

    command:
      - '--collector.textfile.directory=/textfile'
      - '--web.config.file=/etc/exporter/web-config.yml'

And rewrite the node job in prometheus.yml so the scrape speaks HTTPS and presents the credential:

  - job_name: node
    scheme: https
    tls_config:
      ca_file: /etc/prometheus/certs/ca.crt
      server_name: exporter
    basic_auth:
      username: prometheus
      password_file: /etc/prometheus/secrets/exporter.pass
    static_configs:
      - targets: ['exporter:9100']

Apply both edits and bring the stack back up:

docker compose up -d
sleep 25
curl -s http://localhost:9090/api/v1/targets \
  | jq -r '.data.activeTargets[] | "\(.labels.job) \(.health) \(.lastError)"'

The node job should be up with an empty lastError. Now check the outsider’s position again — the same request that worked in Task 2:

curl -s -o /dev/null -w 'plain http: %{http_code}\n' http://localhost:9100/metrics
curl -sk -o /dev/null -w 'https, no credential: %{http_code}\n' https://localhost:9100/metrics
curl -s -o /dev/null https://localhost:9100/metrics \
  && echo 'https, no CA: connected' || echo 'https, no CA: certificate verification failed'
curl -s --cacert certs/ca.crt --resolve exporter:9100:127.0.0.1 \
  -u "prometheus:$PASSWORD" -o /dev/null -w 'https, with CA and credential: %{http_code}\n' \
  https://exporter:9100/metrics

Four different answers to the same URL, and each one is a different control doing its job: the plain HTTP request no longer speaks the right protocol, the credential-less HTTPS request is rejected with 401, the request without the CA fails verification before any HTTP happens, and the fully equipped request gets 200. The --resolve is there because the certificate is issued for the name exporter, and asking for localhost fails hostname verification exactly as it should.

Task 4: Take away the reachability

Two edits to the Compose file are worth more than everything in Task 3.

First, drop the admin API. Remove --web.enable-admin-api from the Prometheus command: list, leave --web.enable-lifecycle for now, and restart:

docker compose up -d
sleep 15
curl -s -X POST 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match%5B%5D=node_uname_info' \
  -w '\nadmin delete: %{http_code}\n'

The response says the admin APIs are disabled, and the status code is a refusal rather than a 204. Confirm the series survived:

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=node_uname_info' | jq '.data.result | length'

Second, unpublish the exporter. Delete the ports: block from the exporter service in compose.yaml entirely, then:

docker compose up -d
sleep 25
curl -s -m 5 -o /dev/null -w 'host to exporter: %{http_code}\n' https://localhost:9100/metrics \
  || echo 'host to exporter: no route to the port'
curl -s http://localhost:9090/api/v1/targets \
  | jq -r '.data.activeTargets[] | select(.labels.job=="node") | "\(.health) \(.lastScrape)"'

The host can no longer reach the exporter at all. Prometheus, which is on the Compose network, is still scraping it every fifteen seconds. Nothing about the exporter changed — the credential and the certificate are still there, and they are now a second layer behind a boundary that no longer admits the caller in the first place.

Task 5: The secret authentication cannot help with

Everything so far has been about the transport. Look at what the transport is carrying:

$ curl -sG http://localhost:9090/api/v1/query --data-urlencode 'query=app_build_info' | jq -r '.data.result[].metric.db_dsn'

postgres://app:hunter2@db.example.com:5432/app

A database password, with its username, host and database name, sitting in a metric label. It arrived over a mutually verified TLS connection with a bcrypt-hashed credential, and it is now in the TSDB, in every backup of the TSDB, in the federation feed if there is one, and in any dashboard that renders that metric.

The scrape-side control is metric_relabel_configs, which runs after the response is parsed and before the samples are stored. Add it to the node job:

    metric_relabel_configs:
      - regex: 'db_dsn'
        action: labeldrop

labeldrop matches label names, not values, and drops every matching label from every metric in this scrape. Reload and watch:

curl -s -X POST http://localhost:9090/-/reload -w 'reload: %{http_code}\n'
sleep 30
curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=app_build_info' | jq -r '.data.result[].metric'

New samples have version and commit and no db_dsn. Now ask for the samples from five minutes ago:

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=app_build_info' \
  --data-urlencode "time=$(( $(date -u +%s) - 300 ))" | jq -r '.data.result[].metric'

The old series still carries the secret, and will until retention expires or somebody deletes it deliberately — which needs the admin API you just turned off, and is one of the few legitimate reasons to turn it back on briefly.

Validation

Each of these is checkable, and each fails visibly if a step was skipped.

1. In Task 2, an unauthenticated curl returned node_uname_info, the full scrape configuration, and a successful delete.
2. promtool check web-config web-config.yml succeeded before the exporter was restarted with it.
3. openssl x509 -in certs/exporter.crt -noout -text shows DNS:exporter under Subject Alternative Name.
4. After Task 3, the node target is up with an empty lastError, and an anonymous HTTPS request to the exporter returns 401.
5. A request using --cacert and -u returns 200, and the same request without --cacert fails certificate verification.
6. chmod 0600 secrets/exporter.pass puts the target into a failing state naming a permission error, and 0644 restores it.
7. After Task 4, the admin delete is refused and the series survives; the host cannot reach port 9100 while Prometheus still scrapes it.
8. After Task 5, a current query for app_build_info returns no db_dsn label, and a query five minutes in the past still does.

Expected Outcome

rb-obs-security/
├── certs/
│   ├── ca.crt
│   ├── ca.key
│   ├── exporter.crt
│   ├── exporter.ext
│   └── exporter.key
├── compose.yaml
├── prometheus.yml
├── secrets/
│   └── exporter.pass
├── textfile/
│   └── app.prom
└── web-config.yml

A stack where the exporter is unreachable from outside its network, requires TLS and a password from inside it, and no longer stores a credential it was being handed. A findings note that says what each of those changes was worth, in the order they were made.

Troubleshooting

The node target reports x509: certificate relies on legacy Common Name field. The certificate has no subjectAltName. Re-issue with the exporter.ext file; the extension is what Go verifies.

x509: certificate signed by unknown authority. The scrape job’s ca_file does not point at the CA that signed the exporter certificate, or certs/ is not mounted into the Prometheus container.

x509: certificate is valid for exporter, not localhost. Expected from the host. Use --resolve exporter:9100:127.0.0.1 so curl asks for the name on the certificate.

The target reports a permission error opening the password file. The container runs as uid 65534 and cannot read a file owned by you with mode 0600. This is the deliberate failure in Task 3; chmod 0644 restores it for the lab.

The exporter container restarts in a loop after Task 3. Run docker compose logs exporter | tail -20. A malformed web-config.yml or an unreadable key file is the usual cause — the same promtool check web-config that passed on the host may not have seen the same file permissions the container sees.

/-/reload returns 403. --web.enable-lifecycle was removed along with --web.enable-admin-api. Restart with it, or reload by restarting the container.

app_build_info is missing entirely. The textfile collector reads files ending in .prom from its directory. Check the mount with docker compose exec exporter ls /textfile, and note that a syntax error in the file makes the collector report a failure rather than partial metrics.

Cleanup

The lab created two containers, one named volume, one Compose network, one directory, and a private key.

Step 1. Stop the stack and remove the volume:

$ cd "$HOME/rb-obs-security" && docker compose down -v

Step 2. Confirm nothing is listening on the lab ports:

docker compose ps
ss -ltnp 2>/dev/null | grep -E ':(9090|9100)\b' || echo 'ports free'

$ mkdir -p "$HOME/obs-lab-deliverables/telemetry-security" && cp -a "$HOME/rb-obs-security/compose.yaml" "$HOME/rb-obs-security/prometheus.yml" "$HOME/rb-obs-security/web-config.yml" "$HOME/obs-lab-deliverables/telemetry-security/" && rm -rf "$HOME/rb-obs-security"

Note that the copied web-config.yml still contains a bcrypt hash of the lab password. It is a hash of a password you published in a lab and should never be reused, which is the only reason it is safe to keep.

Step 3. The two images stay in the local cache. Remove them if you are finished:

docker image rm prom/prometheus:v2.55.1 prom/node-exporter:v1.8.2 httpd:2.4-alpine

Production notes

Mapping this exercise onto a real estate:

• Write the exposure list before the hardening plan. The four findings in Task 2 took ten minutes and they are what makes the argument for the work. A plan without them protects whatever the author read about most recently.
• Bind addresses and network policy come first. Every component in this course defaults to 0.0.0.0. The production bind is the monitoring interface, and the rest is firewall — a control that does not expire, cannot be misconfigured into insecure_skip_verify, and costs nothing to evaluate.
• --web.enable-admin-api and --web.enable-lifecycle are off by default for a reason. Leave them off and reload with SIGHUP from the supervisor. If a workflow genuinely needs a remote reload, scope it by bind address and put authentication in front of it.
• Credentials go in files, never in the config. password_file and bearer_token_file keep the secret out of Git, out of /api/v1/status/config, and out of /proc. The redaction marker in the config API is a display convenience, not a control.
• Certificate expiry is an outage with a date on it. These are thirty-day certificates; a real one is a renewal process with a monitored expiry metric, because a silent expiry takes every scrape down at once.
• A secret in telemetry is an incident, not a cleanup. Once a credential has been scraped it is in the TSDB, its backups, and any federation downstream. Rotate the credential first; the relabel rule only stops the next sample.

What You Learned

• The default stack answers to anyone who can route to it. Host identity, kernel version, scrape configuration, feature flags and a working delete, all without a credential.
• The admin API turns read access into write access. One unauthenticated POST removed a series, and the same flag exposes a snapshot of the entire TSDB.
• Go verifies the subjectAltName, not the Common Name. A certificate that looks right and has no SAN fails the scrape with a message most people meet for the first time during an outage.
• Removing a published port outperformed every certificate in the lab. Reachability is the cheapest boundary and the one a default quietly decides for you.
• A security fix can break collection. The 0600 password file is the ordinary shape of that failure: correct on paper, unreadable by the process that needs it, and only visible in lastError.
• Authentication says nothing about the payload. A database password travelled the hardened path intact, and the relabel rule that dropped it protected only the samples that had not been taken yet.