Admin API Risks

A small team runs Prometheus in a Kubernetes cluster. They set --web.enable-lifecycle so they can reload the config without a pod restart, and they set --web.enable-admin-api so they can query snapshot metadata from an internal tool. The Prometheus Service is a ClusterIP, but a misconfigured NetworkPolicy lets the management subnet reach it. An attacker on that subnet discovers the Prometheus UI through the internal DNS, sends a POST /api/v1/admin/tsdb/snapshot to create a snapshot of the local TSDB, downloads the resulting archive, and exfiltrates the contents. The scrape configs in the archive contain every credential the team uses for federation. The Prometheus was not breached in the sense of “code execution”. It was breached in the sense of “everything the Prometheus knows left the cluster”.

The admin API and the lifecycle endpoints are the parts of Prometheus that change state and read state. They are also the parts an attacker reaches first, because they are the most useful. This lesson is about hardening them.

What it is

Prometheus 2.55.x exposes four endpoint families on the same HTTP listener. They share the bind address, the TLS configuration, and the authentication middleware. They differ in what they do:

• API endpoints (/api/v1/query, /api/v1/query_range, /api/v1/targets, /api/v1/series, /api/v1/status/config, and the rest of /api/v1/*). Read-only queries. No state change. On by default.
• Admin API endpoints (/api/v1/admin/tsdb/snapshot, /api/v1/admin/tsdb/clean_series, and the rest of /api/v1/admin/*). Read and write state on the local TSDB. Off by default; enabled with --web.enable-admin-api.
• Lifecycle endpoints (PUT /-/reload, PUT /-/quit). Reload the scrape config from disk and stop the process cleanly. Off by default; enabled with --web.enable-lifecycle.
• UI endpoints (/, /graph, /alerts, /targets). The embedded React UI. Read-only. On by default.

   caller          Prometheus 2.55.x HTTP listener (--web.listen-address)
     |                    |
     |--GET  /api/v1/query_range---->[router]----[query engine]-->response
     |--GET  /api/v1/status/config--->[router]----[config dump]-->response
     |--GET  /api/v1/admin/tsdb/.....->[router]----[403 admin API disabled]
     |--POST /api/v1/admin/tsdb/.....->[router]----[admin handler]---->snapshot
     |--PUT  /-/reload---------------->[router]----[lifecycle?]-->[reload]
     |--PUT  /-/quit------------------>[router]----[lifecycle?]-->[exit 0]
     |--GET  /------------------------>[router]----[static]---->React UI

The mental model: the router checks the request path and either serves it (when the corresponding feature is enabled) or returns 403 Forbidden (when it is not). Authentication, when configured, applies before the router runs.

Why a sysadmin cares

The admin API and lifecycle endpoints change the threat model for Prometheus. Without them, the worst a network-adjacent attacker can do is read metrics (which are usually less sensitive than the configuration). With them, the attacker can:

• Snapshot the TSDB. POST /api/v1/admin/tsdb/snapshot creates a consistent copy of the local TSDB on disk and returns a path. The copy is downloadable from the same host’s filesystem. If the snapshot can be exfiltrated, it contains every scrape configuration, every metric, and every label.
• Clean series. POST /api/v1/admin/tsdb/clean_series deletes series from the TSDB. Useful for cleaning up after a deletion; dangerous if invoked against the wrong series selector.
• Reload the configuration. PUT /-/reload re-reads prometheus.yml from disk. If the attacker can write to that file (a separate compromise), they can swap the scrape config and reload in one step.
• Stop Prometheus. PUT /-/quit causes the process to exit cleanly. A kill from the network rather than from the kernel. The blast radius is “every alert goes stale, every dashboard blanks, every remote-write queue backs up”.

The right answer in production is to leave both flags disabled and reload via SIGHUP from the local process supervisor (systemd, the kubelet’s liveness probe handler, an in-cluster operator). When a remote reload workflow is genuinely required, scope it to a CIDR via the bind address (lesson 01) and protect the route with basic auth (lesson 02).

How it works

The admin API

The admin API endpoints live under /api/v1/admin/*. The full list in Prometheus 2.55.x:

All three require --web.enable-admin-api to be set. Without the flag, the router returns 403 Forbidden with a body explaining the flag.

The tsdb/snapshot endpoint creates a hard-link copy of the TSDB under --storage.tsdb.path/snapshots/. The path is returned in the JSON response. The copy is not encrypted; it is the on-disk representation of every series Prometheus has ingested.

The tsdb/clean_series and tsdb/delete_series endpoints accept a match[] selector and (for delete) a start/end time range. They operate on the local TSDB and emit no audit log beyond Prometheus’s own request log.

The lifecycle endpoints

The lifecycle endpoints live at /-/reload and /-/quit:

• PUT /-/reload re-reads prometheus.yml and any referenced rule files. If the new config is valid, it is applied. If the new config is invalid, the existing config is left in place and the response is 400 Bad Request with the parse error.
• PUT /-/quit causes the Prometheus process to exit cleanly with code 0. systemd restarts it (because of Restart=always); the kubelet restarts the pod. The blast radius is “metrics are unavailable for the duration of the restart”.

Both require --web.enable-lifecycle. Without the flag, the router returns 403 Forbidden.

The reload endpoint is a convenience for the operator. The same effect is achieved with SIGHUP from the local shell. The reason the HTTP endpoint exists is so that an automation tool (Ansible, an in-cluster operator, a CI job) can reload without shell access to the Prometheus host.

How to configure it

Default posture: both flags off

# /etc/default/prometheus
ARGS="--config.file=/etc/prometheus/prometheus.yml \
      --storage.tsdb.path=/var/lib/prometheus \
      --web.listen-address=127.0.0.1:9090 \
      --web.external-url=https://prometheus.example.com"

No admin API. No lifecycle endpoints. Configuration reloads are sent via SIGHUP from the local shell (or the systemd unit, or the kubelet). The bind address is loopback, so even if the flags were set, only the local host could reach the endpoints.

Lifecycle enabled for automation, admin API off

When an automation tool needs to reload the configuration without shell access.

ARGS="--config.file=/etc/prometheus/prometheus.yml \
      --storage.tsdb.path=/var/lib/prometheus \
      --web.listen-address=127.0.0.1:9090 \
      --web.enable-lifecycle \
      --web.config.file=/etc/prometheus/web.yml \
      --web.external-url=https://prometheus.example.com"

# /etc/prometheus/web.yml
basic_auth_users:
  automation: $2y$10$bcrypt-hash-here

The automation tool uses curl -u automation:$TOKEN -X PUT https://prometheus.example.com/-/reload. The bind address keeps the endpoint off the LAN; the reverse proxy enforces authentication for any external caller; the basic auth map in the web config keeps the endpoint off the path of an unauthenticated operator.

Admin API enabled for an internal tool

When an internal tool genuinely needs to snapshot the TSDB (for example, to ship a backup to object storage) and there is no alternative path. Rare.

ARGS="--config.file=/etc/prometheus/prometheus.yml \
      --storage.tsdb.path=/var/lib/prometheus \
      --web.listen-address=127.0.0.1:9090 \
      --web.enable-admin-api \
      --web.config.file=/etc/prometheus/web.yml"

Bind to loopback. Configure basic_auth_users in web.yml. Restrict the proxy to the automation subnet. Document the flag in the team’s runbook.

Disable --web.enable-admin-api and --web.enable-lifecycle in CI

The CI pipeline should refuse to deploy a Prometheus manifest that contains either flag, except for the narrow exception paths documented in the runbook.

# CI guard rail: refuse the flag in production manifests.
grep -E '\-\-web\.enable-(admin-api|lifecycle)' \
  manifests/prometheus/production.yaml \
  && { echo 'admin/lifecycle enabled in prod'; exit 1; }

How to validate it

# READ-ONLY: confirm the admin API is disabled (default).
curl -s -o /dev/null -w '%{http_code}\n' \
  -X POST http://127.0.0.1:9090/api/v1/admin/tsdb/snapshot
# 403

# READ-ONLY: confirm the lifecycle endpoints are disabled (default).
curl -s -o /dev/null -w '%{http_code}\n' \
  -X PUT http://127.0.0.1:9090/-/reload
# 403

# READ-ONLY: confirm the same is true from an external peer (the proxy).
curl -s -o /dev/null -w '%{http_code}\n' \
  -X PUT https://prometheus.example.com/-/reload
# 403

# READ-ONLY: when the flags are enabled, confirm the endpoint exists.
curl -s -X PUT http://127.0.0.1:9090/-/reload
# (no body; reload happens)

# READ-ONLY: confirm the config reflects the reload.
curl -fsS http://127.0.0.1:9090/api/v1/status/config | jq '.data.yaml | length'
# 4237

# READ-ONLY: confirm a snapshot can be created (admin API on).
curl -fsS -X POST http://127.0.0.1:9090/api/v1/admin/tsdb/snapshot | jq
# {
#   "status": "success",
#   "data": {
#     "name": "20260814T120000Z-1a2b3c4d5e6f"
#   }
# }

# READ-ONLY: clean up the snapshot.
rm -rf /var/lib/prometheus/snapshots/20260814T120000Z-1a2b3c4d5e6f

A clean validation: every unauthorised request returns 403, the reload endpoint changes the loaded config when used, and the snapshot path is captured before it is removed.

How it can fail

The five failure modes that show up when admin API or lifecycle endpoints are exposed.

1. Snapshot exfiltration. A peer that can reach the admin API calls POST /api/v1/admin/tsdb/snapshot, downloads the resulting archive, extracts the scrape configs and every metric. The fix is --web.enable-admin-api=false and a bind to loopback.
2. /-/reload with a poisoned config. The attacker writes to prometheus.yml (a separate compromise) and reloads. The new config points federation at a server they control. The fix is the same as above; the reload endpoint is the accelerator, not the root cause.
3. /-/quit invoked during an incident. The on-call is diagnosing an alert storm. The management subnet reaches Prometheus through a permissive NetworkPolicy and the attacker triggers /-/quit to add noise. The fix is to keep lifecycle off in production, or to restrict the bind address.
4. Admin API enabled for “just this once”. A team enables --web.enable-admin-api for a one-off snapshot operation, documents the change, and forgets to remove the flag from the next deployment. The flag survives in production for months. The fix is CI guard rails that refuse the flag in production manifests.
5. PUT /-/reload accepted with a parse-error config. Prometheus 2.55.x rejects an invalid config at reload time and returns 400 with the parse error. The previous config remains in place. This is fail-closed behaviour; the risk is in automation tools that retry on any non-200 response and accidentally DoS the reload path.

How to troubleshoot it

Diagnostic order: is the flag set, can the request reach Prometheus, is the response what the caller expects.

1. Check the running flags.

   cat /proc/$(pgrep -f 'prometheus --config.file')/cmdline | tr '\0' ' '
   # /usr/bin/prometheus --config.file=/etc/prometheus/prometheus.yml \
   #   --storage.tsdb.path=/var/lib/prometheus \
   #   --web.listen-address=127.0.0.1:9090

   No --web.enable-admin-api and no --web.enable-lifecycle in the output means both endpoints return 403. The flag is set at start; it cannot be enabled at runtime.
2. Reproduce the request. curl -v -X POST http://127.0.0.1:9090/api/v1/admin/tsdb/snapshot shows the response code and body. 403 Forbidden means the flag is off; 200 OK means it is on.
3. Inspect Prometheus’s log. A successful reload prints msg="Loading configuration file". An unsuccessful one prints the parse error and the previous config remains active.
4. Reload via SIGHUP from the local shell as the alternative. sudo systemctl reload prometheus sends SIGHUP to the process; the same reload happens without requiring the lifecycle flag.

Security implications

The admin API and lifecycle endpoints together represent the “write surface” of Prometheus. The query API is the “read surface”. The right security posture treats them differently:

• Read surface (query API). Reachable from the Grafana subnet, the on-call engineer’s workstation, and any tool that needs to query metrics. Authentication recommended (lesson 02); network exposure controlled (lesson 01).
• Write surface (admin API + lifecycle). Reachable only from the local automation subnet, behind basic auth, with both flags justified in the runbook. The default is “off”; the activation is a deliberate operational change.

The admin API is also the only endpoint family that touches the on-disk TSDB. A snapshot taken through the admin API is the on-disk TSDB at the moment of the snapshot; it contains the scrape configurations that produced the metrics, which is why exfiltrating a snapshot is functionally equivalent to exfiltrating the configuration.

Verification

You should now be able to answer:

• What does --web.enable-admin-api enable, and what does it not enable?
• What does --web.enable-lifecycle enable, and why is the default “off”?
• Why is a snapshot taken via the admin API effectively an exfiltration of the scrape configuration?
• What is the alternative to PUT /-/reload when the lifecycle endpoints are disabled?

Quiz