Security Overview · Observability

A Grafana install has been live for nine months. Nobody has touched the firewall rules since launch. The default --web.listen-address=:9090 on Prometheus still binds to every interface. The node_exporter on the database host binds to 0.0.0.0:9100 because the systemd unit file was copied from a public gist. Loki’s ingester listens on 0.0.0.0:3100 because the docker-compose example did not pin it. A single Shodan-style scan finds all three. The database-host metrics include the connection-pool exhaustion that occurred at 03:00 last Tuesday, the query that named a table that contained user PII, and the fact that the disk is filling up. The metrics are doing their job. The metrics are also the leak.

This is what the word security means in an observability context: the platform that records everything is also the platform that exposes everything. The lesson is about the boundaries that decide what is reachable, what is authenticated, what is authorised, what is encrypted, and what is auditable.

What it is

The observability platform security model is the set of boundaries between the public internet, the operator browser, the internal service mesh, the storage backends, and the secrets manager. Five components appear in every reference architecture in this course:

                       Public Internet
                              |
                  +-----------+-----------+
                  |       Reverse Proxy   |     TLS terminator
                  +-----------+-----------+
                              |
                  +-----------+-----------+
                  |        Grafana        |     user-facing UI
                  +-----------+-----------+
                              |
              +---------------+----------------+
              |               |                |
       +------+------+ +------+------+ +-------+-------+
       |  Prometheus  | |    Loki     | |     Tempo     |   data-plane APIs
       +------+------+ +------+------+ +-------+-------+
              |               |                |
              +-------+-------+--------+-------+
                      |                |
              +-------+------+  +------+-------+
              |   Object     |  |   Secrets    |
              |   Storage    |  |   Manager    |
              +-------------+  +--------------+

Each box has its own authentication, its own authorisation model, its own TLS posture, and its own secrets-management story. Five boxes, five different decisions. The wrong shape for one of them is a leak in the others.

Why a sysadmin cares

Production observability carries three risks that production applications usually do not.

The platform holds the keys to the kingdom. Metrics include the response codes for the admin endpoints, the request rate against the payments API, and the timestamps of every certificate rotation. Logs include the application error messages that name the columns, the tables, and the queries. Traces include the dependency map and the timing profile of every internal call. The platform that lets an engineer investigate a production incident is also the platform that, in the wrong hands, lets an attacker reconnoitre one.
The platform is treated as “internal.” It is not. A Grafana behind a permissive firewall rule, a Loki without authentication, a Prometheus with the admin API exposed, and a node_exporter on 0.0.0.0 are all reachable from any compromised host on the network. The blast radius of a single developer laptop compromise is “everything the observability stack has ever recorded.”
The platform is rarely patched as aggressively as the applications. Security teams focus on the products that face customers. The Grafana, the Prometheus, and the Loki that the security team uses for their own dashboards are often a release or two behind. The exposure window is longer.

How it works

Five components, five boundary decisions. The boundary model is the same for every component; the configuration choices differ.

   Component       | Public surface          | Internal surface
   ----------------+-------------------------+-------------------------
   Prometheus      | /api/v1/query, /-/...    | /api/v1/admin/...
                   | federation, query        | tsdb, snapshot, delete
                   |                         |
   Loki            | /loki/api/v1/query       | /loki/api/v1/push
                   | /loki/api/v1/tail        | /loki/api/v1/admin
                   |                         |
   Tempo           | /api/search              | /api/traces
                   | /api/traces/{id}         | /api/ingest (OTLP)
                   |                         |
   OTel Collector  | /api/v1/trace, /metrics  | exporters (otlp, prom...)
   Alloy           | /metrics, /debug/pprof   | otlp, loki, prom exporters
                   |                         |
   Grafana         | /login, /api/*           | /api/datasources/proxy/*
                   |                         | /metrics, /debug/pprof

Two patterns stand out:

Every component exposes a read path and an admin path. The read path is what the dashboards need; the admin path is what the operators need. The same component serves both. The configuration decision is whether the admin path is reachable from the same network as the read path.
The data-plane API is the dangerous one. Loki’s /loki/api/v1/push accepts log lines from anywhere it can reach. Prometheus’s remote-write accepts metrics from anywhere it can reach. Tempo’s OTLP receiver accepts spans. An open data-plane API is a write surface for an attacker; an open admin API is a control surface.

The right discipline: defence in depth

No single control makes an observability stack secure. The discipline is layered, and each layer has a job that the others do not.

   Layer        | Job                              | What fails if it is missing
   --------------+----------------------------------+--------------------------------
   Network       | Decide what is reachable        | Components are exposed on
                 |                                  | the public internet
                 |                                  |
   TLS           | Encrypt traffic in transit      | Credentials and tokens
                 |                                  | are readable on the wire
                 |                                  |
   Authentication| Decide who is calling            | Anyone reachable can read
                 |                                  | or write
                 |                                  |
   Authorization | Decide what a caller may do      | A read-only user can
                 |                                  | delete series; an editor
                 |                                  | can see every tenant
                 |                                  |
   Secrets       | Decide where credentials live    | Credentials are in Git,
                 |                                  | in the database, in logs
                 |                                  |
   Audit         | Decide who did what              | An incident has no
                 |                                  | accountable trail

The order matters. Network controls fail open when a firewall rule is wrong; TLS fails closed when a certificate expires; auth fails open when the default user is left enabled; authz fails open when the default role is Admin; secrets fail open when the secret is in cleartext in the provisioning YAML. Every layer has a default that is wrong. The job of the operator is to override every default.

How to configure it

The right discipline is a baseline configuration per component. The configuration shown here is the minimum; later lessons in this module expand each section. The point of this lesson is to show what the baseline is; the lessons that follow show how.

Network exposure: bind to specific interfaces

# /etc/prometheus/prometheus.yml
# Bind the HTTP listener to the monitoring VLAN interface.
web:
  listen-address: 10.0.10.5:9090
  external-url: https://prometheus.internal.example.com

# /etc/loki/loki-config.yaml
server:
  http_listen_address: 10.0.10.6:3100
  grpc_listen_address: 10.0.10.6:9096
  log_level: info

# /etc/tempo/tempo.yaml
server:
  http_listen_address: 10.0.10.7:3200
  grpc_listen_address: 10.0.10.7:9095

# /etc/grafana/grafana.ini
[server]
http_addr = 127.0.0.1
http_port = 3000

The Grafana loopback bind is intentional; the lesson on reverse proxying shows why the proxy and not the Grafana should be the public surface.

Authentication: a per-component baseline

# /etc/prometheus/web_config.yml
basic_auth_users:
  admin: $2y$10$bcrypt-hash-of-password
  alertmanager: $2y$10$bcrypt-hash-of-password

# /etc/loki/loki-config.yaml
auth_enabled: true

# /etc/tempo/tempo.yaml
authentication:
  enabled: true

# /etc/grafana/grafana.ini
[auth.basic]
enabled = false

[auth.github]
enabled = true
allow_sign_up = false
client_id = ${GITHUB_OAUTH_CLIENT_ID}
client_secret = ${GITHUB_OAUTH_CLIENT_SECRET}
scopes = user:email
auth_url = https://github.com/login/oauth/authorize
token_url = https://github.com/login/oauth/access_token
api_url = https://api.github.com/user

Authorization: the principle of least privilege

# /etc/loki/loki-config.yaml
auth_enabled: true

# Tenant isolation: every request must carry X-Scope-OrgID.
# The single-tenant Loki does not need this; multi-tenant Loki
# relies on it.

# /etc/grafana/grafana.ini
[users]
auto_assign_org_role = Viewer

[access_control]
enabled = true

TLS: per component or terminator

# /etc/prometheus/web_config.yml
tls_server_config:
  cert_file: /etc/prometheus/certs/prometheus.crt
  key_file: /etc/prometheus/certs/prometheus.key

# /etc/loki/loki-config.yaml
server:
  http_tls_config:
    cert_file: /etc/loki/certs/loki.crt
    key_file: /etc/loki/certs/loki.key

# /etc/grafana/grafana.ini
[server]
protocol = http
# TLS terminates at the reverse proxy.
cert_file =
key_file =

Secrets: never in Git, never in log lines

# /etc/grafana/provisioning/datasources/loki.yaml
secureJsonData:
  basicAuthPassword: ${LOKI_PASSWORD}

# /etc/prometheus/prometheus.yml
remote_write:
  - url: https://prometheus-remote.internal.example.com/api/v1/write
    basic_auth:
      username: ${REMOTE_WRITE_USERNAME}
      password_file: /etc/prometheus/secrets/remote_write_password

How to validate it

# READ-ONLY: confirm Prometheus listens on the expected interface.
ss -tlnp | grep -E ':9090|:9100|:3100|:3200|:3000'
# LISTEN 0  4096  10.0.10.5:9090   prometheus
# LISTEN 0  4096  10.0.10.5:9100   node_exporter
# LISTEN 0  4096  10.0.10.6:3100   loki
# LISTEN 0  4096  10.0.10.7:3200   tempo
# LISTEN 0  4096  127.0.0.1:3000   grafana

# READ-ONLY: confirm Grafana is not on the public network.
curl -fsS --max-time 3 http://grafana.internal.example.com:3000/api/health
# {"database":"ok","version":"11.3.0"}
curl -fsS --max-time 3 http://grafana.example.com:3000/api/health
# curl: (7) Failed to connect to grafana.example.com port 3000

# READ-ONLY: confirm Prometheus requires authentication.
curl -fsS --max-time 3 http://prometheus.internal.example.com:9090/api/v1/query?query=up
# {"status":"error","errorType":"internal","error":"authentication required"}
curl -fsS -u 'alertmanager:secret' \
  http://prometheus.internal.example.com:9090/api/v1/query?query=up
# {"status":"success","data":{"resultType":"vector","result":[{"metric":...}]}}

# READ-ONLY: confirm Loki requires authentication.
curl -fsS --max-time 3 http://loki.internal.example.com:3100/ready
# ready
curl -fsS --max-time 3 -X POST -H 'Content-Type: application/json' \
  -d '{"streams":[{"stream":{"job":"test"},"values":[["1","x"]]}]}' \
  http://loki.internal.example.com:3100/loki/api/v1/push
# {"status":"error","error":"authentication required"}

# READ-ONLY: confirm Grafana is behind the reverse proxy.
curl -fsSI https://grafana.example.com/api/health
# HTTP/2 200
# server: nginx
# strict-transport-security: max-age=15768000

# CONFIGURATION: run a port scan from outside the network.
nmap -p 9090,9100,3100,3200 grafana.example.com
# Not shown: 9090, 9100, 3100, 3200 closed or filtered

A clean baseline: every component binds to the expected interface, the data-plane APIs reject unauthenticated requests, the user-facing surface is HTTPS only, and a port scan from the public network returns no observability components.

How it can fail

The high-frequency security baseline failure modes from real incidents.

Prometheus still on 0.0.0.0. The --web.listen-address flag is absent, or set to :9090. Prometheus binds every interface. A port scan finds it; an attacker scrapes every metric, including the ones that name internal services. The visible symptom is ss -tlnp showing *:9090 instead of 10.0.10.5:9090.
Grafana with auto_assign_org_role = Admin. The first user to log in via OAuth becomes an Org Admin. The visible symptom is the Org Admin role being assigned to a service account or to an external identity provider account that should never have that role.
Loki with auth_enabled: false and a permissive firewall. Loki accepts log lines from anyone. The visible symptom is a sustained spike in ingester_bytes_received that does not match the application log volume, with no corresponding application logs at the same timestamp.
Tempo’s OTLP receiver on the public network. Tempo accepts spans on :4317 (gRPC) and :4318 (HTTP). An attacker pushes malformed spans that fill the block storage. The visible symptom is the Tempo ingester OOMing on the next query.
node_exporter on 0.0.0.0:9100 with no firewall rule. The exporter exposes host metrics, including mount point and filesystem metrics, that name the databases. The visible symptom is an external IP showing up in node_network_receive_bytes_total with no corresponding internal traffic.
Plaintext credentials in web_config.yml. The basic auth file is admin:admin in cleartext. The visible symptom is grep admin /etc/prometheus/web_config.yml returning a cleartext pair.

How to troubleshoot it

The diagnostic order is “is the component reachable where I expect?”, “is authentication enforced?”, “is authorisation enforced?”, “is traffic encrypted?”, “is the credential stored safely?”

Re-read ss -tlnp for every component on every host. Anything on 0.0.0.0 is a finding; anything on a wildcard IPv6 address ([::]) is the same finding on the IPv6 side.
Send an unauthenticated request to every read and admin path. A 200 is a finding; a 401 or a 403 is the expected response.
Send an authenticated request as a read-only user to every admin path. A 200 is a privilege escalation; a 403 is the expected response.
Inspect the certificate chain with openssl s_client -connect for every TLS endpoint. A self-signed certificate on a public endpoint is a finding.
Search the configuration tree for cleartext secrets with a scanner such as gitleaks or trufflehog. Any match is a finding.
Audit the user list with grafana-cli admin list-users or the equivalent API call. The first OAuth user should not be an Admin by default.

Quiet data exfiltration and the loud alert

A silent failure looks like: the platform is live, the dashboards render, the alerts fire, and a quiet data exfiltration is in progress. An attacker on the same network reads the Prometheus admin API, snapshots the TSDB, walks every metric name, identifies the /actuator/env endpoint from the JVM metrics, and reads the environment variables of every service. The metrics are doing their job; the metrics are also the attacker reconnaissance.

A loud failure looks like: a Grafana instance is compromised via a stolen OAuth refresh token. The attacker creates a new service account with Admin role, exports every dashboard, reads every data source credential, and pushes a fake alert. The alerting engine fires on a fake alert that pages the on-call team. The on-call team discovers the compromise the moment they try to silence the fake alert and the silence is denied. The fix is to revoke the token, rotate every credential the Grafana service account could reach, and audit every change the Grafana Admin role made in the window.

Security implications

The platform is a high-value target. A read-only observer can reconstruct the deployment, the dependency map, and the secrets in the environment variables. Treat the observability stack with the same threat model as a customer database.
The default install is not a baseline. The right baseline is the configuration in the “How to configure it” section, applied at install time, not the upstream default.
Audit trails are part of the security model. A platform that does not record who read what is a platform that cannot detect a compromise. Grafana audit logs, Loki per-tenant access logs, and Prometheus admin API audit logs are all required.
Secrets management is the boundary. Credentials that live in cleartext in the configuration are a leak; credentials that live in Vault are auditable. The lesson on secrets expands this.

Performance implications

Authentication costs CPU. Basic auth on the Prometheus HTTP path is bcrypt, which is intentionally slow. The default is acceptable for a Prometheus that receives one scrape every 15 seconds; a Prometheus with thousands of scrapers and admin API users will see measurable CPU on the auth path.
TLS termination costs CPU at the proxy. A reverse proxy terminates TLS for Grafana; the cost is on the proxy, not on the Grafana. With modern AES-NI hardware and session resumption, the cost is small.
Authorization checks on Loki’s query path. Every query carries a tenant ID; the query path resolves the tenant limits. A Loki with many tenants and a long query history shows the cost on the query path, not on the ingester.

Production guidance

Bind every component to the specific interface that should reach it. The loopback bind is the only safe default for Grafana; the monitoring VLAN bind is the safe default for the data plane.
Require authentication on every component. The default user account is the first finding in any audit.
Run Grafana behind a TLS-terminating reverse proxy. Do not terminate TLS on the Grafana process; the proxy is the security boundary.
Store credentials in a secrets manager. Provisioning YAML uses ${VAR} interpolation; runtime configuration uses password_file references.
Audit at least once per quarter. The audit confirms the network bind, the authentication enforcement, the authorisation posture, and the secrets-management shape.
Patch on a predictable cadence. A Grafana that is two releases behind is a Grafana with known CVEs; the security team will ask why.

Verification

You should now be able to answer:

What is the largest blast radius when a single observability endpoint is compromised, and which endpoint is the highest value target?
Why is the upstream default for every component in this stack not the right production baseline?
Why is “authenticate everything” the wrong discipline on its own, and what does the defence-in-depth model add?
What is the difference between an open data-plane API (Loki push, Tempo OTLP) and an open admin API, and which is more dangerous in a particular scenario?
Why is the Grafana auto_assign_org_role = Admin default the most expensive line in any production grafana.ini?