Secrets Management

A Loki remote-write target was added to the production Prometheus in a hurry during an incident. The operator copy- pasted the Authorization: Bearer <token> line from a runbook that had been edited by a contractor. The token was a service-account token that was rotated quarterly. The token lived in the runbook; the runbook lived in the ops wiki; the ops wiki was indexed by an external search engine. The token was valid for another two months. The Loki ingester accepted writes from anyone with the token. Two months later, the quarterly rotation happened. The token was revoked. Loki stopped accepting writes. The dashboards turned red. The on-call team discovered the rotation from the alert. The fix took four hours.

This is what the word secrets means in an observability context: the boundary that decides where credentials live. The lesson is about the right model per component, and the failure modes of the wrong one.

What it is

Secrets management is the discipline that decides where credentials are born, where they live, how they are rotated, and how they are audited. In the observability stack, five patterns appear:

   Pattern              | Strengths                  | Weaknesses               | Where it fits
   ---------------------+----------------------------+--------------------------+---------------------
   Cleartext in config  | Simple                     | Credential in Git        | Never in production
                       |                            |                          |
   Env-var              | Decoupled from config      | Restart to pick up new   | Slow-rotation
   interpolation        |                            | value                    | credentials
   (${VAR})             |                            |                          |
                       |                            |                          |
   File reference       | Decoupled from config      | File permissions must    | Long-lived service
   (password_file)      | Rotated by file write      | be set correctly         | credentials
                       |                            |                          |
   Vault integration    | Source of truth at Vault   | Vault must be reachable  | Fast-rotation
                       | Rotation at Vault is       |                          | credentials
                       | reflected without restart  |                          |
                       |                            |                          |
   External secrets     | Source of truth at the     | Restart to pick up new   | Kubernetes-style
   operator             | external system            | value                    | deployments
                       |                            |                          |

The right pattern is different for every credential. The fast- rotation credentials (OAuth client secrets, database passwords) use Vault integration. The slow-rotation credentials (service account passwords, API tokens) use file references or env-var interpolation. The never-in-production pattern is cleartext in config.

Why a sysadmin cares

Three production failure modes map directly to wrong secrets- management choices.

1. A credential in Git. The credential was checked into the provisioning YAML; the YAML is in the ops repo; the repo is indexed by an external search engine; the credential is valid until rotation. The blast radius is “every developer with read access to the repo, plus every attacker who has indexed the repo.”
2. A credential in log lines. The application logs the Authorization header at debug level; the logs are shipped to Loki; the Loki label cardinality is high enough that the credential is searchable by X-Scope-OrgID. The blast radius is “every Grafana user with access to the logs.”
3. A credential that is not rotated. The credential was issued at install time; the rotation is a quarterly cron job that no one has looked at in a year. The blast radius is “the credential is valid forever; the audit trail is the Git history.”

How it works

Every consumer reads a credential from a known source and uses it to authenticate to the producer. The source of the credential varies by pattern.

   Vault (source of truth)
      |
      |  kv put secret/observability/loki-remote-write
      |     username=...
      |     password=...
      |
      v
   Vault Agent (sink)
      |
      |  writes /run/observability/loki.env
      |     LOKI_USERNAME=...
      |     LOKI_PASSWORD=...
      |
      v
   Consumer (Prometheus / Loki / Tempo / Grafana)
      |
      |  reads LOKI_PASSWORD from environment
      |  or reads /etc/observability/secrets/loki.password
      |
      v
   Producer (Loki / Tempo / remote-write target)
      |
      |  validates credential on every request
      |
      v
   Audit log
      |
      |  records who read the credential and when

The Vault is the source of truth. The Vault Agent writes the credential to a file or environment variable that the consumer reads. The producer validates the credential on every request. The audit log records every read.

How to configure it

Prometheus: file references

# /etc/prometheus/prometheus.yml
remote_write:
  - url: https://prometheus-remote.internal.example.com/api/v1/write
    basic_auth:
      username: ${REMOTE_WRITE_USERNAME}
      password_file: /etc/prometheus/secrets/remote_write_password

The password_file reference is read on every remote-write attempt. The file is owned by the Prometheus user; permissions are 0600. The Vault Agent writes the file; the Prometheus user reads it.

Loki: file references

# /etc/loki/loki-config.yaml
common:
  storage:
    s3:
      s3forcepathstyle: true
      access_key_id: ${S3_ACCESS_KEY_ID}
      secret_access_key_file: /etc/loki/secrets/s3_secret_access_key

# /etc/systemd/system/loki.service
[Service]
EnvironmentFile=/etc/loki/secrets/loki.env

The Vault Agent writes /etc/loki/secrets/loki.env with the environment variables; systemd reads the file and sets the environment for the Loki process.

Tempo: file references

# /etc/tempo/tempo.yaml
storage:
  trace:
    backend: s3
    s3:
      access_key: ${S3_ACCESS_KEY_ID}
      secret_key_file: /etc/tempo/secrets/s3_secret_key

OpenTelemetry Collector: env-var interpolation

# /etc/otelcol/config.yaml
exporters:
  prometheusremotewrite:
    endpoint: http://prometheus.internal.example.com:9090/api/v1/write
    auth:
      authenticator: basicauth

extensions:
  basicauth:
    username: ${REMOTE_WRITE_USERNAME}
    password: ${REMOTE_WRITE_PASSWORD}

service:
  extensions: [basicauth]
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [prometheusremotewrite]

# /etc/systemd/system/otelcol.service
[Service]
EnvironmentFile=/etc/otelcol/secrets/otelcol.env

Grafana Alloy: env-var interpolation

# /etc/alloy/config.alloy
basic_auth "remote_write" {
  username = sys.env("REMOTE_WRITE_USERNAME")
  password = sys.env("REMOTE_WRITE_PASSWORD")
}

prometheus.remote_write "default" {
  endpoint {
    url = "http://prometheus.internal.example.com:9090/api/v1/write"
    basic_auth {
      username = sys.env("REMOTE_WRITE_USERNAME")
      password = sys.env("REMOTE_WRITE_PASSWORD")
    }
  }
}

Grafana: Vault integration

# /etc/grafana/grafana.ini
[secrets.manager.grafana.vault]
vault_address = https://vault.internal.example.com:8200
vault_token = ${VAULT_TOKEN}
vault_kv_path = secret/data/grafana

# /etc/grafana/provisioning/datasources/loki.yaml
apiVersion: 1
datasources:
  - name: Loki
    type: loki
    uid: loki-prod
    orgId: 1
    url: https://loki.internal.example.com
    access: proxy
    jsonData:
      basicAuth: true
    secureJsonData:
      basicAuthUser: vault://secret/data/grafana/loki#username
      basicAuthPassword: vault://secret/data/grafana/loki#password

Vault Agent sink (the source pattern)

# /etc/vault-agent/template.hcl
auto_auth {
  method "approle" {
    config = {
      role_id_file_path   = "/run/vault/role-id"
      secret_id_file_path = "/run/vault/secret-id"
    }
  }
  sink "file" {
    config = {
      path = "/run/vault/token"
    }
  }
}

template {
  destination = "/etc/loki/secrets/loki.env"
  contents = <<EOT
S3_ACCESS_KEY_ID={{ with secret "secret/data/observability/loki" }}{{ .Data.data.access_key_id }}{{ end }}
EOT
}

template {
  destination = "/etc/loki/secrets/s3_secret_access_key"
  perms = "0600"
  contents = <<EOT
{{ with secret "secret/data/observability/loki" }}{{ .Data.data.secret_access_key }}{{ end }}
EOT
}

How to validate it

# READ-ONLY: confirm a credential does not appear in cleartext in the provisioning YAML.
grep -r 'password:' /etc/prometheus /etc/loki /etc/tempo /etc/grafana/provisioning \
  | grep -v password_file | grep -v '${'
# (no output; every password is either a file reference or env-var interpolation)

# READ-ONLY: confirm the credential file exists and has the right permissions.
ls -la /etc/prometheus/secrets/remote_write_password
# -rw------- 1 prometheus prometheus 32 Aug 14 03:00 /etc/prometheus/secrets/remote_write_password

# READ-ONLY: confirm the credential file is current.
stat -c '%y' /etc/prometheus/secrets/remote_write_password
# 2026-08-14 03:00:00

# READ-ONLY: confirm the credential works against the producer.
curl -fsS -u "${REMOTE_WRITE_USERNAME}:$(cat /etc/prometheus/secrets/remote_write_password)" \
  http://prometheus-remote.internal.example.com/api/v1/write -X POST
# (empty body; the producer accepts the credential)

# READ-ONLY: confirm Grafana Vault integration is enabled.
curl -fsS -H "Authorization: Bearer ${GF_SA_TOKEN}" \
  https://grafana.example.com/api/frontend/settings | jq '.secretManager'
# "vault"

# CONFIGURATION: rotate a credential at Vault.
vault kv put secret/observability/loki \
  access_key_id="$(openssl rand -hex 16)" \
  secret_access_key="$(openssl rand -hex 32)"

# CONFIGURATION: restart Vault Agent to pick up the new credential.
sudo systemctl restart vault-agent

# CONFIGURATION: run a secret scanner against the provisioning tree.
gitleaks detect --source /etc/grafana/provisioning --no-git
# (no findings; every credential is a reference)

# CONFIGURATION: run a secret scanner against the ops repo.
gitleaks detect --source /opt/ops/repo --no-git
# (no findings; the runbook is in the repo with the credential redacted)

A clean validation: every credential is a file reference or env-var interpolation, the credential files have the right permissions, the Vault integration is enabled, and the secret scanner reports no findings.

How it can fail

The high-frequency secrets-management failure modes from real incidents.

1. Credential in Git. A provisioning YAML was edited with the credential in cleartext; the credential was rotated later; the credential in Git is still valid until the Git history is rewritten. The visible symptom is a gitleaks finding on the historical commit.
2. Credential in log lines. An application logs the Authorization header at debug level; the logs are shipped to Loki; the credential is searchable. The visible symptom is grep -r 'Authorization' /var/log/app returning the credential in cleartext.
3. Credential not rotated. The credential was issued at install time; the rotation is a quarterly cron job that no one has looked at in a year. The visible symptom is the credential’s age exceeding the rotation cadence.
4. Vault unreachable. The Vault server is down or the network path is broken. The consumer cannot read the credential. The visible symptom is every data source returning “Vault unavailable” on the next cache miss.
5. Vault token expired. The token the consumer was issued has a finite lifetime. A token that expired produces “permission denied” on every Vault read. The visible symptom is a sudden wave of authentication failures.
6. File permissions wrong. The credential file is readable by every user on the host. The visible symptom is ls -la /etc/secrets/ showing -rw-r--r-- instead of -rw-------.

How to troubleshoot it

The diagnostic order is “where does the credential live?”, “is the credential current?”, “can the consumer reach the credential?”, “does the producer accept the credential?”.

1. Inspect the provisioning YAML. A password: line without a password_file reference and without ${VAR} interpolation is a finding.
2. Inspect the credential file permissions. The file must be 0600 and owned by the consumer’s user.
3. Inspect the credential age. A credential that has not been rotated in longer than the rotation cadence is a finding.
4. Test the credential against the producer. A 401 is a credential mismatch; a 200 is the expected response.
5. Inspect the Vault audit log for failed reads. A pattern of “permission denied” indicates an expired token.
6. Inspect the consumer’s log for “Vault unavailable” or “permission denied”. The error shape names the failure.

Security implications

• Vault is the source of truth. The Grafana database is a backup target, not a secrets manager.
• A scanner in CI is the safety net. gitleaks or trufflehog in CI catches the operator who writes basicAuthPassword: hunter2 in the YAML.
• Credential rotation is a lifecycle discipline. A credential that is not rotated is a credential that is valid until someone notices.
• The audit log is the operational surface. The Vault audit log records who read which secret when; the Grafana audit log records that a query was issued; the two together are a complete picture.

Performance implications

• Vault reads add latency. Every cache miss reads from Vault over the network. With a 5-minute TTL and one query per second per data source, the cost is negligible.
• File references are fast. A password_file reference is one disk read per remote-write attempt; the cost is invisible.
• Env-var interpolation is the fastest. The value is loaded once at process start; the cost is zero at request time.

Production guidance

• Vault as the source of truth for every fast-rotation credential (OAuth client secrets, database passwords, API tokens).
• File references for every long-lived service credential (S3 access keys, S3 secret keys, basic-auth passwords).
• Env-var interpolation for every slow-rotation operator credential (Grafana admin password, OAuth client secret).
• A scanner in CI that fails the build on cleartext credentials in the provisioning YAML.
• A rotation cadence review quarterly: every credential has an owner, a TTL, and a last-rotated date.
• An alert on Vault unavailable in the consumer log, surfaced to the channel that handles the on-call rotation.

Verification

You should now be able to answer:

• What is the right secrets-management pattern for a fast-rotation credential, and why?
• What is the right secrets-management pattern for a long-lived service credential, and why?
• Why is cleartext in the provisioning YAML a credential in the Git history, and what scanner discipline catches the operator who writes it?
• What is the failure shape of an unreachable Vault, and how is it distinguished from an expired Vault token?
• Why is the Vault audit log the rotation trail, and how does it combine with the consumer’s audit log to give a complete picture?

Quiz