Config as Code Basics

A 03:00 incident. A junior engineer has hot-edited /etc/prometheus/prometheus.yml on the production host to add a new scrape job. They remember to reload Prometheus. They forget to commit the file. Eight months later, the disk on that host fails, the host is rebuilt from the immutable image, and the scrape job is gone. The team only notices when the on-call dashboard for that service goes silent. The investigation question is “when did this break?” and the answer is “eight months ago, but we have no record”. This is what config-as-code exists to prevent.

Every observability configuration file that ships in this course (Prometheus, Alertmanager, Grafana provisioning, Loki, Tempo, Alloy) lives in a Git repository, is reviewed before it merges, and is the single source of truth for what runs in each environment. The opposite shape is “click-ops”: a config file that lives on one server, edited through an admin UI, documented in a wiki page that drifts.

What it is

Config-as-code is the discipline of treating every operational configuration file the same way application code is treated: it lives in version control, is reviewed before merge, is built and validated by automation, and the deployed artefact is reproducible from Git alone. The four operational properties it gives a platform team are:

• Review. Every change has a diff, a reviewer, and a rationale. The on-call engineer can audit the past six months of changes to a Prometheus configuration in five minutes.
• Audit. A change merged on a date is the change that ran on that date. A compliance review can answer “which alerting rule fired during the incident” with a commit hash, not a guess.
• Roll-back. A failed change can be reverted with git revert or by redeploying the previous known-good SHA. There is no “we do not remember what was there”.
• Canary. The same configuration can be applied to a staging environment first, validated, and then promoted to production. The same YAML may serve two clusters with environment-specific values injected at deploy time.

The cost of these properties is latency: a config change now moves at the speed of pull-request review rather than the speed of SSH. That latency is the price of safety.

Why a sysadmin cares

The single most common cause of “our observability is wrong and nobody can tell why” is that the configuration is not in Git. Symptoms appear over years, not hours:

1. Drift. The config on disk matches the config in someone’s head; the wiki page that was supposed to match it is six months stale; nobody can tell which one is the truth.
2. Unrecoverable. The disk dies, the host is rebuilt from an immutable image, and the configuration is gone. The rebuild from image is correct; the configuration is not.
3. Unreviewed. The on-call engineer can change anything in the cluster at 03:00 because nobody is in the approval path. The same engineer cannot, after the fact, tell a post-mortem author what they changed and why.
4. Untestable. A change cannot be dry-run against a copy of production telemetry because the production configuration is not in a file that can be copied.

None of these is solvable with runbook discipline alone. Each is solved by Git, by code review, by CI, and by the refusal to host editable configuration on a running server.

How it works

The mental model is straightforward. The repository is the source of truth. The deploy pipeline reads from the repository and writes to the running services. The running services never edit their own configuration.

  Author branches                 Protected branch
       \                              |
        +---> PR + review ---> main --+
                                  |
                          CI: validate
                                  |
                          Artefact store (GitOps)
                                  |
                          Reconciliation loop
                                  |
                       Prometheus | Alertmanager | Grafana
                       Loki       | Tempo        | Alloy

The author branches the repo, edits the YAML, opens a pull request. CI runs promtool check config, amtool check-config, the Grafana provisioning linter, and any unit tests (Prometheus unit test files, alert tests). Failure stops the merge. A reviewer approves. The branch is merged to main. A GitOps controller (ArgoCD, Flux) or a manual deploy pipeline picks the change up. For services that reload on SIGHUP (Prometheus, Alertmanager), the deploy is a file copy and a reload. For Grafana and Loki it is a config map reload. Production now matches main at HEAD.

How to configure it

The shape of config-as-code work for a single observability stack tends to converge on a common layout:

infra/
  observability/
    README.md
    Makefile
    prometheus/
      prometheus.yml         # main config
      rules/
        recording.rules.yml
        alerts.rules.yml
      file_sd/
        api-prod.json
        api-stage.json
    alertmanager/
      alertmanager.yml
      templates/
        default.tmpl
    grafana/
      provisioning/
        datasources/
          prometheus.yml
        dashboards/
          dashboards.yml
        alerting/
          alertmanager.yml
    loki/
      loki.yaml
    tempo/
      tempo.yaml

The Makefile is the smallest thing that pays for itself:

.PHONY: validate
validate:
  promtool check config prometheus/prometheus.yml
  promtool check rules    prometheus/rules/*.yml
  amtool  check-config    alertmanager/alertmanager.yml
  grafana-cli --homepath /tmp/grafana-check lint-dashboards \
              grafana/provisioning/dashboards

Pull requests that change prometheus/prometheus.yml must pass make validate before the reviewer is notified. This is the minimum bar; the dedicated CI-validation lessons later in this part go deeper.

Branch policy:

   main                 protected
     |                  merge only via PR
     |                  squash merge, no force-push
   feat/*  fix/*        branch from main

How to validate it

The minimum validation is that the configuration in Git matches the configuration running on the host, byte for byte. The principle is the same for every service.

Check Prometheus 2.55.x:

# 1. The file parses with the same loader Prometheus uses at runtime
promtool check config /etc/prometheus/prometheus.yml

# 2. The live configuration matches Git
ssh prom-prod-01 sha256sum /etc/prometheus/prometheus.yml
git -C infra/observability sha256sum prometheus/prometheus.yml

# 3. Prometheus reloaded and the new config is active
curl -fsS http://prom-prod-01:9090/-/ready
curl -fsS http://prom-prod-01:9090/api/v1/status/config

The /-/ready endpoint returns 200 only once Prometheus has finished loading the new configuration. The /api/v1/status/config endpoint returns the live YAML that Prometheus is running. Diffing that against Git is the production check:

diff \
  <(ssh prom-prod-01 curl -fsS http://localhost:9090/api/v1/status/config) \
  infra/observability/prometheus/prometheus.yml \
  || echo DRIFT

A non-empty diff is a config-as-code violation. The fix is git push from the host back to Git, followed by a CI review that confirms the diff came from an approved change.

How it can fail

Six shapes of failure appear repeatedly in teams adopting config-as-code.

1. Hot-edit on production. A 02:00 incident, an engineer edits /etc/prometheus/prometheus.yml over an emergency SSH session, fixes the symptom, and forgets to commit. Six months later, the on-call pager fires for a config drift that was already paid for once. The fix is a post-mortem action item with teeth: no SSH keys for ops in the runbook, only break-glass credentials that are logged.
2. Two sources of truth. The production config is in Git and in a Helm values file inside the Kubernetes manifest. They drifted. Helm wins at runtime; Git is what the audit says. The fix is to pick one (Git, in this course’s discipline) and delete the other.
3. Branch sprawl. Engineering teams each have their own monitoring config repo. Three repos for one observability stack. The fix is a single repo with a directory per team and CODEOWNERS enforcing reviewers per path.
4. Reload forgotten. The YAML landed on the host, but no one sent SIGHUP to Prometheus. The new scrape job does not appear in /api/v1/targets. The fix is to wire the GitOps controller to emit SIGHUP on file-change events, and to diff /api/v1/status/config against main after every deploy.
5. Review theatre. A reviewer approved a YAML that broke a rule because the linter was not in CI. The change landed; the rule file was syntax-valid but semantically wrong. The fix is to require promtool check rules to pass before merge.
6. Encrypted secret in Git. A basic-auth password to a private exporter committed in plaintext. The fix is secureJsonData for Grafana, basic_auth.password_file for Prometheus, and secrets sourced from a vault with the YAML file containing only ${SECRET_NAME} references.

Security implications

Config-as-code centralises secrets. The plaintext-password-in-YAML shape is the most common config-as-code breach. Mitigations:

• Replace basic-auth usernames and passwords with basic_auth.password_file in prometheus.yml. Pass the password through an environment variable resolved from a vault.
• Use secureJsonData for Grafana data source credentials. The provisioning loader writes the secret to the data source object; the file on disk does not contain it.
• Mark secrets-bearing files in .gitignore. The CI runs gitleaks or trufflehog against the PR; failures block merge.

The single biggest permission question is “who can merge to main?” The answer is: nobody, except by pull request, with two reviewers from the affected team as defined in CODEOWNERS.

Performance implications

Config-as-code adds zero CPU, memory, or network cost to the running services. It adds cost only at deploy time (a promtool check rules pass is in milliseconds). The performance trade-off is where the latency budget for “change a config” goes. With config-as-code, deploy latency (PR review plus CI plus reconciliation) replaces the latency of an SSH session (zero) with a structured process (minutes to hours). That latency is the cost of safety; it is invisible during quiet weeks and very visible during incidents.

Production guidance

• One repo per observability stack. A CODEOWNERS file per directory.
• promtool, amtool, and the Grafana CLI must be available in CI before any merge.
• Deploy via GitOps where available. Manual kubectl apply -f is acceptable only if every apply is preceded by git diff and a git push.
• Forbid host-level edits to any path under /etc/\{prometheus, alertmanager,grafana,loki,tempo\} outside the GitOps pipeline. SSH keys are pulled, not pushed.
• Use environment-specific overlays (prometheus.prod.yml, prometheus.stage.yml) with a shared prometheus.common.yml and a small kustomize or Helm-values patch at the layer above.

Verification

You should now be able to answer:

• What four operational properties does config-as-code give a platform team?
• Why is a private scratch repository correct for a sandbox environment, while production lives in a protected branch?
• What is the failure shape when a hot-edit on a production host is not committed back to Git?
• What three commands confirm the live Prometheus configuration matches Git?

Quiz