Reference Architecture

A canonical reference architecture: collectors, Prometheus, Loki, Tempo, Grafana, Alertmanager.

What it is

A precise definition scoped to production operations.

Why a sysadmin cares

Production framing.

How it works

The mental model.

example_setting: value

How to configure it

Real configuration examples with annotated options.

promtool check config /etc/prometheus/prometheus.yml

How to validate it

Commands the operator runs to confirm the configuration is live and correct.

How it can fail

Failure modes: silent misconfiguration, crash on load, performance regression, permissions failure, schema / version drift.

How to troubleshoot it

Diagnostic order: was it working before? what does the service view say? what does the platform view say? form hypothesis, find evidence, test, validate.

The Final Reference Architecture

The complete production observability reference:

graph TB
    subgraph Workloads
        WL1[Linux + Docker<br/>+ Services]
    end
    
    subgraph Collectors
        WL1 --> EX[Exporters + OTel Collector]
    end
    
    subgraph Storage Backends
        EX --> PROM[Prometheus<br/>TSDB]
        EX --> LOKI[Loki<br/>Object Store]
        EX --> TEMP[Tempo<br/>Object Store]
    end
    
    subgraph Presentation
        PROM --> GM[Grafana]
        LOKI --> GM
        TEMP --> GM
    end
    
    subgraph Alerting
        PROM --> AM[Alertmanager]
        AM --> RC[Receivers]
    end
    
    subgraph Configuration
        CFG[Git Repository] -->|CI| PROM
        CFG -->|CI| LOKI
        CFG -->|CI| TEMP
        CFG -->|CI| GM
    end
    
    subgraph Security
        SEC[TLS + RBAC + Secrets] --> PROM
        SEC --> LOKI
        SEC --> TEMP
        SEC --> GM
    end

This is the course’s final production reference architecture. It integrates every Part: the production workloads, the collector layer, the storage backends, the presentation layer, alerting, configuration as code, and security. The capstone validates this architecture end-to-end.

Security implications

The topic has security implications wherever the relevant component exposes an HTTP endpoint, an authentication layer, or a credential.

Performance implications

Performance implications come from cardinality, scrape / push interval, rule size, retention, and query cost.