Observability · Self-assessment
Knowledge checks
Every knowledge check in this course, in curriculum order. Each link opens the page at its quiz. The questions are auto-graded in the browser and nothing is recorded — a wrong answer costs you only the explanation, which is the part worth reading.
- Knowledge checks
- 684
- Parts covered
- 114
- Of all lessons
- 100%
Part I
Foundations
6 checks
- Why Observability ExistsThe boundary between monitoring and observability, what telemetry signals buy you, and what they do not.→
- The Three Telemetry SignalsWhat metrics, logs and traces measure, how they are stored, and the questions each answers uniquely.→
- White-Box and Black-Box MonitoringThe fundamental distinction between monitoring a service from the inside and from the outside.→
- Symptoms, Evidence, and CausalityHow an investigator separates a symptom (observed) from a cause (inferred).→
- Service Health, User Impact, and TelemetryThe chain that connects an internal signal to a user impact decision.→
- Context, Correlation, and CausalityWhy an observability platform must carry context across services and how correlation IDs enable pivot-across-signal investigation.→
Part II
Production Monitoring Fundamentals
6 checks
- SLIs, SLOs, and Error BudgetsThe metrics definitions that anchor production observability: SLI, SLO, error budget.→
- Availability, Latency, Throughput, Errors, SaturationThe five operational dimensions that the platform measures for every service.→
- USE Methodology: Utilisation, Saturation, ErrorsResource-focused methodology for systems where the bottleneck is the resource itself.→
- RED Methodology: Rate, Errors, DurationService-focused methodology for distributed services.→
- Choosing the Right Methodology for Each SystemHow to pick between RED, USE, and other methodologies.→
- The Operational SLO PracticeHow to design, operate, and iterate on SLOs in production.→
Part III
Metrics Fundamentals
6 checks
- Counters, Gauges, Histograms, and SummariesThe four Prometheus metric types, what each captures, and when to choose which.→
- Labels, Dimensions, and Time SeriesHow labels and dimensions define a time series, and the design rules for using them.→
- Metric Naming ConventionsHow to name a metric so it survives across teams, dashboards, and alerts. A production-focused deep dive.→
- Aggregation Across Instancessum / avg / by / without aggregations; what they mean for the dashboard. A production-focused deep dive.→
- Scrape Intervals and ResolutionHow scrape interval affects what you can detect. A production-focused deep dive.→
- Time Series Storage and the TSDBHow Prometheus stores samples, and what 2.55 changes. A production-focused deep dive.→
Part IV
Cardinality
6 checks
- The Cardinality BudgetThe hard math of how many time series the platform can hold. A production-focused deep dive.→
- Dangerous Labels in PracticeWhy user IDs and request IDs are dangerous. A production-focused deep dive.→
- Cardinality From InstrumentationHow instrumentation patterns drive cardinality. A production-focused deep dive.→
- Cardinality From Loki LabelsLoki labels must be low-cardinality; log lines must not. A production-focused deep dive.→
- Cardinality Incident ResponseDiagnose, mitigate, recover, and prevent a cardinality blow-up. A production-focused deep dive.→
- Cardinality GovernanceReviews, lint rules, and audit cadence. A production-focused deep dive.→
Part V
Prometheus Architecture
6 checks
- The Prometheus Pull ModelWhy scrape is the primary ingestion mechanism. A production-focused deep dive.→
- The Scrape LifecycleWhat happens during a scrape: HTTP, parses, writes. A production-focused deep dive.→
- The TSDB EngineBlocks, compaction, mmapped files, what changed in 2.55. A production-focused deep dive.→
- Rules and the Alerting PipelineRecording rules, alerting rules, evaluation, Alertmanager. A production-focused deep dive.→
- FederationHierarchical Prometheus for multi-tenant estates. A production-focused deep dive.→
- Remote Write and Remote ReceiveHow scaling Prometheus works in 2026. A production-focused deep dive.→
Part VI
Installing Prometheus
6 checks
- Prometheus Installation Methodsbinaries, packages, containers, Compose. A production-focused deep dive.→
- Filesystem LayoutWhere configuration, data, and logs live. A production-focused deep dive.→
- systemd IntegrationService user, lifecycle, sandboxing. A production-focused deep dive.→
- Docker Compose InstallationThe lab default. A production-focused deep dive.→
- Retention and Storageretention.time, retention.size, TSDB. A production-focused deep dive.→
- Permissions and Service UserLeast privilege, capabilities, filesystem permissions. A production-focused deep dive.→
Part VII
Prometheus Configuration
6 checks
- Global Settingsscrape_interval, evaluation_interval, external_labels. A production-focused deep dive.→
- Scrape Configsjob_name, scheme, targets, metrics_path, tls_config. A production-focused deep dive.→
- Rule Filesgroups, recording rules, alerting rules. A production-focused deep dive.→
- Alerting Sectionalertmanagers, alert_relabel_configs. A production-focused deep dive.→
- Remote Write SectionThe full anatomy of the remote_write block: url, remote_timeout, headers, write_relabel_configs, queue_config tuning, metadata_config, and multi-destination patterns.→
- Configuration ValidationThe full validation pipeline: promtool check config, promtool check rules, promtool test rules, --dry-run, CI integration, and the reload-path test pattern.→
Part VIII
Service Discovery
6 checks
- Static Discoverystatic_configs. A production-focused deep dive.→
- File-Based DiscoveryJSON / YAML file_sd_configs. A production-focused deep dive.→
- DNS-Based Discoverydns_sd_configs and SRV records. A production-focused deep dive.→
- Container DiscoveryDocker, Kubernetes, container_sd_configs. A production-focused deep dive.→
- Relabeling and Target Managementaction, regex, source_labels, target_label. A production-focused deep dive.→
- Metric RelabelingFiltering, dropping, modifying samples at ingestion. A production-focused deep dive.→
Part IX
Exporters
6 checks
- The Exporter ContractWhat an exporter is and what its metric exposition looks like. A production-focused deep dive.→
- Exporter TypesHost, application, database, queue exporters. A production-focused deep dive.→
- Community Exporter TrustProject health, security, version pinning. A production-focused deep dive.→
- Writing an ExporterWhen to write one and when not. A production-focused deep dive.→
- Exporter PerformanceCost, scraping, parallelism, failure. A production-focused deep dive.→
- Exporter SecurityListening interface, TLS, basic auth. A production-focused deep dive.→
Part X
node_exporter
6 checks
- node_exporter OverviewCollectors, default-enabled, collector selection. A production-focused deep dive.→
- CPU Metricsnode_cpu_seconds_total, modes, multi-CPU. A production-focused deep dive.→
- Memory MetricsMemTotal, MemAvailable, buffers, cache. A production-focused deep dive.→
- Filesystem Metricsinodes, size, free, mountinfo. A production-focused deep dive.→
- Disk Metricsnode_disk_*, read/write/IOPS, queues. A production-focused deep dive.→
- Network Metricsnode_network_*, bytes, packets, errors, drops. A production-focused deep dive.→
Part XI
Blackbox Monitoring
6 checks
- blackbox_exporter OverviewProbes, modules, target parameters. A production-focused deep dive.→
- TCP Probestcp_connect, IP family fallback, payload matching. A production-focused deep dive.→
- ICMP Probesicmp module, NET_RAW capability, latency versus packet loss. A production-focused deep dive.→
- DNS Probesdns module, query name, query_type, validate_answer_rrs. A production-focused deep dive.→
- Blackbox Incident RunbookA probe starts failing for several targets. Isolate service, route, and configuration.→
- TLS Certificate Probesprobe_ssl_earliest_cert_expiry, Alertmanager thresholds, ACME and manual renewal. A production-focused deep dive.→
Part XII
PromQL Foundations
6 checks
- PromQL AnatomyExpression types, the evaluation pipeline, and where the query engine fits in the Prometheus server.→
- Instant and Range VectorsWhat an instant vector is, what a range vector is, and why [5m] only appears inside functions.→
- Label Matching OperatorsThe four operators = != =~ !~ and the difference between matching a label name and matching a label value.→
- Binary OperatorsArithmetic, comparison, set operations, and the bool modifier on comparisons.→
- Vector Matchingon(), ignoring(), group_left(), group_right() and the one-to-one, one-to-many and many-to-many cases.→
- Functions and String Operationsabs, ceil, floor, deriv, topk, bottomk, label_replace, label_join, vector, sort.→
Part XIII
Rates and Counters
6 checks
- rate() and Counter ResetsHow rate() detects counter resets and extrapolates over partial windows.→
- irate() and increase()When to use irate vs rate; increase for totals. A production-focused deep dive.→
- Why Counters Aren't Plotted Directly'Resets, scale, comparison over time.→
- Counter Monotonicity in PracticeDetecting monotonicity violations. A production-focused deep dive.→
- Rate Window Selection5m, 1m, 30s — when does the choice matter. A production-focused deep dive.→
- derivative() for Gaugesrate of change for non-monotonic metrics. A production-focused deep dive.→
Part XIV
Aggregation
6 checks
- sum() and avg()The two operators most dashboards use. A production-focused deep dive.→
- min(), max(), count()Distribution operators. A production-focused deep dive.→
- topk() and bottomk()The rank operators. A production-focused deep dive.→
- by() and without()The grouping modifiers. A production-focused deep dive.→
- Quantile AggregationComputing fleet-wide quantiles correctly. A production-focused deep dive.→
- Over-Aggregation RisksWhen dropping labels hides the incident. A production-focused deep dive.→
Part XV
Histograms and Latency
6 checks
- Histogram Bucketsle="..." — understanding the bucket layout. A production-focused deep dive.→
- histogram_quantile()p50, p95, p99 — what works, what does not. A production-focused deep dive.→
- Cumulative Bucketsle="..." = "observations less than X." A production-focused deep dive.→
- _sum and _countAverage latency, rate of observations. A production-focused deep dive.→
- Native HistogramsPrometheus 2.50+: sparse histograms, exact quantiles. A production-focused deep dive.→
- Choosing the Bucket LayoutDefault buckets, ExponentialBuckets, SLO-driven. A production-focused deep dive.→
Part XVI
PromQL Troubleshooting
6 checks
- Common PromQL MistakesThe seven anti-patterns that produce wrong alerts and dashboards in Prometheus.→
- Stale SeriesDetecting that a target has stopped scraping, and why hidden series still consume TSDB resources.→
- absent() and Missing DataDistinguishing "no sample" from "value 0" and alerting on missing metrics reliably.→
- Cardinality Explosion in QueriesPatterns that explode query output and how to diagnose query-path latency with the engine metrics.→
- Vector Matching MistakesBinary operators, many-to-many, group_left and label collisions in production queries.→
- PromQL Performance TipsRecording rules, lookback windows, subqueries, and the engine flags that bound the cost of a query.→
Part XVII
Recording Rules
6 checks
- Recording Rule PurposeWhy expensive or repeated PromQL becomes a recording rule. A production-focused deep dive.→
- Naming Recording RulesConventions for what to name a recording rule. A production-focused deep dive.→
- Rule EvaluationWhen rules evaluate, what happens on failure. A production-focused deep dive.→
- Rule PerformanceCost of rules, when rules become expensive. A production-focused deep dive.→
- Rule OrganisationFiles, groups, comments, naming. A production-focused deep dive.→
- Recording Rules for SLOsThe specific role rules play in SLO computation. A production-focused deep dive.→
Part XVIII
Alerting Rules
6 checks
- The Alert Rule Anatomyexpr, for, keep_firing_for, labels, annotations. A production-focused deep dive.→
- Severity and Routing LabelsHow labels carry routing decisions. A production-focused deep dive.→
- `for:` and HysteresisHow `for:` reduces flapping. A production-focused deep dive.→
- Annotations: Runbook, Dashboard, DescriptionThe human-facing context that comes with the alert. A production-focused deep dive.→
- Common Rule PatternsBlackbox probe failure, scrape failure, missing metric, error rate, latency breach. A production-focused deep dive.→
- Rule Lint and ReviewHow rules are reviewed before production. A production-focused deep dive.→
Part XIX
Alertmanager
6 checks
- Alertmanager AnatomyThe dispatch loop from Prometheus to the on-call channel, the storage layer that survives restarts, and the tools used to inspect it.→
- The Routing TreeHow routes are organised: matchers, child routes, group timing, and the design choices that decide who gets paged for what.→
- Grouping of AlertsHow Alertmanager collapses N firing alerts into one notification. The trade-off between correlation and alert fatigue.→
- InhibitionSuppression of dependent alerts. The rules, the matcher design, and the gotchas that hide real incidents.→
- Silences and MutesTime-bounded mute of an alert. The matchers, the audit trail, the API, and the cost of "we will silence this forever".→
- Receivers and TemplatesEmail, Slack, PagerDuty, Opsgenie, webhooks. Integration URLs, routing keys, and the limits of Go templating.→
Part XX
Alert Quality
6 checks
- Good Alerts, Bad AlertsWhat makes an alert useful. A production-focused deep dive.→
- Symptoms vs CausesWhy alerting on CPU is wrong; alerting on latency is right. A production-focused deep dive.→
- The Page vs Ticket DecisionWhen to wake someone up. A production-focused deep dive.→
- Alert FatigueThe failure shape that breaks the system. A production-focused deep dive.→
- Actionable AlertsWhat every page-worthy alert must include. A production-focused deep dive.→
- Alert Quality ReviewThe quarterly audit. A production-focused deep dive.→
Part XXI
Alert Inhibition
6 checks
- Why Inhibition ExistsSuppressing consequence alerts. A production-focused deep dive.→
- Basic Inhibitionhost down -> service down. A production-focused deep dive.→
- Dependency-Aware AlertingMapping service dependencies. A production-focused deep dive.→
- Inhibit by ClusterReaching across the platform. A production-focused deep dive.→
- Inhibit, then PageCoordinating with severity. A production-focused deep dive.→
- Inhibit Rule ReviewWhen the dependency map changes. A production-focused deep dive.→
Part XXII
SLO-Based Alerting
6 checks
- Burn Rate 101The math behind error budget consumption, the canonical page-worthy thresholds, and the sloth helper library.→
- Multi-Window BurningThe four-window, four-burn-rate pattern that catches both sharp spikes and slow cumulative burns.→
- The Error Budget PolicyWhat the team actually does with the budget — freeze, slow down, document, and review.→
- SLO Alert TemplatesThe full package — recording rules, alert rules, and dashboard panels for one SLO.→
- SLO Alerting RisksWhere the indicator and the reality diverge — and what to do about it.→
- SLO IterationThe discipline of starting, running, learning from, and refining an SLO over months and years.→
Part XXIII
Grafana Foundations
6 checks
- Grafana AnatomyServer, users, orgs, datasources, dashboards. A production-focused deep dive.→
- DatasourcesPrometheus, Loki, Tempo as Grafana datasources. A production-focused deep dive.→
- Panels and QueriesTime series, stat, table, bar gauge, text. A production-focused deep dive.→
- Organisations and TeamsTenant model, teams, roles, folders, permissions. A production-focused deep dive.→
- Alerting in GrafanaUnified alerting, contact points, notification policies, and the boundary with Prometheus alerting. A production-focused deep dive.→
- Provisioning ModelDatasources, dashboards, alerting, and plugins as code. A production-focused deep dive.→
Part XXIV
Grafana Installation
6 checks
- Grafana Installation MethodsTarball, apt, rpm, and Docker. The four supported install paths for Grafana 11.x, with the upgrade, isolation, and backup consequences of each.→
- Storage and SQLiteWhere Grafana stores dashboards, users, and audit logs. The default sqlite store, the migration to MySQL or PostgreSQL for high availability, and the sqlite write-rate gotcha.→
- systemd and Process ManagementThe grafana-server service. A hardened systemd unit, the Ready / Serving lifecycle states, resource limits, and the start-time ordering against the data source and database.→
- Reverse Proxy and TLSPutting Grafana behind nginx. The reverse proxy terminates TLS, rate-limits per IP, and forwards WebSocket upgrade headers for Grafana Live. The trusted_proxies and X-Forwarded-Proto settings that close the spoofable-header class of attack.→
- Initial Admin and AuthenticationThe first admin user, why anonymous admin is a security failure, the GF_SECURITY_ADMIN_PASSWORD environment variable, and the discipline that locks the admin password out of git and into a secret store.→
- Installation ValidationConfirm Grafana is up, the database is reachable, the data sources answer, the dashboard load test passes, and the reverse proxy is the public entry point. The shape of "valid Grafana" in production terms.→
Part XXV
Grafana Data Sources
6 checks
- Prometheus as a Data SourceHow Grafana 11.x speaks to Prometheus over the data-source proxy, what the provisioning YAML controls, and the scrape health check that catches credential, TLS and timeout regressions before a panel goes red.→
- Loki as a Data SourceHow Grafana 11.x speaks to Loki, what the provisioning YAML controls, the maxLines and maxSeries caps that protect the UI from runaway queries, and the derived-fields links that turn a log line into an investigation pivot.→
- Tempo as a Data SourceHow Grafana 11.x speaks to Tempo, what the provisioning YAML controls, the trace-to-logs pivot that ties a span to its log lines, the service map that derives the dependency graph from span metrics, and the search-recent-traces query path.→
- Tags and FilteringHow Grafana data source tags attach metadata to a source, how BuiltIn and legacy naming affect Grafana\'s automatic behaviour, and how the UID governs the data source proxy URLs that dashboards and Explore depend on.→
- Datasource Secret RotationThe lifecycle of a Grafana data source credential, the secret_key that protects secureJsonData, the in-memory copy that survives a rotation, and the impact of a credential change on running dashboards and active sessions.→
- Rotating Datasource CredentialsThe end-to-end rotation procedure: pulling a new credential from Vault or a sealed secret, swapping the data source UID or updating in place, the staging-then-production pipeline, the audit trail that proves the rotation happened, and the rollback path that recovers when the rotation fails.→
Part XXVI
Dashboard Design
6 checks
- Dashboard HierarchyService overview vs service detail. A production-focused deep dive.→
- Units and ThresholdsConsistency across dashboards. A production-focused deep dive.→
- Overview DashboardsThe fleet-state panel set. A production-focused deep dive.→
- Service DashboardsThe per-service operating page. A production-focused deep dive.→
- Links and AnnotationsDrilling from one dashboard to another. A production-focused deep dive.→
- Dashboard OwnershipWho owns what. A production-focused deep dive.→
Part XXVII
Dashboard Anti-Patterns
6 checks
- Wallpaper DashboardsWhy a 40-panel dashboard of all green panels is a liability, not an asset, and how to retire it.→
- Meaningless GaugesCounters, percentages and absolute numbers shown without context, and how they train operators to ignore real signals.→
- Rainbow Color PalettesWhy a rainbow-coloured palette trains the eye to ignore thresholds, and the discipline that replaces it.→
- No Context and No OwnershipWhy a panel without a service name, a runbook and a team is operationally unactable, and the discipline that fixes it.→
- Inconsistent UnitsWhy mixing milliseconds and seconds, bytes and megabytes, or requests per minute and per second on the same dashboard is a bug, and how to fix it.→
- No Troubleshooting LinkWhy a panel that turns red but offers no path forward is the worst kind of decoration, and how to wire runbooks, drill-downs and panel-level narrative.→
Part XXVIII
Grafana Variables
6 checks
- Template Variable PurposeA runtime parameter that re-binds every panel query; how URL state, chaining, and curated versus ad-hoc variables shape a production dashboard.→
- Variable TypesQuery, custom, constant, datasource, interval, textbox: the seven variable kinds Grafana 11 offers and what each is good for.→
- Variable InterpolationHow $var and [[var]] enter queries, titles, and URLs; what the two syntaxes buy you and where each one breaks.→
- Common Variable PatternsA canonical $cluster, $namespace, $service layout; how regex exclusion works, and why a regex that excludes a label also excludes the All-value.→
- Variable Query CostThe N-by-M query blast, the regex refresh cost, the chained-variable storm, and the variable-cookie state that survives a reload.→
- Repeated Panels and RowsWhen one panel becomes many: the repeat-by-variable pattern, the row-with-variable layout, and the cost of repeated panels on every refresh.→
Part XXIX
Grafana Provisioning
6 checks
- Provisioning AnatomyThe /etc/grafana/provisioning/ folder, the seven providers, the polling loader, and the coupling between data sources, dashboards, and alerts.→
- Provisioning DatasourcesThe datasources YAML, the secure JSON field, the API-vs-file trade-off, and the secrets insertion at provisioning time.→
- Provisioning DashboardsThe dashboards YAML, the provider block, the UID, the folder pattern, and the deletion-when-removed-from-yaml semantic.→
- Sidecar ProvisioningA synced filesystem as the source of truth, the sidecar pattern, and the discipline of git or S3 as the upstream.→
- Testing Provisioning LocallyA disposable Grafana with the same provisioning, a synthetic data source, and a sandbox that exercises the loaders before production.→
- Drift PreventionThe difference between UI edits and git edits, the policy of edit-only-in-code, the audit trail, and the role of code review.→
Part XXX
Grafana Security
6 checks
- Authentication and UsersHow Grafana authenticates. A production-focused deep dive.→
- RBAC and TeamsRole-based access. A production-focused deep dive.→
- Anonymous AccessWhen it is appropriate and when it is not. A production-focused deep dive.→
- Data Source CredentialsSecrets in datasources. A production-focused deep dive.→
- Secrets and Vault IntegrationVault as a credential source. A production-focused deep dive.→
- Plugins and TLSThird-party plugins. A production-focused deep dive.→
Part XXXI
Logging Foundations
6 checks
- Logs vs Structured LogsThe difference between unstructured and structured logs. A production-focused deep dive.→
- Log Levels and SeveritySeverity levels, when to use which. A production-focused deep dive.→
- Timestamps and Time ZonesLog timing discipline. A production-focused deep dive.→
- Correlation IDs in LogsTrace IDs across services. A production-focused deep dive.→
- PII and Secrets in LogsWhy logs should be inspected for sensitive content. A production-focused deep dive.→
- Machine-Readable LogsJSON / logfmt / structured metadata. A production-focused deep dive.→
Part XXXII
Logging Pipeline Architecture
6 checks
- The Logging PipelineHosts to agent to transport to Loki to query. A production-focused deep dive.→
- Agents vs SidecarsProcess to ship logs. A production-focused deep dive.→
- Grafana Alloy vs OpenTelemetry CollectorTwo strategic collectors. A production-focused deep dive.→
- Promtail in Maintenance ModeWhy Promtail is no longer the default. A production-focused deep dive.→
- Pipeline ResilienceBackpressure, batching, retries. A production-focused deep dive.→
- Pipeline SecurityAuthentication, TLS, redaction. A production-focused deep dive.→
Part XXXIII
Loki Architecture
6 checks
- Loki Architecture OverviewThe seven Loki components, the simple-scalable and microservices deployment modes, the wiring between roles, and the order in which to scale them.→
- StreamsA Loki stream is a unique label set with continuous log lines. The stream lifecycle, the per-tenant limits, and the cost of high-cardinality labels.→
- ChunksTime-bounded compressed bundles of log lines. The head block, the flush triggers, the chunk_target_size and 1.5x rule, and the WAL that survives a restart.→
- IndexesHow Loki maps a label selector to chunks. The boltdb-shipper and TSDB index stores, the per-tenant isolation, and the schema_config that pins which store Loki reads.→
- Ingester LifecycleHow ingesters are stateful, how the replication factor fans pushes out, how the ingester ring assigns streams to replicas, and how the handoff moves ownership on shutdown.→
- CompactorThe singleton that enforces retention and compacts the TSDB index. The writes-only path, the per-tenant cycle, the marker files, and the operational discipline of running exactly one compactor.→
Part XXXIV
Loki Labels and Cardinality
6 checks
- The Loki Label RuleOnly low-cardinality labels. A production-focused deep dive.→
- Good Loki Labelsapplication, env, host, namespace. A production-focused deep dive.→
- Bad Loki Labelsrequest_id, user_id, session_id. A production-focused deep dive.→
- Data in Labels vs Data in Log ContentWhen to use structured metadata. A production-focused deep dive.→
- Loki Cardinality IncidentMemory spikes from a new label. A production-focused deep dive.→
- Loki Label AuditQuarterly review of label design. A production-focused deep dive.→
Part XXXV
Loki Installation
6 checks
- Loki Installation ModesSingle binary vs simple scalable vs microservices. Trade-offs, operational cost, and the choice that decides how you will spend the next three years.→
- Loki ConfigurationThe schema_config block, the common keys, and the per-component sections. How the loki.yaml files map to subsystems and what every key actually changes.→
- Loki StorageThe role of the object store, the bucket layout, the per-tenant folder structure, and the cost of every API call Loki makes.→
- Loki Retention ConfigurationThe retention_period, retention_enabled, the compactor sweep, the stream retention mode, and the per-tenant override path. Why a wrong retention value is a compliance event.→
- Loki Ingestion LimitsThe limits_config rules that protect Loki from runaway ingestion. Per-stream rate, per-tenant global rate, sample age, and what the client sees when a limit fires.→
- Loki ValidationThe readiness endpoints, the synthetic push, logcli, and the smoke test that proves a Loki deployment is healthy before the application fleet is pointed at it.→
Part XXXVI
Log Shipping
6 checks
- Grafana Alloy OverviewThe strategic Grafana collector: River components, the module library, the Grafana-managed config surface, and the boundary with the OpenTelemetry Collector.→
- Alloy ConfigurationRiver syntax, the component graph, and the loki.source.file to loki.process to loki.write pipeline. Stage-by-stage transforms with annotated examples.→
- OpenTelemetry Collector OverviewThe vendor-neutral collector. Receiver, processor, exporter architecture; the agent and gateway deployment modes; the contrib and core distributions.→
- OTel Collector Configurationotelcol.yaml. Receivers (otlp, filelog, journald), processors (batch, resource, attributes), exporters (loki, otlp, otlphttp), and the service.pipelines wiring.→
- Choosing the Right CollectorPer-host agent, sidecar, or central gateway. The trade-offs, the resource cost of the on-host approach, and the role of the central pipeline in a multi-tenant platform.→
- Collector SecurityTLS to Loki, secrets management, mTLS between collectors, redaction at the source, and the audit trail. The threat model and the configuration that addresses it.→
Part XXXVII
LogQL Foundations
6 checks
- LogQL Stream SelectorsThe label matchers that decide what Loki will look at. Equality, regex, negation, and the cost of an unbounded selector.→
- LogQL Line Filters|=, !=, |~, !~. The full-text filters that run after the selector. Substring matches, regex, and the cost of scanning chunks.→
- LogQL Parsers| json, | logfmt, | regexp. Turning log content into extracted fields that the rest of the query can filter, group, and aggregate.→
- LogQL Aggregationssum, count, count_over_time, avg, max, quantile_over_time. Reducing parsed values to a metric, with by clauses and unwrap.→
- Structured Logs in GrafanaStatistics, Fields, derived fields, auto-detected JSON, and data links. The Grafana-side discipline that turns a parsed query into a readable investigation panel.→
- LogQL Query Performancequery_parallelism, split-by-interval, results cache, index cache. Why the same query takes 800 ms in staging and 40 s in production.→
Part XXXVIII
LogQL Metrics
6 checks
- Log-Derived MetricsLoki 3.x metrics rules: deriving PromQL-friendly time series from LogQL queries, the right shape of a rule, and the cost of generating too many of them.→
- Error Rates From LogsComputing RED-method error rates from log streams with Loki 3.x: the right level filter, the per-instance attribution, the parser, and the alert shape that does not page on log-format drift.→
- Log Volume from LogsBytes per second, lines per second, indexing cost per label, and capacity-driven alerting that warns before Loki starts rejecting writes.→
- Latency from LogsWhen logs carry a duration field: the `| unwrap` approach, `histogram_quantile` over log-derived values, the cost of unwrap on a long window, and the right ceiling for substituting for native histograms.→
- Logs vs Native MetricsWhen logs can substitute for metrics and when they cannot: rate and percentage from logs, the cost model, and the cardinality ceiling that makes native metrics irreplaceable.→
- Log-Based AlertingAlerting directly on Loki ruler queries: when to do it, the right shape for a Loki alert, the cost of the query window, and the alert that catches "no logs at all".→
Part XXXIX
Log Troubleshooting
6 checks
- Missing LogsWhen Loki has no logs from a service. A production-focused deep dive.→
- Timestamp IssuesWhen logs arrive with wrong time. A production-focused deep dive.→
- Label MismatchWhen labels are right but queries return nothing. A production-focused deep dive.→
- Ingestion FailuresWhen dist_drop reasons grow. A production-focused deep dive.→
- Malformed Structured LogsWhen parsers fail. A production-focused deep dive.→
- Query PerformanceSlow queries. A production-focused deep dive.→
Part XL
Log Retention
6 checks
- Retention BasicsHow long logs are kept — the compactor, the object store, the global default, the per-tenant override, and the cost of getting it wrong.→
- Legal and ComplianceGDPR, SOC2, and PCI-DSS each impose a retention window — sometimes a minimum, sometimes a maximum. The sysadmins job is to make the policy match the regime before the auditor arrives.→
- Capacity PlanningBytes per second times retention times inverse compression is the bucket size. The sysadmins job is to predict next month, not be surprised by it.→
- Object StorageLoki stores chunks in object storage. Choosing S3, GCS, Azure Blob, or MinIO; tuning storage-class transitions; and paying only for what you actually read.→
- Retention Policies Per StreamSome streams warrant longer retention than others. The map between stream, window, and regime — and the resolution rule when policies overlap.→
- Retention AuditThe quarterly review. Bucket size, per-tenant retention report, orphan streams, and a documented legal sign-off.→
Part XLI
Distributed Tracing Foundations
6 checks
- Trace AnatomyA trace is a directed acyclic graph of spans that records the journey of a single request through a distributed system.→
- Span AttributesKey-value metadata on a span. Semantic conventions, search keys, and the cost of high cardinality.→
- Parent / Child RelationshipsHow spans nest, how context propagates between services, and where the trace breaks.→
- Trace ID and Span IDThe 128-bit trace identifier and the 64-bit span identifier. What they are, who picks them, and how Tempo uses them.→
- Span EventsTimestamped annotations on a span. Structured, attached, and bounded in cost.→
- Span StatusUNSET, OK, ERROR. The summary field, the failure shape, and the TraceQL filter that finds every failing trace.→
Part XLII
Why Tracing Exists
6 checks
- Questions Only Traces AnswerThe questions traces answer that metrics and logs answer only with great effort: where the time went, which dependency failed, where the request died.→
- Dependency Latency TracingReading the per-call cost out of a trace, the parent / child latency math, and a per-dependency SLO that survives contact with reality.→
- Traces vs Metrics CorrelationThe data-model difference between a metric and a trace, what each one is shaped to answer, and the exemplar that bridges them.→
- The Critical PathWhy the slowest span is not always the bottleneck, the difference between total duration and critical-path duration, and the right way to find the work that actually matters.→
- Trace Propagation TestVerifying the trace ID survives every hop end to end: a synthetic request with a known traceparent, the verification at each boundary, and the failure modes that break the chain.→
- Tracing Cost ConsiderationsThe storage arithmetic for traces, the cardinality cliff, the sampling strategies that keep the platform within budget, and the trade between cost and investigation depth.→
Part XLIII
Instrumentation
6 checks
- Manual vs Automatic InstrumentationWhen code-based instrumentation is the right answer, when zero-code is, and how to decide per service.→
- OpenTelemetry SDKThe OTel SDK API surface in production: TracerProvider, Tracer, Span, SpanContext, and the resource detector.→
- Auto-InstrumentationZero-code instrumentation in the OpenTelemetry ecosystem: the Java agent, the Python autoloader, the Node loader, and Go OBI.→
- Context PropagationW3C traceparent and tracestate, the propagation across HTTP, gRPC, and queues, and the failure shape of a broken boundary.→
- Attribute DesignHow to choose attributes for spans and resources, the cost of cardinality, and the role of attributes in TraceQL search.→
- SDK DeploymentBuild-time vs runtime deployment of the OpenTelemetry SDK, the resource detector, and the right approach per platform.→
Part XLIV
Sampling
6 checks
- Head vs Tail SamplingWhere the sampling decision is made. Producer-side head sampling versus gateway tail sampling, the cost trade-off, and the production trade-offs of each.→
- Sampling RateThe math of probabilistic sampling. How rate times request volume equals retained samples, and how to choose a rate per service.→
- Rare Error TracesCapturing rare errors that head sampling misses. Tail sampling policies for status codes, latency thresholds, and error logs.→
- Tail Sampling CostWhere tail sampling pays. The cost of buffering every trace at the gateway, the per-collector volume ceiling, and the trade-off between policy depth and gateway capacity.→
- Sampling Strategy Per ServiceA layered strategy. Head sampling at the producer for volume reduction, tail sampling at the gateway for policy, and the right per-service and per-environment choices.→
- Re-tuning SamplingThe iterative process. Observe the metrics, decide on the policy, change the config, validate the outcome, document the change.→
Part XLV
Tempo Architecture
6 checks
- Tempo Architecture OverviewDistributor, ingester, querier, query-frontend, compactor, metrics-generator, and the role of object storage.→
- The DistributorReceive-side validation, rate limiting, and the fan-out to ingesters via the consistent-hash ring.→
- The IngesterThe stateful write path. Trace lifecycle, head blocks, the write-ahead log, and the flush to object storage.→
- The CompactorWalking the bucket, merging blocks, and the role in trace ID compaction.→
- The QuerierThe read path. Fetching blocks from object storage, executing TraceQL, and the role of the query-frontend.→
- The Metrics GeneratorDeriving RED metrics and service graphs from spans, and the role in trace-to-metrics correlation.→
Part XLVI
Tempo Deployment
6 checks
- Tempo Deployment ModesSingle binary, simple scalable, and microservices. When to choose each, and what each costs to operate.→
- Tempo ReceiversOTLP, Jaeger, and Zipkin receivers. Protocols, ports, TLS, and how to land traces from any client.→
- Tempo StorageObject storage selection, bucket layout, and per-tenant folder structure for Tempo blocks.→
- Tempo Retentionblock_retention, the compactor, per-tenant overrides, and the storage-cost trade-off.→
- Tempo QueryingTraceQL search and lookup, the query-frontend role, and the metrics-generator.→
- Tempo Validationtempo-cli validate-config, readiness probes, synthetic trace push, and the smoke test.→
Part XLVII
Trace Queries
6 checks
- TraceQL IntroductionThe TraceQL query language, its role in trace investigation, and the shape of a query.→
- TraceQL SelectorsFiltering traces by span attributes, resource attributes, and intrinsic fields. Comparison operators and value types.→
- TraceQL AggregationsCounting, summing, and grouping traces with the count aggregator and the by clause.→
- TraceQL IntrinsicsStatus, name, duration, IDs, and structural fields: the canonical, indexed fields on every span and trace.→
- Pivot from Metrics to TracesHow exemplars link a metric datapoint to a single trace, and the operational workflow for moving between signals.→
- Trace Exploration WorkflowsThe disciplined flow from a metric spike to a root cause span. Pivots, drill-downs, and service maps.→
Part XLVIII
Trace Troubleshooting
6 checks
- Missing SpansWhen a trace has fewer spans than expected. The diagnostic ladder and the most common cause.→
- Broken Trace PropagationWhen the trace_id changes mid-request. The W3C traceparent header, the async boundary, and the most common cause.→
- Sampling IssuesWhen a useful trace never reaches Tempo. Tail sampling, parent-based consistency, and the head sampling interaction.→
- Clock Problems in TracesWhen spans appear before their parents. NTP discipline, monotonic time, and wall-clock skew.→
- Collector FailuresWhen spans reach the collector but never reach Tempo. The exporter queue, the retry, and the cost of overflow.→
- High-Cardinality AttributesWhen span attributes explode cost. The index cost, the TraceQL search cost, and the cardinality budget.→
Part XLIX
OpenTelemetry Foundations
6 checks
- OpenTelemetry OverviewThe CNCF project: the specification, the SDKs, the Collector, and the four signals they describe.→
- SignalsMetrics, logs, traces, and profiles: the four signal types, the resource model, and the semantic conventions that make them correlate.→
- SDKs by LanguageThe OpenTelemetry SDK languages, the maturity of each, and the right instrumentation choice per stack.→
- OTLPThe OpenTelemetry wire protocol: gRPC and HTTP/protobuf, payload size, compression, framing, and TLS.→
- Collector vs Agent PatternsAgent mode (per-host) vs gateway mode (central): the OTLP topology, the failure domains, and the right choice per fleet.→
- OpenTelemetry Today and TomorrowThe OTel stability matrix, the experimental signals (profiles, events), the migration paths, and the role of the specification.→
Part L
OpenTelemetry Collector
6 checks
- Collector AnatomyReceivers, processors, exporters, pipelines, extensions. The component lifecycle, the data flow, and the role of each block in otelcol.yaml.→
- ReceiversHow the Collector receives data. otlp, jaeger, zipkin, prometheus, hostmetrics, filelog, journald, k8sobjects. Choosing the right receiver per source.→
- Processorsbatch, attributes, resource, memory_limiter, resourcedetection, tail_sampling, transform. The chain order, the cost of each processor, and the role of memory_limiter.→
- ExportersWhere the Collector sends data. otlp, otlphttp, prometheus, loki, tempo, file, debug. The per-exporter config and the right exporter per backend.→
- PipelinesHow receivers connect to exporters. The per-signal pipelines, the order of processors, and the role of the connectors.→
- Collector Deployment PatternsAgent and gateway. The systemd unit, the Docker container, the docker-compose pattern, the Kubernetes DaemonSet, and the readiness probe.→
Part LI
Correlating Metrics, Logs, and Traces
6 checks
- Correlation AnatomyThe join key across signals. A production-focused deep dive.→
- Metric to Log WorkflowPivot from a metric datapoint to logs. A production-focused deep dive.→
- Metric to Trace WorkflowPivot from a histogram to a trace. A production-focused deep dive.→
- Log to Trace WorkflowPivot from logs to traces. A production-focused deep dive.→
- Grafana CorrelationsConfiguring datasource correlations. A production-focused deep dive.→
- Correlation TestVerifying the workflow. A production-focused deep dive.→
Part LII
Exemplars
6 checks
- Exemplars OverviewLinking metric datapoints to traces. A production-focused deep dive.→
- Exemplar FormatThe OpenMetrics exemplar suffix on a histogram bucket. A production-focused deep dive.→
- Exemplar EmissionWhere the exemplar comes from at observation time. A production-focused deep dive.→
- Grafana Exemplar LinkingThe Grafana panel that surfaces exemplars and the click-through to the trace. A production-focused deep dive.→
- Exemplars CostThe wire, appender, and trace-backend cost of exemplars. A production-focused deep dive.→
- Exemplar TroubleshootingDiagnostic order for missing exemplars. A production-focused deep dive.→
Part LIII
Log / Trace Correlation
6 checks
- Log / Trace LinkThe trace_id field in structured logs. A production-focused deep dive.→
- Service Metadata in LogsThe resource attributes service.name, deployment.environment, and the cross-signal join key they form.→
- LogQL Pivot to TraceThe Grafana data link that takes a Loki line, pulls the trace_id field, and opens the matching Tempo trace in one click.→
- Tempo Derived FieldsThe Tempo data source derivedFields block: extracting attribute values from spans and turning them into links to Loki.→
- Correlation CostWhat pivot queries cost Loki, what correlated telemetry costs Tempo and Loki storage, and the four knobs that keep the bill bounded.→
- End-to-End Correlation TestThe synthetic test that exercises every pivot — application log format, structured metadata promotion, derived fields, Tempo resolution — and fails the deployment when the chain breaks.→
Part LIV
Dashboard-to-Logs Workflows
6 checks
- Dashboard-to-Logs AnatomyHow a Grafana panel becomes a Loki query. The data link, the URL template, and the right shape for a one-click pivot from metric to log.→
- Correlation ConfigurationAuthoring the data link in Grafana 11.x: the title, the url, the target, the URL template variables, and the link that lands in the right Loki data source.→
- Label Passing Through the PivotHow a metric series label becomes a Loki stream selector. The template syntax, the label-name alignment between Prometheus and Loki, and the right approach per label.→
- Filtered Stream PivotFrom a metric point to a Loki stream. The time window, the label filter, the pipe filter, and the shape of the resulting LogQL.→
- Severity and Context PreservationMapping a metric panel failure mode to log severity, preserving the context fields the operator needs, and the right discipline for the resulting LogQL.→
- Avoiding Pivot SpamThe cost of a noisy pivot. The line-count target, the discipline of limiting the pivot, and the four shapes of pivot spam that drive engineers to ignore the link.→
Part LV
Dashboard-to-Traces Workflows
6 checks
- Dashboard-to-Traces OverviewThe metrics to traces pivot: when a metric says something is wrong, how a single click drops the engineer into the trace that proves it.→
- Exemplar Link to TraceThe Grafana data source wiring that turns a histogram diamond into a navigation event: how the Internal link is configured, what data source UID it needs, and what each name on the wire maps to.→
- Latency Pivot WorkflowFrom a tall p99 bar to the slow trace: the histogram that holds the exemplar, the panel options that expose the diamond, and the click sequence that lands the operator on the span that produced the slow observation.→
- Error Pivot WorkflowFrom a rising error rate to the failing trace: the metric that counts failures, the histogram that holds a failed observation, and the trace that explains why the request returned a 5xx.→
- Dependency AttributionFrom a tall request-latency bar to the dependency that produced it: reading the trace timeline, attributing the duration to a specific span, and finding the slow downstream.→
- Trace ExplorationFrom one trace to many: the TraceQL patterns that surface related traces, the panel queries that turn a single exemplars click into a population, and the discipline of exploring without flooding Tempo.→
Part LVI
Linux Observability
6 checks
- Linux Metrics OverviewThe node_exporter baseline. The USE method, the four resource classes, and the right discipline per host class.→
- Memory ObservabilityMemAvailable, swap, OOM kills, and PSI. The right metrics for diagnosing memory pressure in production.→
- Disk Latency ObservabilityI/O latency, queue depth, throughput, and the SSD vs HDD difference. Per-device filtering for production fleets.→
- Network ObservabilityThroughput, errors, drops, and carrier. The right interface filter for production hosts and the failure shape of the silent interface.→
- Service Availabilitysystemd unit state, health checks, and the watchdog. The right approach to pages fire, services stay up.→
- Time Sync Observabilitychrony, systemd-timesyncd, and the time skew impact. The chronyc tracking output and the most common failure shape.→
Part LVII
Docker Observability
6 checks
- Docker Host MetricsNode-level metrics. A production-focused deep dive.→
- cAdvisor OverviewContainer metrics engine. A production-focused deep dive.→
- cAdvisor Metricscontainer_* metrics. A production-focused deep dive.→
- Short-Lived Container MetricsWhen containers live for seconds. A production-focused deep dive.→
- Docker LogsContainer stdout / stderr. A production-focused deep dive.→
- Docker TracesAuto-instrumentation in containers. A production-focused deep dive.→
Part LVIII
Proxmox Observability
6 checks
- Proxmox APIs and Metricspvestatd / pveproxy metrics. A production-focused deep dive.→
- The pve_exporterPrometheus exporter for Proxmox. A production-focused deep dive.→
- Cluster Health Metricscorosync status, quorum. A production-focused deep dive.→
- VM Resource MetricsVM CPU, memory, disk. A production-focused deep dive.→
- Storage MetricsZFS / Ceph / NFS metrics. A production-focused deep dive.→
- Backup Metrics (PBS)PBS metrics. A production-focused deep dive.→
Part LIX
Database Observability
6 checks
- Database MetricsConnections, queries, latency, locks, replication and buffers. The metric surface a production database exposes, and how to bring it into Prometheus the right way.→
- Connection PoolActive, idle and waiting connections. What saturation looks like, why it predicts a queue, and how to expose the database pool and the application pool side by side.→
- Query Latencyp95 and p99 query latency by normalised query text. The shape of slow queries, how PostgreSQL pg_stat_statements and MySQL performance_schema expose it, and how to alert before users feel it.→
- Locks and ContentionLock waits, deadlocks, row vs table vs advisory locks. Why contention is the silent second-place cause of database slowness, and the metrics that catch it.→
- Replication LagReplica health, replication lag in seconds and in transactions, fail-over readiness and the metrics that catch a drifting replica before read-after-write promises break.→
- Caching (Buffer, Redis)Hit ratio and evictions in the buffer pool, the page cache, and the application cache. Why the leading indicator of a slow database is a falling cache hit ratio, not a rising query latency.→
Part LX
Network Observability
6 checks
- Network MetricsHost network metrics. A production-focused deep dive.→
- TCP ObservabilityRetransmits, RTT, and the kernel counters that catch application-side backpressure.→
- SNMP for Network Devicesswitches, routers, firewalls. A production-focused deep dive.→
- DNS MonitoringQuery success, latency, and the distinction between authoritative and recursive health.→
- Network ProbesSynthetic ICMP and TCP probes, their topology, and the failure shapes they catch.→
- Firewall ObservabilityDrop counts, rule counts, and conntrack saturation as Prometheus metrics.→
Part LXI
Application Observability
6 checks
- Application REDRate, Errors, Duration. A production-focused deep dive.→
- Application USEResource use per container / process. A production-focused deep dive.→
- Dependency PerformancePer-dependency latency budget. A production-focused deep dive.→
- Saturation from ApplicationQueue depth, in-flight requests, and pool exhaustion. A production-focused deep dive.→
- Custom Metrics ResponsiblyWhen to add custom metrics, and how to keep them bounded. A production-focused deep dive.→
- Application SLOsPair RED with availability and latency SLOs. A production-focused deep dive.→
Part LXII
Business Metrics
6 checks
- Business vs Observability MetricsWhy technical observability is not a substitute for business metrics, and where the two meet on the same platform.→
- User Journey MetricsFunnel metrics, conversion rates, and the per-stage cardinal counters that make the business legible.→
- Revenue Impact MetricsThe arithmetic that ties a Prometheus counter to a number a finance team can spend.→
- A/B Test MetricsWhy Prometheus is the wrong tool for the analysis and the right tool for the count.→
- Business AlertingWhen a business counter should page, when it should open a ticket, and when it should not be an alert at all.→
- Business DashboardsWhy a business dashboard without an owner is decoration, and what an owned dashboard looks like.→
Part LXIII
Synthetic Monitoring
6 checks
- Synthetic Monitoring OverviewExternal probes versus real-user telemetry, where blackbox_exporter fits, and how to choose the right mix.→
- HTTP ProbesThe http_2xx module: status codes, body validation, redirects, and probe interval.→
- TLS Probesprobe_ssl_earliest_cert_expiry, chain validation, and the canonical 30 / 7 / 1 day alert thresholds.→
- DNS ProbesThe dns module, dns_query_response_time_seconds, and the right DNS server to probe.→
- Multi-Step Journey ProbesScripted end-to-end user flows, the role of k6 and Playwright, and the right cadence.→
- Synthetic Cost and Where to RunExternal probes cost more than internal; right-size the suite, pick the cadence deliberately.→
Part LXIV
TLS Monitoring
6 checks
- TLS Expiry as a Metricprobe_ssl_earliest_cert_expiry. A production-focused deep dive.→
- TLS Expiry Alerting30 days, 7 days, 1 day thresholds. A production-focused deep dive.→
- TLS Handshake FailureWhen handshake fails. A production-focused deep dive.→
- TLS Protocol HealthForcing modern TLS. A production-focused deep dive.→
- Mixed Content DetectionHTTPS resources loaded over HTTP. A production-focused deep dive.→
- TLS Incident ResponseWhen an expiry fires. A production-focused deep dive.→
Part LXV
DNS Monitoring
6 checks
- DNS Resolution Metricdig-style probes. A production-focused deep dive.→
- External DNS MonitoringResolving authoritative servers. A production-focused deep dive.→
- Internal DNS MonitoringResolving cluster-internal names. A production-focused deep dive.→
- DNS Record ValidationMatching expected values. A production-focused deep dive.→
- Slow DNS DetectionA spike in resolution latency. A production-focused deep dive.→
- DNS Incident PatternsWhen to investigate DNS. A production-focused deep dive.→
Part LXVI
Observability Architecture for Production
6 checks
- The Production Observability TopologyWorkloads to collectors to backends to users. A production-focused deep dive.→
- The Collector StrategyWhere collectors live. A production-focused deep dive.→
- Storage ArchitectureLocal disks, object storage, retention. A production-focused deep dive.→
- TLS and AuthenticationSecuring the topology. A production-focused deep dive.→
- Tenants and EnvironmentsProduction, staging, dev. A production-focused deep dive.→
- Reference ArchitectureThe canonical production reference. A production-focused deep dive.→
Part LXVII
High Availability
6 checks
- High Availability BasicsStateless vs stateful components. A production-focused deep dive.→
- Stateless ReplicasAlertmanager, Grafana, query tier. A production-focused deep dive.→
- Stateful HAIngester WAL, store-gateway, replication factor. A production-focused deep dive.→
- Two Instances Is Not HAThe parallel-silo failure shape. A production-focused deep dive.→
- HA EvidenceProving the system survives a single-node failure. A production-focused deep dive.→
- HA Cost Trade-offsWhen the second replica earns its keep. A production-focused deep dive.→
Part LXVIII
Prometheus HA
6 checks
- Duplicate ScrapingTwo Prometheus servers scraping the same targets. The duplicated series, the cost, and the right dedup.→
- External Labelsglobal.external_labels, the replica convention, and the right dedup query.→
- Alertmanager HAAlertmanager gossip cluster, the right number of nodes, and the most common failure shape.→
- Thanos and the Query LayerThanos for global Prometheus HA. Sidecar, Store, Querier, and the right query-layer dedup.→
- The Cost of HACPU, memory, disk, and scrape load doubled by a second Prometheus. The trade-off and when HA is worth it.→
- The Split-Brain ProblemWhen two Prometheus replicas both think they are primary. The fencing discipline that prevents duplicate writes.→
Part LXIX
Long-Term Metrics Storage
6 checks
- Why Long-Term Metrics StorageSingle-node Prometheus has limits. A production-focused deep dive.→
- Remote WritePrometheus to Thanos / Mimir / Cortex. A production-focused deep dive.→
- Thanos OverviewObject storage and query layer. A production-focused deep dive.→
- Mimir OverviewMulti-tenant horizontal scalability. A production-focused deep dive.→
- Cortex OverviewThe predecessor. A production-focused deep dive.→
- When to Scale MetricsTrigger criteria. A production-focused deep dive.→
Part LXX
Loki at Scale
6 checks
- Loki Microservices ModeThe enterprise deployment topology. Each component is its own deployment with its own replicas, ring, and failure domain. The mode you pick decides the next three years of operational budget.→
- Querier ScalingScaling the querier. Stateless, easy, and easy to over-spend on. The split-by-interval, the cache role, and the right number of queriers for the workload.→
- CachingThe Loki cache. Results cache, chunk cache, and the right cache shape for the workload.→
- Loki at Scale ConfigurationScaling up versus tuning the config. The decision tree for adding shards, raising parallelism, or rewriting the query shape.→
- LimitsTenant and stream limits. The per-tenant knobs that protect Loki from runaway ingestion, runaway cardinality, and runaway queries.→
- Loki Performance TroubleshootingDiagnosing slow Loki. The three locations where a performance issue lives, the diagnostic order, and the most common cause.→
Part LXXI
Tempo at Scale
6 checks
- Tempo Microservices ModeEach Tempo role as its own process: distributor, ingester, querier, query-frontend, compactor, metrics-generator, and the right scale order.→
- Tempo Ingester ScalingThe stateful bottleneck. Replication factor, the WAL, the right number of replicas, and what to watch when the ring rebalances.→
- Tempo Compactor ScalingThe singleton that walks the bucket. Per-tenant compaction, the right cadence, the cost of a stopped compactor, and when to shard.→
- Tempo Storage ScalingBucket growth, the compaction cycle, hot/warm/cold tiering, and the cost of trace retention in production.→
- Trace Load PatternsSpans per second, spikey vs steady workloads, distributor and ingester buffer sizing, and the right approach for each load shape.→
- Tempo Performance TuningA slow Tempo query. The diagnostic order, the most common cause, and the knobs that actually move query latency.→
Part LXXII
Grafana HA
6 checks
- Shared DatabaseGrafana with a shared DB. A production-focused deep dive.→
- Session ConsistencySessions across replicas. A production-focused deep dive.→
- Grafana Load BalancingRound-robin / sticky sessions. A production-focused deep dive.→
- Plugin ConsistencySame plugins on each replica. A production-focused deep dive.→
- Configuration PropagationProvisioning the same way. A production-focused deep dive.→
- Grafana HA CostWhen HA matters and when it does not. A production-focused deep dive.→
Part LXXIII
Storage Architecture
6 checks
- Storage Architecture BasicsLocal disks, network volumes, object storage. A production-focused deep dive.→
- Local Disk for TSDBPrometheus local TSDB. A production-focused deep dive.→
- Object StorageLoki and Tempo chunks. A production-focused deep dive.→
- Throughput and IOPSStorage performance. A production-focused deep dive.→
- Capacity Planning for StorageCalculating the storage budget. A production-focused deep dive.→
- Storage TieringHot vs cold storage. A production-focused deep dive.→
Part LXXIV
Capacity Planning
6 checks
- Metrics CapacityThe hard math of series, samples, memory and disk. How many active series a Prometheus can hold, how much RAM each one costs, and where the budget breaks.→
- Logs CapacityBytes per second times retention divided by compression is the bucket size. How to size Loki against the workload rather than against hope.→
- Traces CapacitySpans per second times bytes per span times retention is the bucket. How to size Tempo against the workload, not against the hope of a future sampling change.→
- Capacity HeadroomBuffer for spike. The right safety margin per tier, the metric that proves it, and the alert that fires before the platform runs out of room.→
- Growth ModellingAnticipating growth. Linear regression on the live ingest rate, the forecast for next month, and the cadence that keeps the forecast honest.→
- Avoid False PrecisionOrder-of-magnitude estimation. The methodology for sizing a new observability platform before any real traffic reaches it, and the discipline of living with the error bar.→
Part LXXV
Performance
6 checks
- Scrape OverloadWhen scrape costs exceed capacity. A production-focused deep dive.→
- TSDB CompactionPrometheus TSDB compaction. A production-focused deep dive.→
- Query PressureQueries that consume capacity. A production-focused deep dive.→
- Cardinality BottlenecksWhen cardinality is the dominant cost. A production-focused deep dive.→
- Slow Object StorageWhen object storage is the bottleneck. A production-focused deep dive.→
- OTel Collector PressureOTel Collector bottlenecks. A production-focused deep dive.→
Part LXXVI
Cost Management
6 checks
- Telemetry Cost OverviewThe cost model. A production-focused deep dive.→
- Metric Cost DriversSeries times samples times retention. A production-focused deep dive.→
- Log Cost DriversBytes/sec times retention. A production-focused deep dive.→
- Trace Cost DriversSpans/sec times sampling times retention. A production-focused deep dive.→
- Cost ControlsSampling, retention, drop. A production-focused deep dive.→
- Cost MonitoringTracking telemetry cost over time. A production-focused deep dive.→
Part LXXVII
Security Architecture
6 checks
- Security OverviewThe attack surface of the observability platform. Where each component listens, what it serves, and the right discipline for a production stack.→
- Network ExposureWhere each component listens, the default binds, and the right hardening for Prometheus, Loki, Tempo, OpenTelemetry Collector, Grafana Alloy and Grafana.→
- AuthenticationHow each component decides who is calling. Basic auth, bearer tokens, OAuth, mTLS, and the right per-component choice for the observability stack.→
- AuthorizationWho can see what. The role-based and tenant-based authorization model across Prometheus, Loki, Tempo and Grafana, and the right per-component choice.→
- TLS Across the PlatformEncrypting traffic between every component, the certificate management that keeps it healthy, and the right per-component approach for a production stack.→
- Secrets ManagementWhere credentials live across Prometheus, Loki, Tempo, OpenTelemetry Collector, Grafana Alloy and Grafana, and the right per-component approach for a production stack.→
Part LXXVIII
Securing Prometheus
6 checks
- Prometheus Network ExposureListening interface controls. Where Prometheus binds, what --web.external-url controls, and how to keep the scrape engine on a private surface.→
- Prometheus AuthenticationBasic auth and bearer tokens for Prometheus scrape and federation endpoints. The right approach per scrape target.→
- Prometheus TLSTLS for scrape targets and the Prometheus web API. Server certificate validation, mTLS, and the production right shape.→
- Admin API RisksThe reload and quit endpoints, the right hardening posture, and how to keep configuration changes out of attacker hands.→
- Exporter AuthAuthentication on the exporter side. Most exporters do not authenticate; the right posture is private network plus TLS, with mTLS where the threat model demands it.→
- Prometheus and SecretsSecret management for Prometheus. File-based secrets, environment variables, and Vault. The right approach for scrape credentials and TLS material.→
Part LXXIX
Securing Grafana
6 checks
- Grafana AuthenticationLocal, LDAP, OAuth, SAML, OIDC. A production-focused deep dive.→
- Grafana RBACRoles and teams. A production-focused deep dive.→
- Datasource CredentialsSecrets in Grafana. A production-focused deep dive.→
- Anonymous Access DisciplineWhen anonymous is appropriate. A production-focused deep dive.→
- Plugin SecurityVetting third-party plugins. A production-focused deep dive.→
- Grafana Session SecurityCookie settings, session length. A production-focused deep dive.→
Part LXXX
Securing Loki
6 checks
- Loki AuthenticationThe auth_enabled flag, the X-Scope-OrgID header, and the right way to wire tenant authentication in a production Loki deployment.→
- Tenant IsolationHow Loki keeps one tenant invisible to another. The X-Scope-OrgID model, the storage prefix, the per-tenant limits_config, and the failure shape when the boundary leaks.→
- S3 PermissionsThe IAM role shape for Loki to S3. The actions the compactor and ingester need, the bucket policy that grants per-tenant prefix access, and the failure mode when the policy is too permissive.→
- Loki and PIIThe high-cardinality of PII in labels, the audit method that finds it, and the right place to redact: at the source, not at Loki.→
- Log InjectionHow a user message containing newlines or control characters splits a log line, corrupts queries, and forges log entries. The defence is a structured logger at the source.→
- Loki Hardening ChecklistThe full production hardening checklist for a multi-tenant Loki. Auth, TLS, S3 permissions, PII redaction, log-injection defence, and the runbook that ties them together.→
Part LXXXI
Securing Tempo
6 checks
- Trace Data SensitivityWhat traces carry in attributes, the data classes that arrive, and where the redaction discipline must live in a production pipeline.→
- Tempo Access ControlThe Tempo tenancy model, the per-tenant rate limits, the search gate at the querier, and the fronting proxy that turns a "Tempo with no auth" into "Tempo with users".→
- Tempo Object Storage SecurityThe S3 / GCS / Azure IAM permissions a Tempo cluster needs, the IRSA / Workload Identity shape on Kubernetes, and the boundary between the bucket and the rest of the fleet.→
- Attribute RedactionConfiguring the OpenTelemetry Collector to scrub, hash, and drop span attributes before they reach Tempo, and where this redaction belongs in the pipeline.→
- Tempo TLSTerminating TLS at the receivers, mTLS for service-mesh-backed ingesters, certificate validation with openssl, and what to do when TLS is terminated at a reverse proxy instead.→
- Tempo Hardening ChecklistBringing authentication, tenancy, TLS, attribute redaction, object-storage IAM, and observability-of-the-observability together into a single hardening checklist.→
Part LXXXII
Secrets and Sensitive Telemetry
6 checks
- Sensitive Data in TelemetryThe data classification, the common shapes of leak, and why telemetry is a data store with regulatory weight.→
- Redaction StrategiesWhere to redact, in what order, and the trade-off between source, agent, and backend redaction.→
- Redaction ToolsThe OpenTelemetry Collector attributes, transform, and redaction processors, and the Grafana Alloy loki.process stages, with verified configurations.→
- Secret Scanning TelemetryDetecting leaked secrets in stored metrics, logs, and spans with gitleaks, trufflehog, and detect-secrets.→
- Auditing Telemetry for LeaksThe quarterly review, the alert-and-metric, and the report that catches the leak before the regulator does.→
- Leak Incident ResponseThe runbook for the moment a leak is confirmed: contain, notify, rotate, redact, document.→
Part LXXXIII
Multi-Tenancy
6 checks
- Tenant ModelsTenant boundaries. A production-focused deep dive.→
- Tenant IsolationHow isolation is enforced. A production-focused deep dive.→
- Tenant OnboardingAdding a new tenant. A production-focused deep dive.→
- Cross-Tenant AccessWhen tenants legitimately need cross access. A production-focused deep dive.→
- Tenant Cost AllocationInvoicing / showback. A production-focused deep dive.→
- Tenant AuditQuarterly review. A production-focused deep dive.→
Part LXXXIV
Configuration as Code
6 checks
- Config as Code BasicsWhat goes in Git. A production-focused deep dive.→
- Prometheus Config as Codeprometheus.yml in Git. A production-focused deep dive.→
- Alert Rules as Coderules/*.yml in Git. A production-focused deep dive.→
- Alertmanager Config as Codealertmanager.yml in Git. A production-focused deep dive.→
- Grafana Provisioning as Codedatasources + dashboards in Git. A production-focused deep dive.→
- Loki and Tempo Config as Codeloki.yaml, tempo.yaml in Git. A production-focused deep dive.→
Part LXXXV
CI Validation
6 checks
- CI Validation BasicsThe pipeline gates, the per-tier ordering, and the right approach to fast-fail pre-merge checks for an observability stack.→
- promtool check configThe static configuration check, what it parses, what it does not, and the right way to wire it into CI.→
- promtool check rulesThe static rule check, what it parses inside rules.yml files, and the right way to wire it into CI.→
- promtool test rulesThe unit test format, the synthetic input series, and the right way to test alert and recording rules against known-good fixtures.→
- YAML Lintyamllint, the right configuration, the most common findings, and the way to wire it into pre-commit and CI.→
- End-to-End TestsEphemeral stacks, synthetic scrapes, real API queries, and the right way to prove a configuration works before production cutover.→
Part LXXXVI
Prometheus Rule Testing
6 checks
- Rule Test AnatomySeries, interval, expected output. The YAML shape of a promtool test rules fixture.→
- Test Cases Per RuleTrue positive, true negative, false positive, false negative. The canonical scenarios.→
- Recording Rule TestsAsserting on the value, not just the existence. Recording-rule fixtures and their discipline.→
- Rule Test CoverageHow many tests per rule, by tier. The discipline that catches gaps before production.→
- Regression TestsBug-driven test cases. The discipline that catches the same bug twice.→
- Rule Tests in CIGitHub Actions, required status checks, and the discipline of wiring rule tests into every PR.→
Part LXXXVII
Alert Testing
6 checks
- Alert Testing BasicsThe four-tier model for alert testing, the failure shapes each tier catches, and the discipline that makes alerting a tested artefact.→
- Synthetic Series for AlertsInjecting synthetic time series to drive alert conditions in a controlled way. The right approach and the most common shape.→
- End-to-End Alert TestVerifying the full alerting chain from rule evaluation through Alertmanager routing to receiver delivery.→
- Alert CanaryA scheduled alert that fires on a fixed cadence to verify the alerting chain is alive in production.→
- Alert Time-to-FireHow fast the alert should fire. The right per-tier budget and the most common shape of a tiered time-to-fire discipline.→
- Alert Test CadenceHow often to run the four tiers of alert testing. The right layered cadence and the most common shape of a quarterly review.→
Part LXXXVIII
Dashboard Testing and Review
6 checks
- Dashboard Review BasicsWhy review dashboards. A production-focused deep dive.→
- Query CorrectnessVerify each panel query. A production-focused deep dive.→
- Variable ValidationVariables work across combinations. A production-focused deep dive.→
- Data Source AvailabilityEach datasource is up. A production-focused deep dive.→
- Unit ConsistencyUnits consistent across panels. A production-focused deep dive.→
- Dashboard Test CadenceQuarterly review. A production-focused deep dive.→
Part LXXXIX
Observability Platform Monitoring Itself
6 checks
- Who Monitors the MonitoringThe recursive problem of monitoring the monitoring stack, the most common shape in production, and what fails when nobody owns it.→
- Prometheus HealthThe right approach to reading Prometheus self-health metrics: up, TSDB head, process, HTTP, and engine metrics.→
- Scrape Failure DetectionThe right approach to up == 0 alerts, scrape duration signals, and the failure modes that hide behind a green dashboard.→
- Rule Failure DetectionThe right approach to detecting Prometheus rule evaluation failures: rule_group_last_duration_seconds, rule_evaluation_failures_total, and the silent shapes.→
- Alertmanager DeliveryThe right approach to Alertmanager delivery metrics: notifications_total, notifications_failed_total, cluster health, and the failure modes that suppress pages.→
- Storage MonitoringThe right approach to monitoring disk usage for Prometheus TSDB and object storage for Loki and Tempo: WAL, blocks, buckets, retention.→
Part XC
Meta-Monitoring
6 checks
- Meta-Monitoring BasicsMonitoring the monitoring stack. A production-focused deep dive.→
- Separate Monitoring PlatformA second Prometheus for the first. A production-focused deep dive.→
- Federation for MetaPulling the platform Prometheus into a separate one. A production-focused deep dive.→
- Meta AlertsAlerts that fire when the meta is down. A production-focused deep dive.→
- Avoiding Circular AssumptionsDo not let meta depend on what it monitors. A production-focused deep dive.→
- Proving the Meta WorksTesting the meta monitoring. A production-focused deep dive.→
Part XCI
Backup Strategy
6 checks
- Backup BasicsWhat to back up. A production-focused deep dive.→
- Prometheus BackupTSDB snapshot, configuration. A production-focused deep dive.→
- Loki BackupObject store, ruler state, configuration. A production-focused deep dive.→
- Tempo BackupObject store, configuration. A production-focused deep dive.→
- Grafana BackupDatabase + provisioning. A production-focused deep dive.→
- Alertmanager BackupConfiguration + silences. A production-focused deep dive.→
Part XCII
Disaster Recovery
6 checks
- RPO and RTORecovery targets for the observability stack, tiered by signal.→
- Prometheus LossRecovering from Prometheus data loss, the recovery procedure, the most common shape.→
- Grafana LossRecovering the Grafana database, dashboards, and provisioning after loss.→
- Loki LossRecovering Loki after loss: ingesters, WAL, and the object store.→
- Tempo LossRecovering Tempo after loss: ingesters, the block backend, and trace replay.→
- Object Storage LossWhen the bucket is gone: cross-region replication and the recovery procedure.→
Part XCIII
Upgrades
6 checks
- Upgrade BasicsThe shape of an upgrade. Semver, breaking changes, the per-tier approach, and the upgrade window.→
- Reading Release NotesWhat to look for in a CHANGELOG, how to spot breaking changes, and the right way to handle security notes.→
- CompatibilityCross-component compatibility across the observability stack, the per-tier approach, and the most common skew shapes.→
- Rollback StrategyHow to roll back. The snapshot, the plan, the runbook, and the most common shape of a successful reversal.→
- Canary UpgradeOne component at a time. The canary host, the validation gates, and the most common shape of a safe fleet rollout.→
- Upgrade TestTesting the upgrade before production. The synthetic test, the replay, the integration test, and the most common shape of a validated upgrade.→
Part XCIV
Prometheus Upgrades
6 checks
- Prometheus Upgrade BasicsWhat changes between versions. The discipline that prevents a 03:00 outage from an in-place upgrade.→
- TSDB MigrationWhen the on-disk Prometheus TSDB layout changes between versions. WAL block lifecycle, snapshot, and the recovery shape if migration fails.→
- Prometheus Config CompatibilityHow the configuration schema evolves across Prometheus versions. Field removals, default flips, and the validation shape that catches regressions before they reach production.→
- Prometheus Rule CompatibilityHow recording rules and alerting rules behave across versions. PromQL semantic changes, promtool test rules, and the validation shape that catches regressions before production.→
- Upgrade OrderWhich side of the alerting chain to upgrade first, and why. Alertmanager, Prometheus, exporters. The order that prevents a rolling upgrade from breaking notifications.→
- Rollout ValidationAfter the upgrade: the validation procedure that confirms scrape, recording, alerting, remote-write, and dashboards are all behaving correctly. Pass/fail with evidence per subsystem.→
Part XCV
Grafana Upgrades
6 checks
- Grafana Upgrade BasicsThe discipline of upgrading Grafana. Semver, release cadence, configuration drift, and the upgrade window.→
- Database MigrationGrafana database migrations. From sqlite to MySQL or Postgres, schema migrations between Grafana versions, and the right approach to migration in production.→
- Plugin CompatibilityGrafana plugin compatibility across versions. Plugin SDK pinning, signature verification, and the right approach to plugin management during upgrades.→
- Dashboard CompatibilityGrafana dashboard JSON across versions. Schema migrations, deprecated panel types, and the right approach to dashboard portability.→
- Provisioning CompatibilityGrafana provisioning across versions. YAML schema changes, datasource UID stability, and the right approach to provisioning as code during upgrades.→
- Grafana Upgrade ValidationGrafana upgrade validation. Synthetic checks for login, datasources, dashboards, and alerting, and the right approach to post-upgrade verification.→
Part XCVI
Loki Upgrades
6 checks
- Loki Upgrade BasicsThe Loki upgrade discipline, the common failure shape, and the order of operations that keeps a running cluster readable across the cutover.→
- Loki Schema ConfigThe schema_config block, the v11 to v12 to v13 migration, the append-only discipline, and the failure modes of a wrong date.→
- Loki Storage MigrationThe bucket migration, the index migration, the order of operations, and the failure modes of an incomplete copy.→
- Loki Config MigrationThe config key mapping between Loki versions, the loki migrate tool, the runtime config file, and the failures of a half-migrated config.→
- boltdb-shipper to TSDBThe boltdb-shipper to TSDB index migration, the migrator CLI, the order of operations, and the failure modes of a partial migration.→
- Loki Upgrade ValidationThe validation checklist after a Loki upgrade, the synthetic log line, the loki-canary, and the diagnostics that prove the upgrade is complete.→
Part XCVII
Tempo Upgrades
6 checks
- Tempo Upgrade BasicsThe Tempo upgrade discipline, the common production failure shape, and the order of operations that keeps a running Tempo cluster readable across the cutover.→
- Block Format CompatibilityThe block format migrations between Tempo versions, the v1 / v2 / vParquet lineage, and the right way to drain a bucket across a format change.→
- Receiver CompatibilityThe OTLP, Jaeger, Zipkin, and Kafka receivers across Tempo versions, the protocols each one binds, and the right cutover when a receiver is removed.→
- TraceQL CompatibilityThe TraceQL evolution from preview to GA, the new intrinsics and aggregators in each minor release, and the right cutover when a query returns empty because the operator is too new.→
- Tempo Config ChangesThe renamed, restructured, and removed config keys across Tempo versions, the diff that catches a half-migrated config, and the right way to test a new values file before the upgrade window.→
- Tempo Upgrade ValidationThe validation checklist after a Tempo upgrade: per-role readiness, receiver traffic, synthetic trace round-trip, compactor cycle, and the diagnostic that proves the cutover is complete.→
Part XCVIII
Troubleshooting Methodology
6 checks
- Methodology SummaryThe troubleshooting methodology as a whole, and how its phases are weighted differently per severity tier.→
- Three Signals as Investigative ToolsHow to use metrics, logs, and traces as tools inside the investigation loop; the right signal at each phase.→
- Investigation Time BudgetBounding the search: how time budgets are set per tier, and the most common shape of a runaway investigation.→
- Hypothesis and Evidence DisciplineAvoiding red herrings: forming hypotheses with falsifiers, gathering independent evidence, and the discipline that protects against confirmation bias.→
- Mitigation vs Root CauseWhy mitigation precedes full root-cause analysis at P0 and P1, but precedes only after RCA at P3; the most common shape is the parallel investigation.→
- Post-Incident ReviewAfter the incident: blameless timeline, contributing factors, action items, and the most common shape of a PIR that actually changes behaviour.→
Part XCIX
Missing Metrics
6 checks
- Missing Metrics AnatomyThe six-stage chain from exporter to panel, the diagnostic order, and the most common cause.→
- Exporter DownWhen the exporter process is not running. The most common cause of missing metrics, and the diagnostic order.→
- Network BlockWhen the path between Prometheus and the exporter is dropped. Firewall, security group, network policy.→
- Scrape Config ErrorWrong configuration. YAML syntax, schema, and the right discipline to validate before reload.→
- Relabel DropsWhen a relabel rule silently removes a target or a metric. The right approach to find the rule.→
- Query WrongWhen the metric exists but the query returns nothing. PromQL syntax, label matching, and the right approach to debug.→
Part C
Missing Logs
6 checks
- Missing Logs AnatomyWhere to look. A production-focused deep dive.→
- Application Not EmittingWhen the application does not log. A production-focused deep dive.→
- Collector Not RunningWhen the collector is not running. A production-focused deep dive.→
- Pipeline BrokenWhen logs reach Loki but with wrong labels. A production-focused deep dive.→
- Loki Query WrongWhen the query is wrong. A production-focused deep dive.→
- Loki Rate LimitedWhen Loki drops logs. A production-focused deep dive.→
Part CI
Missing Traces
6 checks
- Missing Traces AnatomyThe five-link chain that turns an instrumented call into a stored trace, and the order to check each link when a trace is missing.→
- Instrumentation Not RunningWhen the SDK is loaded, attached, and exporting — or it is not. The most common cause of a missing trace, and how to confirm it in under a minute.→
- Propagation BrokenWhen the W3C traceparent header is missing, stripped, or replaced at a service boundary. The shape of a broken trace and the diagnostic order.→
- Collector DownWhen the OpenTelemetry Collector is unreachable, in CrashLoopBackOff, or not listening on the OTLP ports. The diagnostic order to confirm and recover.→
- Exporter BrokenWhen the collector exporter cannot deliver spans to Tempo. TLS handshake failures, endpoint drift, queue overflow, and the diagnostic order.→
- Sampling Too AggressiveWhen too few traces are kept or too many are kept. Head sampling, tail sampling, decision wait, and the diagnostic order.→
Part CII
Slow Queries
6 checks
- Slow Queries AnatomyHow a query becomes slow in Prometheus, Loki and Tempo, the order to check, and the most common single cause.→
- Large Range QueriesWide time windows, the cost they incur, and the recording-rule pattern that turns them into cheap reads.→
- High-Cardinality QueriesWhen a label set carries too many series, what to drop and where, and how to detect the explosion before the next query times out.→
- Expensive RegexWhen the matcher is a regex rather than an equality, the index cost changes shape. Anchoring, literal fast-paths, and where to put the matcher.→
- Poor AggregationAggregating at the wrong layer, the cost it incurs at the panel, and the recording-rule pattern that fixes it.→
- Storage BottleneckWhen the TSDB or the bucket store is the slow path, what to measure, and where to put the disk.→
Part CIII
Alert Failure
6 checks
- Alert Failure AnatomyWhen an alert should fire but does not. The pipeline from target to receiver, the diagnostic order, and the layer that is most often the cause.→
- Telemetry MissingWhen the upstream telemetry is missing. The chain from exporter health to TSDB, the diagnostic order, and the layer most often at fault.→
- Rule WrongWhen the rule expression is broken. The shape of the defect, the validation steps that catch it, and the symptom that distinguishes it from a threshold or routing defect.→
- Threshold WrongWhen the threshold is far from the operational truth. The shapes of mis-set thresholds, the discipline that finds the right value, and the symptom that distinguishes it from a rule-wrong defect.→
- Alertmanager Routing WrongWhen the Alertmanager route does not match. The shapes of a route-matcher defect, the validation steps that catch it, and the symptom that distinguishes it from a rule or receiver defect.→
- Receiver DownWhen the integration on the other side of Alertmanager is down. The shapes of a receiver-down failure, the diagnostic steps, and the discipline that catches the credential rotation gap.→
Part CIV
False Positive Alert
6 checks
- False Positive AnatomyWhen alerts fire but there is no incident. The diagnostic order to find the cause, and the layer most often to blame.→
- Threshold Too LowWhen a threshold is set against the wrong baseline. Setting it from observation, not from opinion.→
- `for:` Too ShortWhen `for:` matches a single-scrape blip instead of a sustained breach. The right way to lengthen it.→
- Rule Too BroadWhen the rule selects a wider series set than intended. Tightening the matcher.→
- False Positive InvestigationThe structured procedure for working out why an alert fired. Hypothesis, evidence, test, document.→
- Tuning ProcessThe discipline of tuning alerting rules tier by tier, tracking the rate, and reviewing monthly.→
Part CV
Cardinality Incident
6 checks
- Cardinality Incident AnatomyMemory spikes; OOM. A production-focused deep dive.→
- DetectionDetecting the incident. A production-focused deep dive.→
- InvestigationFinding the cause. A production-focused deep dive.→
- MitigationStabilising the platform. A production-focused deep dive.→
- RecoveryRestoring Prometheus. A production-focused deep dive.→
- Cardinality Budget Recurring ReviewQuarterly cardinality review.→
Part CVI
Log Ingestion Incident
6 checks
- Log Ingestion Spike AnatomyVolume rises sharply. The order to check, the most common cause, and the cost discipline of triage.→
- Debug Logging EnabledA code change turned debug logs on. The cost, the correct discipline, and the rollback that does not require a redeploy.→
- Log LoopA retry loop generating logs that trigger themselves. The signature, the cost, and the throttle that breaks the cycle.→
- New High-Volume ServiceA new service emitting logs to Loki. The cardinality risk, the onboarding checklist, and the guard rails that prevent the surprise.→
- Loki Capacity IncidentWhen Loki storage fills. The shapes, the immediate response, and the capacity model that prevents the next one.→
- Post-Incident Cost ReviewThe review that turns an incident into a guard rail. The format, the numbers, and the action items that close the loop.→
Part CVII
Trace Volume Incident
6 checks
- Trace Volume AnatomyWhen trace ingest climbs. A production-focused deep dive.→
- Sampling MisconfigA changed sampling rate. A production-focused deep dive.→
- Tail Sampler ErrorTail sampler accepts all. A production-focused deep dive.→
- Service Mesh EnabledSidecar tracing. A production-focused deep dive.→
- Mitigation and RecoveryStabilising Tempo. A production-focused deep dive.→
- Tempo Rotation BehaviorHow metrics rotate.→
Part CVIII
Clock Skew
6 checks
- Clock Skew AnatomyThe offset, the drift, and the most common shape of clock skew in a production fleet.→
- Effect on LogsLog ordering. The wrong-time symptom, the Loki rejection shapes, and the right discipline.→
- Effect on TracesSpan ordering under clock skew. The child-before-parent symptom and the right discipline.→
- Detecting SkewDetecting the skew. The chronyc commands, the node_timex metrics, and the right alert thresholds.→
- MitigationRestoring sync. Slewing versus stepping, and the right action when the offset crosses the threshold.→
- PreventionPreventing skew. The chrony discipline, the alerts, and the right cadence for the fleet-wide audit.→
Part CIX
Incident Investigation Workflows
6 checks
- Investigation AnatomyThe structure of an end-to-end production investigation: symptom, hypothesis, evidence, root cause, mitigation.→
- Example: Checkout LatencyA worked example of a checkout-latency investigation: symptoms, hypothesis, evidence, root cause, and mitigation.→
- Example: Error Rate SpikeA worked example of an error-rate-spike investigation: symptoms, hypothesis, evidence, root cause, and mitigation.→
- Example: Storage FullA worked example of a storage-full incident: disk saturation, retention configuration, and the recovery procedure.→
- Workflow TemplatesTemplates for common incident classes: the structure, the most common shapes, and how to record one for the team.→
- Workflow IterationRefining templates after each incident: the discipline that turns the first investigation into the second, faster investigation.→
Part CX
Observability During Major Incidents
6 checks
- Major Incident Observability OverviewWhen production is broken. A production-focused deep dive.→
- Incident DashboardsDedicated panels. A production-focused deep dive.→
- Change AnnotationsMarking changes. A production-focused deep dive.→
- Query DisciplineAvoiding runaway queries. A production-focused deep dive.→
- Evidence PreservationSaving the timeline. A production-focused deep dive.→
- Observability During the IncidentWhen observability itself fails. A production-focused deep dive.→
Part CXI
Observability Anti-Patterns
6 checks
- Monitor Everything Without PurposeThe over-collection anti-pattern. Cardinality cost, dashboard noise, and the SLI-first approach that replaces it.→
- Alert on Every MetricThe alert-fatigue anti-pattern. Why low-quality pages destroy on-call, and the symptom-first SLO-based replacement.→
- Unlimited RetentionThe unbounded-storage anti-pattern. Tiered retention, hot/warm/cold/frozen, and the cost curve that forces the decision.→
- High-Cardinality LabelsThe cardinality anti-pattern. Bounded labels, drop rules, and exemplars as the correct escape hatch.→
- Secrets in LogsThe PII and credential exposure anti-pattern. Redaction pipelines, secret scanners, and the assumption that logs are public within the platform.→
- Observability Without OwnerThe unowned-telemetry anti-pattern. CODEOWNERS, runbook ownership, and the discipline that keeps dashboards and alerts alive.→
Part CXII
Production Observability Operating Model
6 checks
- The Operating ModelWho owns what. A production-focused deep dive.→
- Shared vs Dedicated PlatformSharing the platform. A production-focused deep dive.→
- Service OwnershipThe application team owns the service. A production-focused deep dive.→
- Dashboard OwnershipDashboards need owners. A production-focused deep dive.→
- Alert OwnershipAlerts need owners. A production-focused deep dive.→
- Runbook OwnershipRunbooks need owners. A production-focused deep dive.→
Part CXIII
Documentation and Runbooks
6 checks
- Documentation from TelemetryThe discipline of writing runbook documentation that is anchored to live telemetry references, not invented in a meeting room.→
- What Failed?The first section of a runbook: how to state the failure precisely enough that the on-call knows what they are looking at.→
- ImpactThe impact section of a runbook: how to state user-visible effect in numbers, scope, and severity.→
- Dashboard LinkHow to embed a Grafana dashboard link in a runbook and an alert so the on-call lands on the right panel, with the right filters, at the right time.→
- Runbook LinkHow to link an alert to a runbook so the on-call lands on the right page, with the right context, before they have to find it themselves.→
- OwnerHow to declare the team that owns an alert and a runbook, and how to keep the declaration honest as the team and the alert move.→
Part CXIV
Final Production Reference Architecture
6 checks
- Reference Architecture AnatomyA canonical production observability stack. The five layers, the boundary contracts, and the four anti-patterns that ruin every stack that ignores them.→
- The Workload LayerLinux hosts, Docker containers, applications, and databases. The instrumented sources that produce telemetry for the rest of the stack.→
- The Collector LayerOpenTelemetry Collector and Grafana Alloy. The pipeline model, agent versus gateway topology, and the cardinality discipline that prevents the backend from collapsing.→
- The Backend LayerPrometheus, Loki, and Tempo. The three storage engines that own one signal each. Retention, sharding, and the failure modes that take a backend down.→
- The Presentation LayerGrafana 11.x and Alertmanager. The dashboards, the alerts, the routing tree, and the discipline that turns telemetry into an operator-facing surface.→
- The Operations LayerPeople, processes, and the discipline that turns the other five layers into something a team can run. Alert quality, ownership, escalation, and post-incident review.→