Observability · Self-assessment
Knowledge checks
Every knowledge check in this course, in curriculum order. Each link opens the page at its quiz. The questions are auto-graded in the browser and nothing is recorded — a wrong answer costs you only the explanation, which is the part worth reading.
- Knowledge checks
- 684
- Parts covered
- 114
- Of all lessons
- 100%
Part I
Foundations
6 checks
- Why Observability ExistsThe boundary between monitoring and observability, what telemetry signals buy you, and what they do not.→
- The Three Telemetry SignalsWhat metrics, logs and traces measure, how they are stored, and the questions each answers uniquely.→
- White-Box and Black-Box MonitoringThe fundamental distinction between monitoring a service from the inside and from the outside.→
- Symptoms, Evidence, and CausalityHow an investigator separates a symptom (observed) from a cause (inferred).→
- Service Health, User Impact, and TelemetryThe chain that connects an internal signal to a user impact decision.→
- Context, Correlation, and CausalityWhy an observability platform must carry context across services and how correlation IDs enable pivot-across-signal investigation.→
Part II
Production Monitoring Fundamentals
6 checks
- SLIs, SLOs, and Error BudgetsThe metrics definitions that anchor production observability: SLI, SLO, error budget.→
- Availability, Latency, Throughput, Errors, SaturationThe five operational dimensions that the platform measures for every service.→
- USE Methodology: Utilisation, Saturation, ErrorsResource-focused methodology for systems where the bottleneck is the resource itself.→
- RED Methodology: Rate, Errors, DurationService-focused methodology for distributed services.→
- Choosing the Right Methodology for Each SystemHow to pick between RED, USE, and other methodologies.→
- The Operational SLO PracticeHow to design, operate, and iterate on SLOs in production.→
Part III
Metrics Fundamentals
6 checks
- Counters, Gauges, Histograms, and SummariesThe four Prometheus metric types, what each captures, and when to choose which.→
- Labels, Dimensions, and Time SeriesHow labels and dimensions define a time series, and the design rules for using them.→
- Metric Naming ConventionsHow to name a metric so it survives across teams, dashboards, and alerts. A production-focused deep dive.→
- Aggregation Across Instancessum / avg / by / without aggregations; what they mean for the dashboard. A production-focused deep dive.→
- Scrape Intervals and ResolutionHow scrape interval affects what you can detect. A production-focused deep dive.→
- Time Series Storage and the TSDBHow Prometheus stores samples, and what 2.55 changes. A production-focused deep dive.→
Part IV
Cardinality
6 checks
- The Cardinality BudgetThe hard math of how many time series the platform can hold. A production-focused deep dive.→
- Dangerous Labels in PracticeWhy user IDs and request IDs are dangerous. A production-focused deep dive.→
- Cardinality From InstrumentationHow instrumentation patterns drive cardinality. A production-focused deep dive.→
- Cardinality From Loki LabelsLoki labels must be low-cardinality; log lines must not. A production-focused deep dive.→
- Cardinality Incident ResponseDiagnose, mitigate, recover, and prevent a cardinality blow-up. A production-focused deep dive.→
- Cardinality GovernanceReviews, lint rules, and audit cadence. A production-focused deep dive.→
Part V
Prometheus Architecture
6 checks
- The Prometheus Pull ModelWhy scrape is the primary ingestion mechanism. A production-focused deep dive.→
- The Scrape LifecycleWhat happens during a scrape: HTTP, parses, writes. A production-focused deep dive.→
- The TSDB EngineBlocks, compaction, mmapped files, what changed in 2.55. A production-focused deep dive.→
- Rules and the Alerting PipelineRecording rules, alerting rules, evaluation, Alertmanager. A production-focused deep dive.→
- FederationHierarchical Prometheus for multi-tenant estates. A production-focused deep dive.→
- Remote Write and Remote ReceiveHow scaling Prometheus works in 2026. A production-focused deep dive.→
Part VI
Installing Prometheus
6 checks
- Prometheus Installation Methodsbinaries, packages, containers, Compose. A production-focused deep dive.→
- Filesystem LayoutWhere configuration, data, and logs live. A production-focused deep dive.→
- systemd IntegrationService user, lifecycle, sandboxing. A production-focused deep dive.→
- Docker Compose InstallationThe lab default. A production-focused deep dive.→
- Retention and Storageretention.time, retention.size, TSDB. A production-focused deep dive.→
- Permissions and Service UserLeast privilege, capabilities, filesystem permissions. A production-focused deep dive.→
Part VII
Prometheus Configuration
6 checks
- Global Settingsscrape_interval, evaluation_interval, external_labels. A production-focused deep dive.→
- Scrape Configsjob_name, scheme, targets, metrics_path, tls_config. A production-focused deep dive.→
- Rule Filesgroups, recording rules, alerting rules. A production-focused deep dive.→
- Alerting Sectionalertmanagers, alert_relabel_configs. A production-focused deep dive.→
- Remote Write Sectionqueue_config, metadata_config, sigv4. A production-focused deep dive.→
- Configuration Validationpromtool check config, common mistakes. A production-focused deep dive.→
Part VIII
Service Discovery
6 checks
- Static Discoverystatic_configs. A production-focused deep dive.→
- File-Based DiscoveryJSON / YAML file_sd_configs. A production-focused deep dive.→
- DNS-Based Discoverydns_sd_configs and SRV records. A production-focused deep dive.→
- Container DiscoveryDocker, Kubernetes, container_sd_configs. A production-focused deep dive.→
- Relabeling and Target Managementaction, regex, source_labels, target_label. A production-focused deep dive.→
- Metric RelabelingFiltering, dropping, modifying samples at ingestion. A production-focused deep dive.→
Part IX
Exporters
6 checks
- The Exporter ContractWhat an exporter is and what its metric exposition looks like. A production-focused deep dive.→
- Exporter TypesHost, application, database, queue exporters. A production-focused deep dive.→
- Community Exporter TrustProject health, security, version pinning. A production-focused deep dive.→
- Writing an ExporterWhen to write one and when not. A production-focused deep dive.→
- Exporter PerformanceCost, scraping, parallelism, failure. A production-focused deep dive.→
- Exporter SecurityListening interface, TLS, basic auth. A production-focused deep dive.→
Part X
node_exporter
6 checks
- node_exporter OverviewCollectors, default-enabled, collector selection. A production-focused deep dive.→
- CPU Metricsnode_cpu_seconds_total, modes, multi-CPU. A production-focused deep dive.→
- Memory MetricsMemTotal, MemAvailable, buffers, cache. A production-focused deep dive.→
- Filesystem Metricsinodes, size, free, mountinfo. A production-focused deep dive.→
- Disk Metricsnode_disk_*, read/write/IOPS, queues. A production-focused deep dive.→
- Network Metricsnode_network_*, bytes, packets, errors, drops. A production-focused deep dive.→
Part XI
Blackbox Monitoring
6 checks
- blackbox_exporter Overviewmodules, probes, target parameters. A production-focused deep dive.→
- TCP Probestcp_connect, payload matching. A production-focused deep dive.→
- ICMP Probesicmp, ping limits, security context. A production-focused deep dive.→
- DNS ProbesDNS lookup, validation, recursion. A production-focused deep dive.→
- Blackbox Incident RunbookA reference incident response.→
- TLS Certificate Probesprobe_ssl_earliest_cert_expiry. A production-focused deep dive.→
Part XII
PromQL Foundations
6 checks
- PromQL AnatomyMetric selectors, label matchers, operators. A production-focused deep dive.→
- Instant and Range Vectors[5m] selects a range; bare selects instant. A production-focused deep dive.→
- Label Matching Operators=, !=, =~, !~. A production-focused deep dive.→
- Binary OperatorsArithmetic, comparison, set operations. A production-focused deep dive.→
- Vector Matchingon / ignoring / group_left / group_right. A production-focused deep dive.→
- Functions and String Operationstopk, bottomk, absent, label_replace. A production-focused deep dive.→
Part XIII
Rates and Counters
6 checks
- rate() and Counter ResetsHow rate() handles resets gracefully. A production-focused deep dive.→
- irate() and increase()When to use irate vs rate; increase for totals. A production-focused deep dive.→
- Why Counters Aren't Plotted Directly'Resets, scale, comparison over time.→
- Counter Monotonicity in PracticeDetecting monotonicity violations. A production-focused deep dive.→
- Rate Window Selection5m, 1m, 30s — when does the choice matter. A production-focused deep dive.→
- derivative() for Gaugesrate of change for non-monotonic metrics. A production-focused deep dive.→
Part XIV
Aggregation
6 checks
- sum() and avg()The two operators most dashboards use. A production-focused deep dive.→
- min(), max(), count()Distribution operators. A production-focused deep dive.→
- topk() and bottomk()The rank operators. A production-focused deep dive.→
- by() and without()The grouping labels. A production-focused deep dive.→
- Aggregation Across InstancesComputing fleet-wide metrics safely. A production-focused deep dive.→
- Over-Aggregation RisksWhen dropping labels hides the incident. A production-focused deep dive.→
Part XV
Histograms and Latency
6 checks
- Histogram Bucketsle="..." — understanding the bucket layout. A production-focused deep dive.→
- histogram_quantile()p50, p95, p99 — what works, what does not. A production-focused deep dive.→
- Cumulative Bucketsle="..." = "observations less than X." A production-focused deep dive.→
- _sum and _countAverage latency, rate of observations. A production-focused deep dive.→
- Native HistogramsPrometheus 2.50+: sparse histograms, exact quantiles. A production-focused deep dive.→
- Choosing the Bucket LayoutDefault buckets, ExponentialBuckets, SLO-driven. A production-focused deep dive.→
Part XVI
PromQL Troubleshooting
6 checks
- Common PromQL MistakesWrong rate window, missing labels, vector matching. A production-focused deep dive.→
- Stale SeriesWhen a target stops scraping. A production-focused deep dive.→
- absent() and Missing DataDetecting missing metrics reliably. A production-focused deep dive.→
- Cardinality Explosion in QueriesWhen a query produces thousands of series. A production-focused deep dive.→
- Vector Matching MistakesMany-to-many and on / ignoring pitfalls. A production-focused deep dive.→
- PromQL Performance TipsPractical performance tips.→
Part XVII
Recording Rules
6 checks
- Recording Rule PurposeWhy expensive or repeated PromQL becomes a recording rule. A production-focused deep dive.→
- Naming Recording RulesConventions for what to name a recording rule. A production-focused deep dive.→
- Rule EvaluationWhen rules evaluate, what happens on failure. A production-focused deep dive.→
- Rule PerformanceCost of rules, when rules become expensive. A production-focused deep dive.→
- Rule OrganisationFiles, groups, comments, naming. A production-focused deep dive.→
- Recording Rules for SLOsThe specific role rules play in SLO computation. A production-focused deep dive.→
Part XVIII
Alerting Rules
6 checks
- The Alert Rule Anatomyexpr, for, labels, annotations. A production-focused deep dive.→
- Severity and Routing LabelsHow labels carry routing decisions. A production-focused deep dive.→
- `for:` and HysteresisHow `for:` reduces flapping. A production-focused deep dive.→
- Annotations: Runbook, Dashboard, DescriptionThe human-facing context that comes with the alert. A production-focused deep dive.→
- Common Rule PatternsBlackbox probe failure, scrape failure, missing metric, error rate, latency breach. A production-focused deep dive.→
- Rule Lint and ReviewHow rules are reviewed before production. A production-focused deep dive.→
Part XIX
Alertmanager
6 checks
- Alertmanager AnatomyReceivers, routes, inhibition, mute times, group_by. A production-focused deep dive.→
- The Routing TreeHow routes are organised. A production-focused deep dive.→
- Grouping of AlertsHow alertmanager collapses N firing alerts into 1 notification. A production-focused deep dive.→
- InhibitionSuppression of dependent alerts. A production-focused deep dive.→
- Silences and MutesTime-bounded mute of an alert. A production-focused deep dive.→
- Receivers and TemplatesEmail, Slack, PagerDuty, webhook. A production-focused deep dive.→
Part XX
Alert Quality
6 checks
- Good Alerts, Bad AlertsWhat makes an alert useful. A production-focused deep dive.→
- Symptoms vs CausesWhy alerting on CPU is wrong; alerting on latency is right. A production-focused deep dive.→
- The Page vs Ticket DecisionWhen to wake someone up. A production-focused deep dive.→
- Alert FatigueThe failure shape that breaks the system. A production-focused deep dive.→
- Actionable AlertsWhat every page-worthy alert must include. A production-focused deep dive.→
- Alert Quality ReviewThe quarterly audit. A production-focused deep dive.→
Part XXI
Alert Inhibition
6 checks
- Why Inhibition ExistsSuppressing consequence alerts. A production-focused deep dive.→
- Basic Inhibitionhost down -> service down. A production-focused deep dive.→
- Dependency-Aware AlertingMapping service dependencies. A production-focused deep dive.→
- Inhibit by ClusterReaching across the platform. A production-focused deep dive.→
- Inhibit, then PageCoordinating with severity. A production-focused deep dive.→
- Inhibit Rule ReviewWhen the dependency map changes. A production-focused deep dive.→
Part XXII
SLO-Based Alerting
6 checks
- Burn Rate 101How fast the budget is being consumed. A production-focused deep dive.→
- Multi-Window Burning2-window and 6-window combinations. A production-focused deep dive.→
- The Error Budget PolicyWhat to do when the budget is consumed. A production-focused deep dive.→
- SLO Alert TemplatesProduction templates. A production-focused deep dive.→
- SLO Alerting RisksWhen SLO-only alerting misses things. A production-focused deep dive.→
- SLO IterationHow the SLO is revised as the service grows. A production-focused deep dive.→
Part XXIII
Grafana Foundations
6 checks
- Grafana AnatomyServer, users, orgs, datasources, dashboards. A production-focused deep dive.→
- DatasourcesPrometheus, Loki, Tempo as Grafana datasources. A production-focused deep dive.→
- Panels and QueriesTime series, stat, gauge, table, logs, traces. A production-focused deep dive.→
- Organisations and TeamsTenant model. A production-focused deep dive.→
- Alerting in GrafanaGrafana-managed alerts versus Prometheus-managed alerts. A production-focused deep dive.→
- Provisioning ModelDashboards and datasources as code. A production-focused deep dive.→
Part XXIV
Grafana Installation
6 checks
- Grafana Installation Methodspackages, container, Docker Compose. A production-focused deep dive.→
- Storage and SQLiteWhere Grafana stores dashboards, users, and audit logs. A production-focused deep dive.→
- systemd and Process ManagementThe Grafana service. A production-focused deep dive.→
- Reverse Proxy and TLSPutting Grafana behind nginx / Traefik / HAProxy. A production-focused deep dive.→
- Initial Admin and AuthenticationSetting up the first user, locking down anonymous. A production-focused deep dive.→
- Installation ValidationConfirm Grafana is up. A production-focused deep dive.→
Part XXV
Grafana Data Sources
6 checks
- Prometheus as a Data SourceHow Prometheus connects to Grafana. A production-focused deep dive.→
- Loki as a Data SourceQuerying Loki from Grafana. A production-focused deep dive.→
- Tempo as a Data SourceQuerying Tempo from Grafana. A production-focused deep dive.→
- Tags and FilteringTagging datasources for environment / region. A production-focused deep dive.→
- Datasource Secret RotationSecrets must rotate.→
- Rotating Datasource CredentialsWhat to do when secrets rotate. A production-focused deep dive.→
Part XXVI
Dashboard Design
6 checks
- Dashboard HierarchyService overview vs service detail. A production-focused deep dive.→
- Units and ThresholdsConsistency across dashboards. A production-focused deep dive.→
- Overview DashboardsThe fleet-state panel set. A production-focused deep dive.→
- Service DashboardsThe per-service operating page. A production-focused deep dive.→
- Links and AnnotationsDrilling from one dashboard to another. A production-focused deep dive.→
- Dashboard OwnershipWho owns what. A production-focused deep dive.→
Part XXVII
Dashboard Anti-Patterns
6 checks
- Wallpaper DashboardsWhen a dashboard has too many panels. A production-focused deep dive.→
- Meaningless GaugesWhen a panel does not help. A production-focused deep dive.→
- Rainbow Color PalettesWhy not all palettes are equal. A production-focused deep dive.→
- No Context and No OwnershipWhen a panel cannot be acted on. A production-focused deep dive.→
- Inconsistent UnitsWhy `bytes` here and `MB` there is a bug. A production-focused deep dive.→
- No Troubleshooting LinkWhen the panel shows a problem but offers no path forward. A production-focused deep dive.→
Part XXVIII
Grafana Variables
6 checks
- Template Variable PurposeWhy a dashboard takes parameters. A production-focused deep dive.→
- Variable TypesQuery, custom, datasource, interval, text box. A production-focused deep dive.→
- Variable Interpolation`$var` in queries. A production-focused deep dive.→
- Common Variable Patternsinstance, route, region, environment. A production-focused deep dive.→
- Variable Query CostExpensive variable queries. A production-focused deep dive.→
- Repeated Panels and RowsWhen one panel becomes many. A production-focused deep dive.→
Part XXIX
Grafana Provisioning
6 checks
- Provisioning AnatomyWhere provisioning files live. A production-focused deep dive.→
- Provisioning DatasourcesYAML format for datasources. A production-focused deep dive.→
- Provisioning DashboardsYAML format for dashboards. A production-focused deep dive.→
- Sidecar ProvisioningDiscovering dashboards from disk. A production-focused deep dive.→
- Testing Provisioning LocallySpin up a Grafana with the same provisioning. A production-focused deep dive.→
- Drift PreventionHow UI edits are caught. A production-focused deep dive.→
Part XXX
Grafana Security
6 checks
- Authentication and UsersHow Grafana authenticates. A production-focused deep dive.→
- RBAC and TeamsRole-based access. A production-focused deep dive.→
- Anonymous AccessWhen it is appropriate and when it is not. A production-focused deep dive.→
- Data Source CredentialsSecrets in datasources. A production-focused deep dive.→
- Secrets and Vault IntegrationVault as a credential source. A production-focused deep dive.→
- Plugins and TLSThird-party plugins. A production-focused deep dive.→
Part XXXI
Logging Foundations
6 checks
- Logs vs Structured LogsThe difference between unstructured and structured logs. A production-focused deep dive.→
- Log Levels and SeveritySeverity levels, when to use which. A production-focused deep dive.→
- Timestamps and Time ZonesLog timing discipline. A production-focused deep dive.→
- Correlation IDs in LogsTrace IDs across services. A production-focused deep dive.→
- PII and Secrets in LogsWhy logs should be inspected for sensitive content. A production-focused deep dive.→
- Machine-Readable LogsJSON / logfmt / structured metadata. A production-focused deep dive.→
Part XXXII
Logging Pipeline Architecture
6 checks
- The Logging PipelineHosts to agent to transport to Loki to query. A production-focused deep dive.→
- Agents vs SidecarsProcess to ship logs. A production-focused deep dive.→
- Grafana Alloy vs OpenTelemetry CollectorTwo strategic collectors. A production-focused deep dive.→
- Promtail in Maintenance ModeWhy Promtail is no longer the default. A production-focused deep dive.→
- Pipeline ResilienceBackpressure, batching, retries. A production-focused deep dive.→
- Pipeline SecurityAuthentication, TLS, redaction. A production-focused deep dive.→
Part XXXIII
Loki Architecture
6 checks
- Loki Architecture OverviewWhy Loki is different. A production-focused deep dive.→
- StreamsA label set with log lines. A production-focused deep dive.→
- ChunksCompressed bundles of log lines. A production-focused deep dive.→
- IndexesThe label index and chunk index. A production-focused deep dive.→
- Ingester LifecycleHow ingesters write chunks. A production-focused deep dive.→
- CompactorCompaction and retention. A production-focused deep dive.→
Part XXXIV
Loki Labels and Cardinality
6 checks
- The Loki Label RuleOnly low-cardinality labels. A production-focused deep dive.→
- Good Loki Labelsapplication, env, host, namespace. A production-focused deep dive.→
- Bad Loki Labelsrequest_id, user_id, session_id. A production-focused deep dive.→
- Data in Labels vs Data in Log ContentWhen to use structured metadata. A production-focused deep dive.→
- Loki Cardinality IncidentMemory spikes from a new label. A production-focused deep dive.→
- Loki Label AuditQuarterly review of label design. A production-focused deep dive.→
Part XXXV
Loki Installation
6 checks
- Loki Installation ModesSingle binary vs microservices. A production-focused deep dive.→
- Loki Configurationloki.yaml structure. A production-focused deep dive.→
- Loki StorageObject store selection. A production-focused deep dive.→
- Loki Retention Configurationretention_period. A production-focused deep dive.→
- Loki Ingestion Limitslimits_config patterns. A production-focused deep dive.→
- Loki Validationloki-cli checks. A production-focused deep dive.→
Part XXXVI
Log Shipping
6 checks
- Grafana Alloy OverviewThe strategic Grafana collector. A production-focused deep dive.→
- Alloy ConfigurationAlloy syntax and components. A production-focused deep dive.→
- OpenTelemetry Collector OverviewThe vendor-neutral alternative. A production-focused deep dive.→
- OTel Collector Configurationotelcol.yaml. A production-focused deep dive.→
- Choosing the Right CollectorWhen Alloy, when OTel. A production-focused deep dive.→
- Collector SecurityTLS, auth, redaction. A production-focused deep dive.→
Part XXXVII
LogQL Foundations
6 checks
- LogQL Stream SelectorsFiltering by Loki labels. A production-focused deep dive.→
- LogQL Line FiltersFiltering by content. A production-focused deep dive.→
- LogQL ParsersExtracting structured fields. A production-focused deep dive.→
- LogQL Aggregationssum, count, avg, max. A production-focused deep dive.→
- Structured Logs in GrafanaRendering parsed fields. A production-focused deep dive.→
- LogQL Query PerformanceReducing query latency. A production-focused deep dive.→
Part XXXVIII
LogQL Metrics
6 checks
- Log-Derived MetricsWhen logs become metrics. A production-focused deep dive.→
- Error Rates From LogsComputing rate from ERROR-level logs. A production-focused deep dive.→
- Log Volume from LogsDetecting log volume changes. A production-focused deep dive.→
- Latency From LogsWhen logs have latency fields. A production-focused deep dive.→
- Logs vs Native MetricsWhen to prefer each. A production-focused deep dive.→
- Log-Based AlertingAlerting on log-derived metrics. A production-focused deep dive.→
Part XXXIX
Log Troubleshooting
6 checks
- Missing LogsWhen Loki has no logs from a service. A production-focused deep dive.→
- Timestamp IssuesWhen logs arrive with wrong time. A production-focused deep dive.→
- Label MismatchWhen labels are right but queries return nothing. A production-focused deep dive.→
- Ingestion FailuresWhen dist_drop reasons grow. A production-focused deep dive.→
- Malformed Structured LogsWhen parsers fail. A production-focused deep dive.→
- Query PerformanceSlow queries. A production-focused deep dive.→
Part XL
Log Retention
6 checks
- Retention BasicsHow long are logs kept? A production-focused deep dive.→
- Legal and ComplianceSome industries have minimums; some have maximums. A production-focused deep dive.→
- Capacity PlanningBytes / sec times retention. A production-focused deep dive.→
- Object StorageLoki stores chunks in object storage. A production-focused deep dive.→
- Retention Policies Per StreamSome streams warrant longer retention. A production-focused deep dive.→
- Retention AuditQuarterly review. A production-focused deep dive.→
Part XLI
Distributed Tracing Foundations
6 checks
- Trace AnatomyTrace, span, parent / child. A production-focused deep dive.→
- Span AttributesThe metadata on a span. A production-focused deep dive.→
- Parent / Child RelationshipsHow spans nest. A production-focused deep dive.→
- Trace ID and Span IDIdentifiers. A production-focused deep dive.→
- Span EventsAnnotations on spans. A production-focused deep dive.→
- Span StatusOK, ERROR, UNSET. A production-focused deep dive.→
Part XLII
Why Tracing Exists
6 checks
- Questions Only Traces AnswerWhere the time went; which dependency failed. A production-focused deep dive.→
- Dependency Latency TracingFinding slow dependencies. A production-focused deep dive.→
- Traces vs Metrics CorrelationWhen each shines. A production-focused deep dive.→
- The Critical PathIdentifying bottlenecks. A production-focused deep dive.→
- Trace Propagation TestVerifying the trace ID survives. A production-focused deep dive.→
- Tracing Cost ConsiderationsWhy 100% sampling is not sustainable. A production-focused deep dive.→
Part XLIII
Instrumentation
6 checks
- Manual vs Automatic InstrumentationSDK vs agent. A production-focused deep dive.→
- OpenTelemetry SDKHow the SDK exposes traces. A production-focused deep dive.→
- Auto-InstrumentationWrapping frameworks. A production-focused deep dive.→
- Context PropagationHow trace IDs cross service boundaries. A production-focused deep dive.→
- Attribute DesignWhat to put on a span. A production-focused deep dive.→
- SDK DeploymentLibrary version pinning. A production-focused deep dive.→
Part XLIV
Sampling
6 checks
- Head vs Tail SamplingWhere the sampling decision is made. A production-focused deep dive.→
- Sampling RateChoosing the rate. A production-focused deep dive.→
- Rare Error TracesCapturing rare errors that head sampling misses. A production-focused deep dive.→
- Tail Sampling CostWhere tail sampling pays. A production-focused deep dive.→
- Sampling Strategy Per ServiceMixing head and tail. A production-focused deep dive.→
- Re-tuning SamplingQuarterly review. A production-focused deep dive.→
Part XLV
Tempo Architecture
6 checks
- Tempo Architecture OverviewDistributor, ingester, querier, compactor. A production-focused deep dive.→
- The DistributorReceive-side validation. A production-focused deep dive.→
- The IngesterBatching spans into blocks. A production-focused deep dive.→
- The CompactorCompacting blocks. A production-focused deep dive.→
- The QuerierWhere queries happen. A production-focused deep dive.→
- The Metrics GeneratorGenerating RED metrics from traces. A production-focused deep dive.→
Part XLVI
Tempo Deployment
6 checks
- Tempo Deployment ModesSingle binary vs microservices. A production-focused deep dive.→
- Tempo ReceiversOTLP, Jaeger, Zipkin. A production-focused deep dive.→
- Tempo StorageObject storage selection. A production-focused deep dive.→
- Tempo Retentioncompactor.block_retention. A production-focused deep dive.→
- Tempo QueryingQuery via TraceQL. A production-focused deep dive.→
- Tempo Validationtempo-cli validate-config. A production-focused deep dive.→
Part XLVII
Trace Queries
6 checks
- TraceQL IntroductionQuerying traces in TraceQL. A production-focused deep dive.→
- TraceQL SelectorsFiltering traces by attributes. A production-focused deep dive.→
- TraceQL AggregationsCount and rate over traces. A production-focused deep dive.→
- TraceQL Intrinsicstrace.duration, span.name. A production-focused deep dive.→
- Pivot from Metrics to TracesUsing exemplars to find traces. A production-focused deep dive.→
- Trace Exploration WorkflowsPractical patterns for exploring.→
Part XLVIII
Trace Troubleshooting
6 checks
- Missing SpansWhen a trace is incomplete. A production-focused deep dive.→
- Broken Trace PropagationWhen trace ID is not propagated. A production-focused deep dive.→
- Sampling IssuesWhen rare traces never appear. A production-focused deep dive.→
- Clock Problems in TracesWhen spans have inconsistent times. A production-focused deep dive.→
- Collector FailuresWhen spans reach the collector but not Tempo. A production-focused deep dive.→
- High-Cardinality AttributesWhen span attributes explode cost. A production-focused deep dive.→
Part XLIX
OpenTelemetry Foundations
6 checks
- OpenTelemetry OverviewSignals, SDK, Collector. A production-focused deep dive.→
- SignalsMetrics, logs, traces, profiles. A production-focused deep dive.→
- SDKs by LanguageWhat is supported by each language. A production-focused deep dive.→
- OTLPThe OpenTelemetry wire protocol. A production-focused deep dive.→
- Collector vs Agent PatternsAgent on each host; Gateway centrally. A production-focused deep dive.→
- OpenTelemetry Today and TomorrowThe OTel roadmap. A production-focused deep dive.→
Part L
OpenTelemetry Collector
6 checks
- Collector AnatomyReceivers, processors, exporters, pipelines, extensions. A production-focused deep dive.→
- ReceiversHow the Collector receives data. A production-focused deep dive.→
- Processorsbatch, attributes, filter, transform. A production-focused deep dive.→
- ExportersWhere the Collector sends data. A production-focused deep dive.→
- PipelinesHow receivers connect to exporters. A production-focused deep dive.→
- Collector Deployment PatternsAgent and Gateway. A production-focused deep dive.→
Part LI
Correlating Metrics, Logs, and Traces
6 checks
- Correlation AnatomyThe join key across signals. A production-focused deep dive.→
- Metric to Log WorkflowPivot from a metric datapoint to logs. A production-focused deep dive.→
- Metric to Trace WorkflowPivot from a histogram to a trace. A production-focused deep dive.→
- Log to Trace WorkflowPivot from logs to traces. A production-focused deep dive.→
- Grafana CorrelationsConfiguring datasource correlations. A production-focused deep dive.→
- Correlation TestVerifying the workflow. A production-focused deep dive.→
Part LII
Exemplars
6 checks
- Exemplars OverviewLinking metric datapoints to traces. A production-focused deep dive.→
- Exemplar FormatOpenMetrics exemplar representation. A production-focused deep dive.→
- Exemplar EmissionConfiguring the histogram to emit. A production-focused deep dive.→
- Grafana Exemplar LinkingConfiguring Grafana dashboards. A production-focused deep dive.→
- Exemplars CostStorage overhead. A production-focused deep dive.→
- Exemplar TroubleshootingWhen exemplars do not link. A production-focused deep dive.→
Part LIII
Log / Trace Correlation
6 checks
- Log / Trace LinkIncluding trace_id in structured logs. A production-focused deep dive.→
- Service Metadata in LogsIncluding service.name, span_id. A production-focused deep dive.→
- LogQL Pivot to TraceGrafana jump from logs to trace. A production-focused deep dive.→
- Tempo Derived FieldsPulling trace IDs out of unstructured log lines. A production-focused deep dive.→
- Correlation CostWhen pivot queries get slow. A production-focused deep dive.→
- End-to-End Correlation TestValidating correlation features. A production-focused deep dive.→
Part LIV
Dashboard-to-Logs Workflows
6 checks
- Dashboard-to-Logs AnatomyClick from a panel to Loki. A production-focused deep dive.→
- Correlation Configurationdatasource_correlations in Grafana. A production-focused deep dive.→
- Label Passing Through the PivotSharing labels between queries. A production-focused deep dive.→
- Filtered Stream PivotGoing from a single route to its stream. A production-focused deep dive.→
- Severity and Context PreservationKeeping severity, time, and labels. A production-focused deep dive.→
- Avoiding Pivot SpamLimiting pivots. A production-focused deep dive.→
Part LV
Dashboard-to-Traces Workflows
6 checks
- Dashboard-to-Traces OverviewFrom panel to trace. A production-focused deep dive.→
- Exemplar Link to TraceClick the exemplar. A production-focused deep dive.→
- Latency Pivot WorkflowFrom p99 to slow trace. A production-focused deep dive.→
- Error Pivot WorkflowFrom error rate to failure trace. A production-focused deep dive.→
- Dependency AttributionFinding the slow dependency. A production-focused deep dive.→
- Trace ExplorationBrowsing related traces. A production-focused deep dive.→
Part LVI
Linux Observability
6 checks
- Linux Metrics OverviewThe node_exporter baseline. A production-focused deep dive.→
- Memory ObservabilityMemAvailable, swap, OOM. A production-focused deep dive.→
- Disk Latency ObservabilityI/O latency, queue depth, throughput. A production-focused deep dive.→
- Network ObservabilityThroughput, errors, drops. A production-focused deep dive.→
- Service Availabilitysystemd-unit-state metrics. A production-focused deep dive.→
- Time Sync Observabilitychrony / systemd-timesyncd. A production-focused deep dive.→
Part LVII
Docker Observability
6 checks
- Docker Host MetricsNode-level metrics. A production-focused deep dive.→
- cAdvisor OverviewContainer metrics engine. A production-focused deep dive.→
- cAdvisor Metricscontainer_* metrics. A production-focused deep dive.→
- Short-Lived Container MetricsWhen containers live for seconds. A production-focused deep dive.→
- Docker LogsContainer stdout / stderr. A production-focused deep dive.→
- Docker TracesAuto-instrumentation in containers. A production-focused deep dive.→
Part LVIII
Proxmox Observability
6 checks
- Proxmox APIs and Metricspvestatd / pveproxy metrics. A production-focused deep dive.→
- The pve_exporterPrometheus exporter for Proxmox. A production-focused deep dive.→
- Cluster Health Metricscorosync status, quorum. A production-focused deep dive.→
- VM Resource MetricsVM CPU, memory, disk. A production-focused deep dive.→
- Storage MetricsZFS / Ceph / NFS metrics. A production-focused deep dive.→
- Backup Metrics (PBS)PBS metrics. A production-focused deep dive.→
Part LIX
Database Observability
6 checks
- Database MetricsConnections, queries, latency. A production-focused deep dive.→
- Connection PoolActive / idle / wait. A production-focused deep dive.→
- Query Latencyp95 query latency by query. A production-focused deep dive.→
- Locks and ContentionLock waits, deadlocks. A production-focused deep dive.→
- Replication LagReplica health. A production-focused deep dive.→
- Caching (Buffer, Redis)Hit ratio, evictions. A production-focused deep dive.→
Part LX
Network Observability
6 checks
- Network MetricsHost network metrics. A production-focused deep dive.→
- TCP ObservabilityRetransmits, RTT. A production-focused deep dive.→
- SNMP for Network DevicesSwitches, routers. A production-focused deep dive.→
- DNS MonitoringQuery success, latency. A production-focused deep dive.→
- Network ProbesSynthetic ICMP / TCP probes. A production-focused deep dive.→
- Firewall ObservabilityDrop counts, rule counts. A production-focused deep dive.→
Part LXI
Application Observability
6 checks
- Application REDRate, Errors, Duration. A production-focused deep dive.→
- Application USEResource use per container / process. A production-focused deep dive.→
- Dependency PerformancePer-dependency latency. A production-focused deep dive.→
- Saturation from ApplicationQueue depth, thread pool. A production-focused deep dive.→
- Custom Metrics ResponsiblyWhen to add custom metrics. A production-focused deep dive.→
- Application SLOsPair RED with SLOs. A production-focused deep dive.→
Part LXII
Business Metrics
6 checks
- Business vs Observability MetricsWhen business metrics matter. A production-focused deep dive.→
- User Journey MetricsPer-stage counts. A production-focused deep dive.→
- Revenue Impact MetricsTracking user-visible losses. A production-focused deep dive.→
- A/B Test MetricsObservability and experimentation. A production-focused deep dive.→
- Business AlertingWhen business metrics page. A production-focused deep dive.→
- Business DashboardsWhy they need owners. A production-focused deep dive.→
Part LXIII
Synthetic Monitoring
6 checks
- Synthetic Monitoring OverviewExternal probes. A production-focused deep dive.→
- HTTP Probeshttp_2xx module. A production-focused deep dive.→
- TLS Probesprobe_ssl_earliest_cert_expiry. A production-focused deep dive.→
- DNS ProbesResolution success and latency. A production-focused deep dive.→
- Multi-Step Journey ProbesEnd-to-end user flows. A production-focused deep dive.→
- Synthetic Cost and Where to RunExternal probes cost more than internal. A production-focused deep dive.→
Part LXIV
TLS Monitoring
6 checks
- TLS Expiry as a Metricprobe_ssl_earliest_cert_expiry. A production-focused deep dive.→
- TLS Expiry Alerting30 days, 7 days, 1 day thresholds. A production-focused deep dive.→
- TLS Handshake FailureWhen handshake fails. A production-focused deep dive.→
- TLS Protocol HealthForcing modern TLS. A production-focused deep dive.→
- Mixed Content DetectionHTTPS resources loaded over HTTP. A production-focused deep dive.→
- TLS Incident ResponseWhen an expiry fires. A production-focused deep dive.→
Part LXV
DNS Monitoring
6 checks
- DNS Resolution Metricdig-style probes. A production-focused deep dive.→
- External DNS MonitoringResolving authoritative servers. A production-focused deep dive.→
- Internal DNS MonitoringResolving cluster-internal names. A production-focused deep dive.→
- DNS Record ValidationMatching expected values. A production-focused deep dive.→
- Slow DNS DetectionA spike in resolution latency. A production-focused deep dive.→
- DNS Incident PatternsWhen to investigate DNS. A production-focused deep dive.→
Part LXVI
Observability Architecture for Production
6 checks
- The Production Observability TopologyWorkloads to collectors to backends to users. A production-focused deep dive.→
- The Collector StrategyWhere collectors live. A production-focused deep dive.→
- Storage ArchitectureLocal disks, object storage, retention. A production-focused deep dive.→
- TLS and AuthenticationSecuring the topology. A production-focused deep dive.→
- Tenants and EnvironmentsProduction, staging, dev. A production-focused deep dive.→
- Reference ArchitectureThe canonical production reference. A production-focused deep dive.→
Part LXVII
High Availability
6 checks
- High Availability BasicsStateless vs stateful components. A production-focused deep dive.→
- Stateless ReplicasAlertmanager, Grafana stateless. A production-focused deep dive.→
- Stateful HAPrometheus TSDB, Loki chunks. A production-focused deep dive.→
- Two Instances is Not HAThe two replicas failure shape. A production-focused deep dive.→
- HA EvidenceProving the system survives a single failure. A production-focused deep dive.→
- HA Cost Trade-offsWhen to skip HA. A production-focused deep dive.→
Part LXVIII
Prometheus HA
6 checks
- Duplicate ScrapingTwo Prometheus scraping the same targets. A production-focused deep dive.→
- External Labelsglobal.external_labels. A production-focused deep dive.→
- Alertmanager HAGossip protocol. A production-focused deep dive.→
- Thanos and Query LayerThe Thanos-side deduplication. A production-focused deep dive.→
- The Cost of HAStorage doubled. A production-focused deep dive.→
- The Split-Brain ProblemWhen two replicas both think they are primary. A production-focused deep dive.→
Part LXIX
Long-Term Metrics Storage
6 checks
- Why Long-Term Metrics StorageSingle-node Prometheus has limits. A production-focused deep dive.→
- Remote WritePrometheus to Thanos / Mimir / Cortex. A production-focused deep dive.→
- Thanos OverviewObject storage and query layer. A production-focused deep dive.→
- Mimir OverviewMulti-tenant horizontal scalability. A production-focused deep dive.→
- Cortex OverviewThe predecessor. A production-focused deep dive.→
- When to Scale MetricsTrigger criteria. A production-focused deep dive.→
Part LXX
Loki at Scale
6 checks
- Loki Microservices ModeThe enterprise deployment mode. A production-focused deep dive.→
- Querier ScalingScaling queriers. A production-focused deep dive.→
- CachingCaching layers. A production-focused deep dive.→
- Loki at Scale ConfigurationConfiguration choices for scale.→
- LimitsTenant and stream limits. A production-focused deep dive.→
- Loki Performance TroubleshootingDiagnosing Loki performance.→
Part LXXI
Tempo at Scale
6 checks
- Tempo Microservices ModeThe enterprise deployment mode. A production-focused deep dive.→
- Tempo Ingester ScalingScaling ingesters. A production-focused deep dive.→
- Tempo Compactor ScalingScaling compactors. A production-focused deep dive.→
- Tempo Storage ScalingObject storage. A production-focused deep dive.→
- Trace Load PatternsSpikey vs steady workloads. A production-focused deep dive.→
- Tempo Performance TuningQuery and ingest tuning. A production-focused deep dive.→
Part LXXII
Grafana HA
6 checks
- Shared DatabaseGrafana with a shared DB. A production-focused deep dive.→
- Session ConsistencySessions across replicas. A production-focused deep dive.→
- Grafana Load BalancingRound-robin / sticky sessions. A production-focused deep dive.→
- Plugin ConsistencySame plugins on each replica. A production-focused deep dive.→
- Configuration PropagationProvisioning the same way. A production-focused deep dive.→
- Grafana HA CostWhen HA matters and when it does not. A production-focused deep dive.→
Part LXXIII
Storage Architecture
6 checks
- Storage Architecture BasicsLocal disks, network volumes, object storage. A production-focused deep dive.→
- Local Disk for TSDBPrometheus local TSDB. A production-focused deep dive.→
- Object StorageLoki and Tempo chunks. A production-focused deep dive.→
- Throughput and IOPSStorage performance. A production-focused deep dive.→
- Capacity Planning for StorageCalculating the storage budget. A production-focused deep dive.→
- Storage TieringHot vs cold storage. A production-focused deep dive.→
Part LXXIV
Capacity Planning
6 checks
- Metrics CapacitySeries times samples times retention. A production-focused deep dive.→
- Logs CapacityBytes / sec times retention. A production-focused deep dive.→
- Traces CapacitySpans / sec times sampling. A production-focused deep dive.→
- Capacity HeadroomBuffer for spike. A production-focused deep dive.→
- Growth ModellingAnticipating growth. A production-focused deep dive.→
- Avoid False PrecisionOrder-of-magnitude estimation. A production-focused deep dive.→
Part LXXV
Performance
6 checks
- Scrape OverloadWhen scrape costs exceed capacity. A production-focused deep dive.→
- TSDB CompactionPrometheus TSDB compaction. A production-focused deep dive.→
- Query PressureQueries that consume capacity. A production-focused deep dive.→
- Cardinality BottlenecksWhen cardinality is the dominant cost. A production-focused deep dive.→
- Slow Object StorageWhen object storage is the bottleneck. A production-focused deep dive.→
- OTel Collector PressureOTel Collector bottlenecks. A production-focused deep dive.→
Part LXXVI
Cost Management
6 checks
- Telemetry Cost OverviewThe cost model. A production-focused deep dive.→
- Metric Cost DriversSeries times samples times retention. A production-focused deep dive.→
- Log Cost DriversBytes/sec times retention. A production-focused deep dive.→
- Trace Cost DriversSpans/sec times sampling times retention. A production-focused deep dive.→
- Cost ControlsSampling, retention, drop. A production-focused deep dive.→
- Cost MonitoringTracking telemetry cost over time. A production-focused deep dive.→
Part LXXVII
Security Architecture
6 checks
- Security OverviewWhere the platform is exposed. A production-focused deep dive.→
- Network ExposureListening interfaces. A production-focused deep dive.→
- AuthenticationHow users authenticate. A production-focused deep dive.→
- AuthorizationWho can see what. A production-focused deep dive.→
- TLS Across the PlatformEncrypting all traffic. A production-focused deep dive.→
- Secrets ManagementStoring credentials. A production-focused deep dive.→
Part LXXVIII
Securing Prometheus
6 checks
- Prometheus Network ExposureListening interface controls. A production-focused deep dive.→
- Prometheus AuthenticationBasic auth / bearer. A production-focused deep dive.→
- Prometheus TLSTLS for the web API. A production-focused deep dive.→
- Admin API Risksreload, quit endpoints. A production-focused deep dive.→
- Exporter AuthAuth on exporters. A production-focused deep dive.→
- Prometheus and SecretsSensitive values in metrics. A production-focused deep dive.→
Part LXXIX
Securing Grafana
6 checks
- Grafana AuthenticationLocal, LDAP, OAuth, SAML, OIDC. A production-focused deep dive.→
- Grafana RBACRoles and teams. A production-focused deep dive.→
- Datasource CredentialsSecrets in Grafana. A production-focused deep dive.→
- Anonymous Access DisciplineWhen anonymous is appropriate. A production-focused deep dive.→
- Plugin SecurityVetting third-party plugins. A production-focused deep dive.→
- Grafana Session SecurityCookie settings, session length. A production-focused deep dive.→
Part LXXX
Securing Loki
6 checks
- Loki AuthenticationTenant auth. A production-focused deep dive.→
- Tenant IsolationMulti-tenant boundaries. A production-focused deep dive.→
- S3 PermissionsLoki to S3 credentials. A production-focused deep dive.→
- Loki and PIIPII in logs. A production-focused deep dive.→
- Log InjectionWhen a user message becomes a log. A production-focused deep dive.→
- Loki Hardening ChecklistA summary. A production-focused deep dive.→
Part LXXXI
Securing Tempo
6 checks
- Trace Data SensitivityWhat traces carry. A production-focused deep dive.→
- Tempo Access ControlRestricting access. A production-focused deep dive.→
- Tempo Object Storage SecurityTempo to object store credentials. A production-focused deep dive.→
- Attribute RedactionRedacting PII in spans. A production-focused deep dive.→
- Tempo TLSTempo encryption. A production-focused deep dive.→
- Tempo Hardening ChecklistA summary. A production-focused deep dive.→
Part LXXXII
Secrets and Sensitive Telemetry
6 checks
- Sensitive Data in TelemetryThe risk. A production-focused deep dive.→
- Redaction StrategiesHow to redact. A production-focused deep dive.→
- Redaction ToolsOTel Collector, logfmt rewriters. A production-focused deep dive.→
- Secret Scanning TelemetryDetecting secrets in metrics / logs. A production-focused deep dive.→
- Auditing Telemetry for LeaksQuarterly review. A production-focused deep dive.→
- Leak Incident ResponseWhen a leak is discovered. A production-focused deep dive.→
Part LXXXIII
Multi-Tenancy
6 checks
- Tenant ModelsTenant boundaries. A production-focused deep dive.→
- Tenant IsolationHow isolation is enforced. A production-focused deep dive.→
- Tenant OnboardingAdding a new tenant. A production-focused deep dive.→
- Cross-Tenant AccessWhen tenants legitimately need cross access. A production-focused deep dive.→
- Tenant Cost AllocationInvoicing / showback. A production-focused deep dive.→
- Tenant AuditQuarterly review. A production-focused deep dive.→
Part LXXXIV
Configuration as Code
6 checks
- Config as Code BasicsWhat goes in Git. A production-focused deep dive.→
- Prometheus Config as Codeprometheus.yml in Git. A production-focused deep dive.→
- Alert Rules as Coderules/*.yml in Git. A production-focused deep dive.→
- Alertmanager Config as Codealertmanager.yml in Git. A production-focused deep dive.→
- Grafana Provisioning as Codedatasources + dashboards in Git. A production-focused deep dive.→
- Loki and Tempo Config as Codeloki.yaml, tempo.yaml in Git. A production-focused deep dive.→
Part LXXXV
CI Validation
6 checks
- CI Validation BasicsThe pipeline. A production-focused deep dive.→
- promtool check configConfiguration check. A production-focused deep dive.→
- promtool check rulesRule check. A production-focused deep dive.→
- promtool test rulesRule unit tests. A production-focused deep dive.→
- YAML LintLint configuration files. A production-focused deep dive.→
- End-to-End TestsStack tests. A production-focused deep dive.→
Part LXXXVI
Prometheus Rule Testing
6 checks
- Rule Test AnatomySeries, interval, alert. A production-focused deep dive.→
- Test Cases Per RuleWriting rule tests. A production-focused deep dive.→
- Recording Rule TestsTests for recording rules. A production-focused deep dive.→
- Rule Test CoverageCoverage discipline. A production-focused deep dive.→
- Regression TestsBug-driven test cases. A production-focused deep dive.→
- Rule Tests in CIWiring into the pipeline. A production-focused deep dive.→
Part LXXXVII
Alert Testing
6 checks
- Alert Testing BasicsWhy test alerts. A production-focused deep dive.→
- Synthetic Series for AlertsFeeding alerts synthetic data. A production-focused deep dive.→
- End-to-End Alert TestVerifying the receiver path. A production-focused deep dive.→
- Alert CanaryA scheduled alert that verifies the chain. A production-focused deep dive.→
- Alert Time-to-FireHow fast the alert should fire. A production-focused deep dive.→
- Alert Test CadenceQuarterly review of all alert tests. A production-focused deep dive.→
Part LXXXVIII
Dashboard Testing and Review
6 checks
- Dashboard Review BasicsWhy review dashboards. A production-focused deep dive.→
- Query CorrectnessVerify each panel query. A production-focused deep dive.→
- Variable ValidationVariables work across combinations. A production-focused deep dive.→
- Data Source AvailabilityEach datasource is up. A production-focused deep dive.→
- Unit ConsistencyUnits consistent across panels. A production-focused deep dive.→
- Dashboard Test CadenceQuarterly review. A production-focused deep dive.→
Part LXXXIX
Observability Platform Monitoring Itself
6 checks
- Who Monitors the MonitoringThe recursive problem. A production-focused deep dive.→
- Prometheus Healthup and friends. A production-focused deep dive.→
- Scrape Failure Detectionup{job="..."} == 0 alerts. A production-focused deep dive.→
- Rule Failure Detectionprometheus_rule_evaluation_failures_total. A production-focused deep dive.→
- Alertmanager DeliveryAlertmanager health. A production-focused deep dive.→
- Storage MonitoringDisk usage, object store health. A production-focused deep dive.→
Part XC
Meta-Monitoring
6 checks
- Meta-Monitoring BasicsMonitoring the monitors. A production-focused deep dive.→
- Separate Monitoring PlatformA second Prometheus for the first. A production-focused deep dive.→
- Federation for MetaPulling the platform Prometheus into a separate one. A production-focused deep dive.→
- Meta AlertsAlerts that fire when the meta is down. A production-focused deep dive.→
- Avoiding Circular AssumptionsDo not let meta depend on what it monitors. A production-focused deep dive.→
- Proving the Meta WorksTesting the meta monitoring. A production-focused deep dive.→
Part XCI
Backup Strategy
6 checks
- Backup BasicsWhat to back up. A production-focused deep dive.→
- Prometheus BackupTSDB snapshot, configuration. A production-focused deep dive.→
- Loki BackupObject store, ruler state, configuration. A production-focused deep dive.→
- Tempo BackupObject store, configuration. A production-focused deep dive.→
- Grafana BackupDatabase + provisioning. A production-focused deep dive.→
- Alertmanager BackupConfiguration + silences. A production-focused deep dive.→
Part XCII
Disaster Recovery
6 checks
- RPO and RTOSetting targets. A production-focused deep dive.→
- Prometheus LossRecovering from Prometheus loss. A production-focused deep dive.→
- Grafana LossRecovering from Grafana loss. A production-focused deep dive.→
- Loki LossRecovering from Loki loss. A production-focused deep dive.→
- Tempo LossRecovering from Tempo loss. A production-focused deep dive.→
- Object Storage LossWhen chunks / blocks go missing. A production-focused deep dive.→
Part XCIII
Upgrades
6 checks
- Upgrade BasicsThe shape of an upgrade. A production-focused deep dive.→
- Reading Release NotesWhat to look for. A production-focused deep dive.→
- CompatibilityCross-component compat. A production-focused deep dive.→
- Rollback StrategyHow to roll back. A production-focused deep dive.→
- Canary UpgradeOne component at a time. A production-focused deep dive.→
- Upgrade TestTesting the upgrade before production. A production-focused deep dive.→
Part XCIV
Prometheus Upgrades
6 checks
- Prometheus Upgrade BasicsWhat changes between versions. A production-focused deep dive.→
- TSDB MigrationWhen TSDB migrates. A production-focused deep dive.→
- Prometheus Config CompatibilityBackwards compat. A production-focused deep dive.→
- Prometheus Rule CompatibilityRules across versions. A production-focused deep dive.→
- Upgrade OrderA safe order. A production-focused deep dive.→
- Rollout ValidationProduction validation. A production-focused deep dive.→
Part XCV
Grafana Upgrades
6 checks
- Grafana Upgrade BasicsGrafana upgrades. A production-focused deep dive.→
- Database MigrationWhen migration is required. A production-focused deep dive.→
- Plugin CompatibilityPlugin match. A production-focused deep dive.→
- Dashboard CompatibilityDashboard JSON. A production-focused deep dive.→
- Provisioning CompatibilityProvisioning files. A production-focused deep dive.→
- Grafana Upgrade ValidationValidating after upgrade. A production-focused deep dive.→
Part XCVI
Loki Upgrades
6 checks
- Loki Upgrade BasicsLoki upgrades. A production-focused deep dive.→
- Loki Schema Configschema_config schema version. A production-focused deep dive.→
- Loki Storage MigrationWhen the storage layer changes. A production-focused deep dive.→
- Loki Config MigrationBackwards compat. A production-focused deep dive.→
- boltdb-shipper to TSDBThe TSDB migration. A production-focused deep dive.→
- Loki Upgrade ValidationValidation after upgrade. A production-focused deep dive.→
Part XCVII
Tempo Upgrades
6 checks
- Tempo Upgrade BasicsTempo upgrades. A production-focused deep dive.→
- Block Format CompatibilityWhen block format changes. A production-focused deep dive.→
- Receiver CompatibilityOTLP / Jaeger / Zipkin. A production-focused deep dive.→
- TraceQL CompatibilityWhen the query language changes. A production-focused deep dive.→
- Tempo Config ChangesRenamed, deprecated configs. A production-focused deep dive.→
- Tempo Upgrade ValidationValidation after upgrade. A production-focused deep dive.→
Part XCVIII
Troubleshooting Methodology
6 checks
- Methodology SummaryThe investigation loop. A production-focused deep dive.→
- Three Signals as ToolsEach signal in the investigation. A production-focused deep dive.→
- Investigation Time BudgetBounding the search. A production-focused deep dive.→
- Hypothesis and Evidence DisciplineAvoiding red herrings. A production-focused deep dive.→
- Mitigation vs Root CauseBoth happen. A production-focused deep dive.→
- Post-Incident ReviewAfter the incident. A production-focused deep dive.→
Part XCIX
Missing Metrics
6 checks
- Missing Metrics AnatomyWhere to look. A production-focused deep dive.→
- Exporter DownWhen the exporter is not running. A production-focused deep dive.→
- Network BlockWhen the network prevents scraping. A production-focused deep dive.→
- Scrape Config ErrorWrong configuration. A production-focused deep dive.→
- Relabel DropsWhen relabeling removes targets. A production-focused deep dive.→
- Query WrongWhen the metric exists, but the query is wrong. A production-focused deep dive.→
Part C
Missing Logs
6 checks
- Missing Logs AnatomyWhere to look. A production-focused deep dive.→
- Application Not EmittingWhen the application does not log. A production-focused deep dive.→
- Collector Not RunningWhen the collector is not running. A production-focused deep dive.→
- Pipeline BrokenWhen logs reach Loki but with wrong labels. A production-focused deep dive.→
- Loki Query WrongWhen the query is wrong. A production-focused deep dive.→
- Loki Rate LimitedWhen Loki drops logs. A production-focused deep dive.→
Part CI
Missing Traces
6 checks
- Missing Traces AnatomyWhere to look. A production-focused deep dive.→
- Instrumentation Not RunningWhen the SDK is not active. A production-focused deep dive.→
- Propagation BrokenWhen trace IDs are not carried. A production-focused deep dive.→
- Collector DownWhen the collector is unavailable. A production-focused deep dive.→
- Exporter BrokenWhen the exporter cannot reach Tempo. A production-focused deep dive.→
- Sampling Too AggressiveWhen too few traces are sampled. A production-focused deep dive.→
Part CII
Slow Queries
6 checks
- Slow Queries AnatomyWhy queries are slow. A production-focused deep dive.→
- Large Range QueriesWide time windows. A production-focused deep dive.→
- High-Cardinality QueriesWhen series count is high. A production-focused deep dive.→
- Expensive RegexPromQL / LogQL regexes. A production-focused deep dive.→
- Poor AggregationAggregating at the wrong level. A production-focused deep dive.→
- Storage BottleneckStorage I/O. A production-focused deep dive.→
Part CIII
Alert Failure
6 checks
- Alert Failure AnatomyWhere alerts can fail. A production-focused deep dive.→
- Telemetry MissingWhen the metric is missing. A production-focused deep dive.→
- Rule WrongWhen the rule is wrong. A production-focused deep dive.→
- Threshold WrongWhen the threshold is wrong. A production-focused deep dive.→
- Alertmanager Routing WrongWhen Alertmanager routes wrong. A production-focused deep dive.→
- Receiver DownWhen the receiver is down. A production-focused deep dive.→
Part CIV
False Positive Alert
6 checks
- False Positive AnatomyWhen alerts fire but there is no incident. A production-focused deep dive.→
- Threshold Too LowAdjusting the threshold. A production-focused deep dive.→
- for: Too ShortAdjusting for:. A production-focused deep dive.→
- Rule Too BroadTighten the matcher. A production-focused deep dive.→
- False Positive InvestigationWhy did it fire? A production-focused deep dive.→
- Tuning ProcessThe discipline of tuning. A production-focused deep dive.→
Part CV
Cardinality Incident
6 checks
- Cardinality Incident AnatomyMemory spikes; OOM. A production-focused deep dive.→
- DetectionDetecting the incident. A production-focused deep dive.→
- InvestigationFinding the cause. A production-focused deep dive.→
- MitigationStabilising the platform. A production-focused deep dive.→
- RecoveryRestoring Prometheus. A production-focused deep dive.→
- Cardinality Budget Recurring ReviewQuarterly cardinality review.→
Part CVI
Log Ingestion Incident
6 checks
- Log Ingestion Spike AnatomyVolume rises sharply. A production-focused deep dive.→
- Debug Logging EnabledA code change enabled debug logs. A production-focused deep dive.→
- Log LoopA retry loop generating logs. A production-focused deep dive.→
- New High-Volume ServiceA new tenant. A production-focused deep dive.→
- Loki Capacity IncidentWhen Loki storage fills.→
- Post-Incident Cost ReviewReviewing cost after the incident.→
Part CVII
Trace Volume Incident
6 checks
- Trace Volume AnatomyWhen trace ingest climbs. A production-focused deep dive.→
- Sampling MisconfigA changed sampling rate. A production-focused deep dive.→
- Tail Sampler ErrorTail sampler accepts all. A production-focused deep dive.→
- Service Mesh EnabledSidecar tracing. A production-focused deep dive.→
- Mitigation and RecoveryStabilising Tempo. A production-focused deep dive.→
- Tempo Rotation BehaviorHow metrics rotate.→
Part CVIII
Clock Skew
6 checks
- Clock Skew AnatomyWhen clocks disagree. A production-focused deep dive.→
- Effect on LogsLog ordering. A production-focused deep dive.→
- Effect on TracesSpan ordering. A production-focused deep dive.→
- Detecting SkewDetecting the skew. A production-focused deep dive.→
- MitigationRestoring sync. A production-focused deep dive.→
- Preventionchrony / systemd-timesyncd alerts. A production-focused deep dive.→
Part CIX
Incident Investigation Workflows
6 checks
- Investigation AnatomyEnd-to-end. A production-focused deep dive.→
- Example: Checkout LatencyA worked example. A production-focused deep dive.→
- Example: Error Rate SpikeA second example. A production-focused deep dive.→
- Example: Storage FullA third example. A production-focused deep dive.→
- Workflow TemplatesTemplates for common incidents. A production-focused deep dive.→
- Workflow IterationImproving after each incident. A production-focused deep dive.→
Part CX
Observability During Major Incidents
6 checks
- Major Incident Observability OverviewWhen production is broken. A production-focused deep dive.→
- Incident DashboardsDedicated panels. A production-focused deep dive.→
- Change AnnotationsMarking changes. A production-focused deep dive.→
- Query DisciplineAvoiding runaway queries. A production-focused deep dive.→
- Evidence PreservationSaving the timeline. A production-focused deep dive.→
- Observability During the IncidentWhen observability itself fails. A production-focused deep dive.→
Part CXI
Observability Anti-Patterns
6 checks
- Monitor Everything Without PurposeThe over-collection anti-pattern. A production-focused deep dive.→
- Alert on Every MetricThe alert-fatigue anti-pattern. A production-focused deep dive.→
- Unlimited RetentionUnbounded storage. A production-focused deep dive.→
- High-Cardinality LabelsThe cardinality anti-pattern. A production-focused deep dive.→
- Secrets in LogsThe PII anti-pattern. A production-focused deep dive.→
- Observability Without OwnerThe unowned anti-pattern. A production-focused deep dive.→
Part CXII
Production Observability Operating Model
6 checks
- The Operating ModelWho owns what. A production-focused deep dive.→
- Shared vs Dedicated PlatformSharing the platform. A production-focused deep dive.→
- Service OwnershipThe application team owns the service. A production-focused deep dive.→
- Dashboard OwnershipDashboards need owners. A production-focused deep dive.→
- Alert OwnershipAlerts need owners. A production-focused deep dive.→
- Runbook OwnershipRunbooks need owners. A production-focused deep dive.→
Part CXIII
Documentation and Runbooks
6 checks
- Documentation from TelemetryAlerts should link to docs. A production-focused deep dive.→
- What Failed?Every alert states the failure. A production-focused deep dive.→
- ImpactEvery alert states the impact. A production-focused deep dive.→
- Dashboard LinkEvery alert links to a dashboard. A production-focused deep dive.→
- Runbook LinkEvery alert links to a runbook. A production-focused deep dive.→
- OwnerEvery alert states the owner. A production-focused deep dive.→
Part CXIV
Final Production Reference Architecture
6 checks
- Reference Architecture AnatomyA canonical production observability stack. A production-focused deep dive.→
- The Workload LayerProduction systems. A production-focused deep dive.→
- The Collector Layernode_exporter, exporters, OTel Collector, Alloy. A production-focused deep dive.→
- The Backend LayerPrometheus, Loki, Tempo, Mimir. A production-focused deep dive.→
- The Presentation LayerGrafana, Alertmanager. A production-focused deep dive.→
- The Operations LayerPeople, processes, observability. A production-focused deep dive.→