Metric reliability — the metric quality

Metric reliability is the discipline of ensuring the metrics are accurate, consistent, and fresh. The label design is the foundation. The production patterns are the SLO for the metrics. This lesson walks the metric reliability, the label design, the freshness, and the production patterns.

What is metric reliability

The metric reliability:

flowchart LR
    A[Metric] --> B[Quality]
    A --> C[Consistency]
    A --> D[Accuracy]
    A --> E[Freshness]

The metric reliability is the four pillars.

The metric quality

The metric quality:

Quality = accuracy + consistency + freshness

The quality is the foundation.

The accuracy

The accuracy:

# The metric value matches the reality.
http_requests_total: 1000 actual = 1000 reported

The accuracy is the input for the alerts.

The consistency

The consistency:

# The metric value is consistent across time.
http_requests_total: monotonically increasing

The consistency is the input for the rate queries.

The freshness

The freshness:

# The metric value is recent.
scrape_interval: 15s
max_age: 15s

The freshness is the input for the alerts.

The label design

The label design:

# Bad: high cardinality
labels:
  user_id: abc123
  request_id: def456

# Good: low cardinality
labels:
  service: nginx
  status: 2xx

The label design is the foundation.

The metric naming

The metric naming:

# Good: descriptive, units
http_requests_total: counter
http_request_duration_seconds: histogram
node_cpu_usage: gauge

# Bad: ambiguous, no units
requests: ?
latency: ?

The metric naming is the contract.

The metric annotations

The metric annotations:

# HELP: what the metric represents
# TYPE: counter, gauge, histogram, summary

The annotations are the metadata.

The freshness SLO

The freshness SLO:

# The metric is fresh within N seconds
scrape_interval: 15s
staleness_threshold: 60s

The freshness SLO is the production contract.

The metric validation

The metric validation:

# Verify the metric is fresh
promtool query instant http://prometheus:9090 'http_requests_total'

# Verify the metric is consistent
promtool query instant http://prometheus:9090 'rate(http_requests_total[5m])'

The validation is direct.

The production patterns

The production patterns:

flowchart LR
    A[Metric design] --> B[Low cardinality]
    B --> C[Semantic naming]
    C --> D[Annotations]
    D --> E[Freshness SLO]
    E --> F[Validation]

The pattern is the production discipline.

The cross-course references

• The Prometheus course covers the metric design.
• The Grafana course covers the dashboards.
• The Alertmanager course covers the alerts.

Quiz

Production discipline

• Design the labels. Low cardinality.
• Use semantic naming. With units.
• Add the annotations. HELP, TYPE.
• Configure the freshness SLO. The scrape interval.
• Validate the metrics. promtool query instant.
• Document the metric design. The pattern.

The metric reliability is the production discipline. Operating it well is the label design, the naming, the freshness, and the production patterns.