Signals — metrics, logs, traces, events in depth

The four observability signals have distinct types and structures. The metrics have counter, gauge, histogram, and summary types. The logs have levels. The traces have spans. The events have types. This lesson walks each signal in depth, the production patterns, and the discipline.

The metric types

The Prometheus metric types:

flowchart LR
    A[Metric types] --> B[Counter]
    A --> C[Gauge]
    A --> D[Histogram]
    A --> E[Summary]
    B --> F[Monotonically increasing]
    C --> G[Arbitrary value]
    D --> H[Bucket distribution]
    E --> I[Quantile summary]

Each type has a specific use case.

The counter

The counter is monotonically increasing:

httpRequestsTotal = prometheus.NewCounter(prometheus.CounterOpts{
  Name: "http_requests_total",
  Help: "Total number of HTTP requests",
})

httpRequestsTotal.Inc()
httpRequestsTotal.Add(100)

The counter is reset on restart; the rate is the meaningful value:

# Requests per second
rate(http_requests_total[5m])

The gauge

The gauge is an arbitrary value:

activeConnections = prometheus.NewGauge(prometheus.GaugeOpts{
  Name: "active_connections",
  Help: "Number of active connections",
})

activeConnections.Inc()
activeConnections.Dec()
activeConnections.Set(42)

The gauge is not reset on restart; the current value is the meaningful value.

The histogram

The histogram is a bucket distribution:

requestDuration = prometheus.NewHistogram(prometheus.HistogramOpts{
  Name: "http_request_duration_seconds",
  Help: "Duration of HTTP requests",
  Buckets: []float64{0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10},
})

requestDuration.Observe(0.42)

The histogram is useful for the latency distribution:

# p99 latency
histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))

The summary

The summary is a quantile summary:

requestDuration = prometheus.NewSummary(prometheus.SummaryOpts{
  Name: "http_request_duration_seconds",
  Help: "Duration of HTTP requests",
  Objectives: map[float64]float64{
    0.5: 0.05,
    0.9: 0.01,
    0.99: 0.001,
  },
})

The summary is computed at the agent; the histogram is computed at the query.

The log levels

The log levels:

The convention is the 12-factor app; the level is the discipline.

log.Debug("received request", "path", "/api/users")
log.Info("response sent", "status", 200, "duration_ms", 100)
log.Warn("rate limit approaching", "current", 100, "limit", 120)
log.Error("failed to connect to database", "err", err)

The log structure

The structured logs are JSON:

{
  "timestamp": "2026-08-16T10:00:00.000Z",
  "level": "INFO",
  "message": "response sent",
  "service": "nginx",
  "status": 200,
  "duration_ms": 100,
  "trace_id": "abc123"
}

The structured logs are queryable in Loki.

The trace spans

The trace spans are the request journey:

{
  "trace_id": "abc123",
  "span_id": "def456",
  "name": "GET /api/users",
  "start_time": "2026-08-16T10:00:00.000Z",
  "duration_ms": 100,
  "attributes": {
    "http.method": "GET",
    "http.url": "/api/users",
    "http.status_code": 200
  },
  "parent_span_id": "ghi789"
}

The spans form a tree:

sequenceDiagram
    participant C as Client
    participant A as API
    participant B as Backend
    participant D as Database
    C->>A: GET /api/users (root span)
    A->>B: forward request (child span)
    B->>D: SELECT * FROM users (child span)
    D-->>B: result
    B-->>A: response
    A-->>C: 200 OK

The span tree is the trace.

The event types

The Kubernetes event types:

The events are the cluster’s change log.

kubectl get events --field-selector type=Warning

The signal-to-noise ratio

The signal-to-noise ratio is the discipline:

# Good: structured logs with relevant fields
{"level": "INFO", "message": "response sent", "status": 200, "duration_ms": 100, "trace_id": "abc123"}

# Bad: unstructured logs with no context
2026-08-16T10:00:00  INFO  response sent

The structured logs are queryable; the unstructured logs are not.

Cross-course references

The Observability course covers the tools in detail.

• The Prometheus course (Part LXXXVIII) covers the metrics.
• The Loki course (Part LXXXIX) covers the logs.
• The Jaeger course (Part XC) covers the traces.
• The Events course (Part XCI) covers the events.

Quiz

Production discipline

• Choose the right metric type. Counter, gauge, histogram, summary.
• Emit structured logs. JSON with relevant fields.
• Use the OpenTelemetry SDK. Standard for traces.
• Configure the events collection. event-exporter.
• Document the signal design. Per workload.
• Test the four pillars. End-to-end observability.

The four signals are the cluster’s observability. Operating it well is choosing the right metric types, emitting structured logs, using the OTel SDK, and configuring the events collection.