Latency and saturation — the application performance

A report that the application is slow names a symptom and no layer, and the same symptom comes from a saturated database connection pool, a CPU limit the container is being throttled against, and an external API that has started taking two seconds. The four golden signals localise it before you change anything: latency says how bad, traffic says whether the load changed, errors say whether it is failing as well as slow, and saturation says which resource has run out. Skipping that step is how a team scales a Deployment that was never resource-bound.

The four golden signals

The four golden signals are the application performance metrics:

1. Latency. The time to serve a request.
2. Traffic. The number of requests per second.
3. Errors. The number of failed requests.
4. Saturation. The degree of resource utilisation.

flowchart TD
    A[Application] --> B[Latency]
    A --> C[Traffic]
    A --> D[Errors]
    A --> E[Saturation]

The four golden signals are the application’s performance.

The diagnostic

The canonical diagnostic:

# Substitute your own values before running:
NS=production
POD=checkout-api-5f9c7d8b6c-2xk9p

# 1. Check the application's metrics
kubectl port-forward -n "$NS" "$POD" 8080:8080
# Browse to http://localhost:8080/metrics

# 2. Check the application's logs
kubectl logs -n "$NS" "$POD" --tail=200

# 3. Check the application's traces
# (tracing-specific)

# 4. Check the resource usage
kubectl top pod "$POD" -n "$NS"

# 5. Check the events
kubectl get events -n "$NS" --field-selector involvedObject.name="$POD"

The diagnostic is the application’s metrics, the logs, the traces, the resource usage, and the events.

Common failures

• High latency. The application is slow. The remediation is to investigate the application’s bottleneck.
• High errors. The application is failing. The remediation is to investigate the error rate.
• High saturation. The application is resource-bound. The remediation is to scale or optimise the application.
• CPU throttling. The application’s CPU is throttled. The remediation is to increase the CPU limit.

flowchart TD
    A[Performance issue] --> B{High latency?}
    B -->|Yes| C[Investigate the bottleneck]
    B -->|No| D{High errors?}
    D -->|Yes| E[Investigate the errors]
    D---|No| F{High saturation?}
    F -->|Yes| G[Scale or optimise]
    F -->|No| H[Unknown]

The latency

The latency is the application’s response time. The latency is composed of:

• Network latency. The time to send the request to the application.
• Application latency. The time to process the request.
• Database latency. The time to query the database.
• External latency. The time to call external services.

# Substitute your own values before running:
SVC=checkout-api
NS=production

# Check the latency
curl -w "time_total: %{time_total}\n" -o /dev/null -s "http://$SVC.$NS.svc.cluster.local"

The latency is the application’s bottleneck.

The saturation

The saturation is the application’s resource utilisation. The saturation is the degree of “fullness” of the resources.

# Substitute your own values before running:
POD=checkout-api-5f9c7d8b6c-2xk9p
NS=production

# Check the saturation
kubectl top pod "$POD" -n "$NS"

The saturation is the application’s resource pressure.

The remediation

The remediation depends on the cause:

# Substitute your own values before running:
DEPLOY=checkout-api
NS=production
REPLICAS=6

# Option 1: Scale the application
kubectl scale deployment "$DEPLOY" --replicas="$REPLICAS" -n "$NS"

# Option 2: Increase the resources
kubectl set resources deployment "$DEPLOY" -n "$NS" --limits=cpu=1,memory=1Gi

# Option 3: Optimise the application
# (application-specific)

# Option 4: Add caching
# (caching-specific)

The remediation is the performance recovery.

Production discipline

Capture all four signals before changing anything, and record which one moved first — that ordering is the evidence, and it is gone once a fix perturbs the system. Raising a limit or adding replicas against a latency number alone routinely moves the bottleneck somewhere less visible rather than removing it.

• Read the metrics and the logs together. kubectl top pod and the application’s /metrics endpoint name the saturated resource; the logs say whether it is failing as well as slowing.

Quiz