Continuous profiling and tracing — the performance insight

Neither a profile nor a trace can be collected retroactively. When a latency incident ends the evidence ends with it, and the team is left arguing over dashboards that establish only that something was slow. That is why profiling and tracing are standing infrastructure rather than incident tools, and this lesson covers what has to be instrumented and running in advance, and what the sampling costs on a busy cluster.

The continuous profiling

The continuous profiling is the application’s CPU and memory profile. The profiling is collected by a profiler (e.g., pprof, Pyroscope) and visualised as flame graphs.

flowchart LR
    A[Application] --> B[Profiler]
    B --> C[Profile data]
    C --> D[Flame graph]

The profiling is the application’s performance insight.

The distributed tracing

The distributed tracing is the application’s request flow. The tracing is collected by a tracer (e.g., OpenTelemetry, Jaeger) and visualised as spans.

flowchart LR
    A[Client] --> B[Application]
    B --> C[Database]
    B --> D[External service]
    B --> E[Span tree]

The tracing is the application’s request flow.

The diagnostic

The canonical diagnostic:

# Substitute your own values before running:
NS=production                            # namespace holding both Pods
POD=checkout-api-7d9f6c8b45-r4nq2        # the application Pod
TRACING_POD=jaeger-query-6c58d9fbb4-8vt7k   # the trace UI Pod

# 1. Check the profiling
kubectl port-forward -n "$NS" "$POD" 8080:8080
# Browse to http://localhost:8080/debug/pprof

# 2. Check the tracing
kubectl port-forward -n "$NS" "$TRACING_POD" 16686:16686
# Browse to http://localhost:16686

# 3. Check the application's metrics
kubectl port-forward -n "$NS" "$POD" 8080:8080
# Browse to http://localhost:8080/metrics

# 4. Check the logs
kubectl logs -n "$NS" "$POD" --tail=200

The diagnostic is the profiling, the tracing, the metrics, and the logs.

Common failures

• Profiler not running. The profiler is not running. The remediation is to enable the profiler.
• Tracer not configured. The tracer is not configured. The remediation is to configure the tracer.
• Application not instrumented. The application is not instrumented. The remediation is to instrument the application.
• Continuous profiling overhead. The profiler is using too much CPU. The remediation is to reduce the profiler’s sampling rate.

flowchart TD
    A[Performance insight missing] --> B{Profiler running?}
    B -->|No| C[Enable the profiler]
    B -->|Yes| D{Tracer configured?}
    D -->|No| E[Configure the tracer]
    D -->|Yes| F{Application instrumented?}
    F -->|No| G[Instrument the application]
    F -->|Yes| H{Profiler overhead?}
    H -->|Yes| I[Reduce the sampling rate]
    H -->|No| J[Unknown]

The flame graph

The flame graph is the application’s CPU profile. The x-axis is the time; the y-axis is the call stack. The widest layer is the hottest function.

flowchart TD
    A[main] --> B[process_request]
    B --> C[query_database]
    B --> D[call_external]
    C --> E[read_file]
    C --> F[parse_json]

The flame graph is the application’s CPU profile.

The span

The span is the tracing’s atomic unit. The span represents a single operation in the request’s flow.

flowchart LR
    A[Client] -->|Span 1| B[Application]
    B -->|Span 2| C[Database]
    B -->|Span 3| D[External service]

The span is the tracing’s atomic unit.

The remediation

The remediation depends on the cause:

# Option 1: Enable the profiler
# (application-specific)

# Option 2: Configure the tracer
# (tracer-specific)

# Option 3: Instrument the application
# (application-specific)

# Option 4: Reduce the sampling rate
# (profiler-specific)

The remediation is the performance insight.

Production discipline

A performance insight issue is the cluster’s hypothesis. The discipline is to walk the 11-step methodology applied to the performance layer, identify the cause, apply the remediation. The performance is the cluster’s user experience; the remediation is the performance insight.

• Enable the profiler. The profiler is the CPU/memory insight.
• Configure the tracer. The tracer is the request flow insight.
• Instrument the application. The instrumentation is the application’s insight.

Quiz