Dependency Performance

A checkout page is slow. The RED panel shows the checkout service p95 at 1.8 seconds. The hypothesis is “the dependency is slow.” Which dependency? The cart service, the inventory service, the payment service, the database, the cache. The investigation cannot start without a panel that breaks the checkout latency down by dependency. That panel is the dependency performance view.

Dependency performance is the third axis of the application layer. RED tells the on-call that the service is slow. USE tells the on-call that the host is slow. The dependency view tells the on-call which external call is slow. The shape is the same — rate, errors, duration — but the unit of analysis is the edge, not the node.

What it is

A dependency performance budget is the per-dependency latency commitment that the application can rely on. It is not a service-level objective; it is a budget that the upstream team agrees to. The budget is published as a number in the application’s runbook: “the payment-svc dependency has a 200 ms p95 latency budget and a 99.5% availability budget.”

The four dimensions of the dependency performance discipline are:

1. Client-side latency — the time the caller waits for the dependency to respond, measured at the caller. This is the user-visible latency of the dependency edge. The canonical metric is http.client.request.duration for HTTP, or db.client.operation.duration for SQL.
2. Server-side latency — the time the dependency handles the request, measured inside the dependency service. This is the cause of the client-side latency. The canonical metric is the dependency’s own http.server.request.duration.
3. Error rate — the rate of failed dependency calls, including timeouts. A timeout is a dependency error, not a caller error.
4. Saturation — the queue depth or in-flight count of dependency calls. The caller waits because the dependency is saturated; the cause is upstream.

The discipline is to measure all four dimensions per edge, with the same RED triad applied to the dependency as if it were its own service. The difference is the lens: RED measures the service as the user sees it; dependency performance measures the same service as the caller’s dependencies see it.

The four dimensions are split between two sides of the network. The client-side latency is in the caller’s metric set; the server-side latency is in the dependency’s metric set. Both must be visible in the same dashboard row for the dependency to be investigable.

Why a sysadmin cares

The single most common shape of a runtime regression in a microservice is a dependency regression. The service the on-call owns is healthy; the dependency it calls is slow. The on-call sees the symptom in the owner’s RED panel; the cause is in someone else’s service. Without the dependency view, the on-call has to ssh into the dependency’s host and read the metric set there. With the dependency view, the on-call reads the latency in their own dashboard and points the dependency team at the right server-side metric.

Three operational problems disappear when dependency performance is in place:

1. The “is it me or them?” question. The on-call sees the checkout p95 at 1.8 s. The dependency view shows the payment-svc client-side latency at 1.75 s and the cart inventory at 12 ms. The dependency is the cause. The on-call pages the payments team with a specific edge and a specific number.
2. The retry storm. A dependency is slow. The client retries three times. The retry amplifies the load on the dependency by 3x. The dependency now times out on more requests. The client retries 3x again. The dependency collapses. The dependency performance view shows the per-dependency retry rate; the alert catches the amplification before the collapse.
3. The configuration drift. The deploy at 14:18 raised the payment-svc dependency timeout from 1.0 s to 4.0 s. The checkout service is configured to wait 2.0 s. The checkout begins to time out on every request that takes more than 2 s. The dependency view shows the checkout-side timeout configuration as a label on the metric; the alert catches the drift before the next deploy.

How it works

The mental model is that every application has a fingerprint of dependencies. The fingerprint is the set of edges that the application’s request path touches. The dependency performance view is the latency, error rate, and saturation of each edge.

  checkout service
       |
       +-- cart-svc        (HTTP)        -- 12 ms p95
       |
       +-- inventory-svc   (HTTP)        -- 28 ms p95
       |
       +-- payment-svc     (HTTP)        -- 1820 ms p95  <-- slow
       |
       +-- postgres        (SQL)         -- 4 ms p95
       |
       +-- redis           (RESP)        -- 1 ms p95

The shape is the edge — the dependency name, the protocol, the operation, the destination attribute. The metric is the client-side duration from the caller’s view; the cause is the server-side duration from the dependency’s view.

The four-dimension split is what makes the investigation fast. The on-call who reads the dependency view in thirty seconds can answer “is this client-side or server-side?” by comparing the two latencies. If the client-side is 1.8 s and the server-side is 12 ms, the dependency is waiting on its downstream (a database, a queue, a third-party API). The on-call who reads the view can page the dependency team with the right diagnosis on the first message.

Under the hood

How to configure it

The dependency performance view is configured in three places: the application SDK at startup, the Collector pipeline, and the Prometheus recording rule.

1. The SDK at service startup

A Python service with OpenTelemetry 0.110.x and the manual configuration for an outbound HTTP client:

# app.py -- dependency instrumentation
import httpx
from opentelemetry.instrumentation.httpx import (
    HTTPXClientInstrumentor,
)
from opentelemetry import metrics
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import (
    PeriodicExportingMetricReader,
)
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import (
    OTLPMetricExporter,
)
from opentelemetry.sdk.resources import Resource

resource = Resource.create({
    "service.name": "checkout",
    "service.version": "1.4.2",
})

# SEVERITY: CONFIGURATION -- OTLP endpoint
reader = PeriodicExportingMetricReader(
    exporter=OTLPMetricExporter(
        endpoint="otel-collector.observability.svc:4317",
        insecure=True,
    ),
    export_interval_millis=10_000,
)
provider = MeterProvider(resource=resource, metric_readers=[reader])
metrics.set_meter_provider(provider)

# SEVERITY: CONFIGURATION -- registers the HTTP client
# instrument. After this call, every httpx.Client call
# emits http.client.request.duration.
HTTPXClientInstrumentor().instrument()

The instrument is registered once at startup. The metric is emitted on every outbound call. The server.address attribute is set automatically from the URL hostname.

For the database, the equivalent is the SQLAlchemy instrument:

from opentelemetry.instrumentation.sqlalchemy import (
    SQLAlchemyInstrumentor,
)

engine = create_engine("postgresql://app:secret@db:5432/checkout")
SQLAlchemyInstrumentor().instrument(
    engine=engine,
    # The dependency name is the database name
    # attribute. Bounded cardinality.
    enable_commenter=True,
)

The db.client.operation.duration metric is emitted on every query. The db.namespace attribute is the database name (checkout); the db.statement is dropped by the configured view to avoid unbounded cardinality.

2. The Collector pipeline

The Collector drops the unbounded attributes and forwards the bounded ones:

# /etc/otelcol-contrib/config.yaml
processors:
  attributes/dependency:
    actions:
      - key: http.url
        action: delete
      - key: url.full
        action: delete
      - key: db.statement
        action: delete
      - key: db.query.text
        action: delete
  batch:
    timeout: 10s

service:
  pipelines:
    metrics:
      processors: [attributes/dependency, batch]

The four delete actions are the guardrail. The http.url and url.full attributes are the un-templated URL; db.statement and db.query.text are the SQL text. All four are dropped before the data reaches Prometheus.

3. The Prometheus recording rule

The dependency performance view is promoted to a recording rule:

# /etc/prometheus/rules/dependency.rules
groups:
- name: dependency
  interval: 30s
  rules:
  - record: dependency:client_request_rate:rate5m
    expr: |
      sum by (service_name, server_address, http_request_method) (
        rate(http_client_request_duration_seconds_count[5m])
      )

  - record: dependency:client_request_duration:p95
    expr: |
      histogram_quantile(0.95,
        sum by (service_name, server_address, http_request_method, le) (
          rate(http_client_request_duration_seconds_bucket[5m])
        )
      )

  - record: dependency:client_error_rate:rate5m
    expr: |
      sum by (service_name, server_address) (
        rate(http_client_request_duration_seconds_count{
          http_response_status_code=~"5.."
        }[5m])
      )

  - record: dependency:open_connections:current
    expr: |
      sum by (service_name, server_address) (
        http_client_open_connections
      )

The promtool validator confirms the rule syntax:

# SEVERITY: READ-ONLY
promtool check rules /etc/prometheus/rules/dependency.rules

Expected output:

SUCCESS: /etc/prometheus/rules/dependency.rules

How to validate it

Validate that the dependency performance view is live with three checks.

1. The client-side metric is emitted for the expected dependency.

# SEVERITY: READ-ONLY
curl -s 'http://prometheus:9090/api/v1/query?query=count%20by%20(server_address)%20(http_client_request_duration_seconds_count{service_name%3D"checkout"})'

Expected output:

server_address
cart-svc.internal       1
inventory-svc.internal  1
payment-svc.internal    1
postgres                1
redis                   1

Five dependencies, one series each. The fingerprint is correct.

2. The client-side latency is bounded.

# SEVERITY: READ-ONLY
curl -s 'http://prometheus:9090/api/v1/query?query=dependency:client_request_duration:p95{service_name%3D"checkout"}'

Expected output:

server_address
cart-svc.internal       0.012
inventory-svc.internal  0.028
payment-svc.internal    0.182
postgres                0.004
redis                   0.001

The payment-svc latency at 182 ms is within its 200 ms budget. The other dependencies are well below budget.

3. The error rate is recorded.

# SEVERITY: READ-ONLY
curl -s 'http://prometheus:9090/api/v1/query?query=dependency:client_error_rate:rate5m{service_name%3D"checkout"}'

Expected output:

server_address
cart-svc.internal       0
inventory-svc.internal  0
payment-svc.internal    0
postgres                0
redis                   0

A zero error rate is the healthy shape. The alert should fire on a non-zero rate for 5 minutes.

How it can fail

Four specific failure shapes appear regularly in dependency performance instrumentation:

1. Cross-service mis-attribution. The on-call sees the checkout p95 at 1.8 s, opens the dependency view, and sees the payment-svc dependency at 1.8 s. The on-call pages the payments team. The payments team opens their server-side view and sees the payment-svc server-side latency at 12 ms. The cause is not in the payment-svc handler; the cause is in the payment-svc’s downstream call to a third-party API. Symptom: the dependency view shows the latency; the server-side view is fast; the investigation stalls.
2. Retry storms. The on-call sees the dependency error rate at 0.1; the retry configuration is set to 3. The retry amplifies the load by 3x. The dependency’s open_connections counter climbs. The retry exhaustion and the connection pool exhaustion are correlated. The alert should fire on the amplified rate, not the original rate. Symptom: error rate doubling every 10 minutes.
3. Connection pool exhaustion. The application has a connection pool of 20; the dependency is loaded with 50 concurrent requests. The pool exhausts. The dependency open_connections counter stays at 20; the request latency climbs because the application is queueing internally. The dependency view shows the latency but not the pool exhaustion. Symptom: dependency latency climbs with open_connections flat.
4. Timeouts that mask the cause. The dependency is configured to time out at 2.0 s. The application sees the timeout as a 503 error. The dependency’s server-side latency is 4.0 s but the client never sees it — the client times out first. The dependency’s server-side panel is fast because the metric is on the successful requests; the 4.0 s requests are the failures. Symptom: client-side latency capped at 2.0 s; server-side latency 12 ms; the actual cause is invisible to both panels.

How to troubleshoot it

When the dependency view says something the operator does not believe, the diagnostic order is:

1. Confirm the dependency name is the canonical name. The server.address attribute may be a hostname, an IP, or a service mesh sidecar. The dashboard should resolve it to the canonical service name.
2. Confirm the client-side and server-side latencies are consistent. A client-side 1.8 s with a server-side 12 ms means the dependency is waiting on its own downstream. The dashboard should show both values.
3. Confirm the metric is the client-side, not the server-side. The two metric names are similar (http.client.request.duration vs http.server.request.duration). A label mix-up produces a panel that drifts from the actual dependency view.
4. Confirm the cardinality is bounded. The server_address attribute should be a small, known set. A new address appearing in the panel is a sign of a dependency that has not been registered in the runbook.
5. Confirm the timeouts are configured on the caller side. A caller-side timeout of 2.0 s caps the client-side latency; the actual dependency latency is hidden. The dependency.timeout label should be on the metric.

Security implications

The dependency metric set is the same shape as the RED metric set: the labels are bounded, but the edge name (server_address) can leak topology. The dependency view tells an attacker which services the application calls, including internal services that are not externally exposed. The view should be restricted to operators with a recorded purpose.

The db.statement attribute is the highest-risk attribute. SQL queries can contain user identifiers, PII, and credentials. The configured view should drop the attribute by default; the Operator that re-enables it should record the reason.

The http.url attribute is the same risk. The full URL contains the query string, which can include session tokens. The Collector should drop the attribute before the data reaches Prometheus; the SDK should not emit it.

Performance implications

The client-side metric is a single histogram per outbound call. The cost is a few microseconds per call; the OTLP exporter serialises the data every 10 seconds. The cardinality is bounded by the number of dependencies — typically under 50.

The on-call cost is the recording rule. The histogram_quantile evaluation across 50 dependencies is bounded by the bucket count. The dashboard reads the recording rule, not the raw histogram.

The retry storm cost is the amplification. The retry configuration must be present in the metric labels so the on-call can read the amplification factor. The http.resend_count attribute is the canonical place.

Production guidance

• The dependency view is the on-call’s first port of call after the RED panel. The dependency view is the breadcrumb the on-call hands to the dependency team.
• The four dimensions are the four metrics. The client-side duration, the server-side duration, the error rate, the saturation. The view contains all four.
• The cardinality is bounded by the dependency name. The server_address attribute is the canonical name. The url.full and db.statement attributes are dropped before the data reaches Prometheus.
• The dependency timeout is a label on the metric. The dependency.timeout configuration is the cap on the client-side latency. The label catches the configuration drift.
• The retry configuration is a label on the metric. The retry amplification is the cause of dependency collapses. The label catches the amplification before the collapse.

Verification

You should now be able to answer:

• What four dimensions are encoded in the dependency performance view, and how do they split between client-side and server-side?
• What is the cardinality trap in the http.url attribute, and how is it prevented?
• Why is the client-side latency the symptom, not the cause, and where does the cause live?
• What does the connection pool exhaustion failure mode look like in the dependency view, and what is the diagnostic?
• Why is the dependency timeout a label on the metric, and what does the label catch?

Quiz