Saturation from Application

A checkout service is reporting slowly rising latency. The RED panel shows the rate is normal, the error rate is near zero, the p95 is climbing. The USE panel shows the host CPU at 20%, the memory at 50%, the disk at 5%. The host is fine. The dependency view shows the four dependencies are all under budget. The latency is coming from inside the application. The next metric to inspect is the application-internal queue depth.

Saturation is the latency signal that the other methodologies miss. The rate may be normal; the host may be idle; the dependencies may be fast. The application is slow because the work is queueing inside the application, behind a fixed-capacity resource: a thread pool, a connection pool, an async queue, a bounded channel.

The shape is the queue. The diagnostic is the depth over time. The mitigation is the capacity or the upstream.

What it is

Application saturation is the state of an application-internal resource where the work in flight exceeds the resource’s capacity to process it. The work is queued; the queue depth grows; the latency per request grows proportionally to the queue depth.

The four most common queue shapes in production are:

1. Thread pool — a fixed number of worker threads servicing a request queue. The Tomcat connector, the FastAPI worker pool, the Java ThreadPoolExecutor. The capacity is the thread count; the queue depth is the bounded request queue.
2. Async queue — a fixed-capacity buffer of pending work, drained by a worker. The asyncio Queue, the Go channel, the Java BlockingQueue. The capacity is the buffer size; the queue depth is the buffer occupancy.
3. Connection pool — a fixed number of reusable connections to a dependency. The HikariCP pool, the psycopg pool, the asyncpg pool. The capacity is the pool size; the depth is the in-use connections.
4. In-flight requests — the unbounded count of concurrent requests the application is serving. The FastAPI middleware, the gRPC interceptor. The “capacity” is implicit; the queue depth is the open_requests counter.

The canonical metric for each shape is a gauge or an up_down_counter (an integer that can rise and fall). The OpenTelemetry convention is http.server.active_requests for the in-flight shape, and process.runtime.{pool,queue} for the internal shapes.

The four shapes are distinct but related. A thread pool that is fully utilised has a queue depth that is growing; the queue depth is the saturation signal for the thread pool. A connection pool that is exhausted has a queue of pending acquirers; the queue depth is the saturation signal for the pool.

Why a sysadmin cares

The single most common shape of a latency incident is application-internal saturation. The visible symptom is the RED duration panel climbing; the visible cause is the queue depth gauge climbing. The team that has the saturation gauge can answer the question “is the service saturated?” in five seconds; the team that has to fall back to logs and traces takes ten minutes.

Three operational problems disappear when saturation gauges are in place:

1. The “is it slow or saturated?” question. A service with rising p95 but low host CPU is probably saturated at the application layer. The saturation gauge distinguishes “the work is slow” from “the work is queueing”.
2. The “is it me or the dependency?” question. A service with rising p95 and a saturated connection pool is bottlenecked on the dependency. The pool gauge is the diagnostic that points at the dependency.
3. The capacity planning question. A service with a queue depth that grows at 2% per hour is going to be saturated within 24 hours. The gauge is the early warning that the capacity plan is wrong.

How it works

The mental model is that every queue has a capacity and a depth. The capacity is fixed (the pool size, the worker count, the buffer size). The depth is variable. When the depth exceeds the capacity, the queue is full; the next request waits.

  rate of work arriving
       |
       v
  +-----------+
  |   queue   |  -- depth = current size
  +-----------+
       |
       v
  rate of work served
       (capacity = max rate)

The latency per request is the time in queue plus the time to serve. As the depth grows, the time in queue grows. The latency grows linearly with the depth.

The four shapes map to four metrics:

  thread pool        -- thread_pool_size, thread_pool_active
  async queue        -- queue_size, queue_depth
  connection pool    -- pool_size, pool_in_use
  in-flight reqs     -- http_server_active_requests

The OpenTelemetry convention is to emit one up_down_counter per shape. The up_down_counter increments on task arrival and decrements on task completion. The Prometheus gauge() is the corresponding primitive; the up_down_counter is the OTel equivalent.

The trap is to instrument the capacity but not the depth. A thread pool that reports thread_pool_size but not thread_pool_active tells the on-call the maximum but not the actual. The dashboard that shows only the size is permanently green; the dashboard that shows the depth catches the saturation.

Under the hood

How to configure it

The four shapes are configured in three places: the SDK at service startup, the Collector pipeline, and the Prometheus recording rule.

1. The SDK at service startup

For Python FastAPI with the OpenTelemetry 0.110.x SDK, the in-flight requests gauge is automatic via the FastAPI instrumentation. The connection pool gauge is wired manually:

# app.py -- saturation instrumentation
import asyncio
from contextlib import asynccontextmanager
from opentelemetry import metrics
from opentelemetry.sdk.metrics import (
    MeterProvider,
    Observation,
)
from psycopg_pool import AsyncConnectionPool

meter = metrics.get_meter("checkout", "1.4.2")

# SEVERITY: CONFIGURATION -- wire the pool gauge
def observe_pool(pool: AsyncConnectionPool) -> list:
    stats = pool.get_stats()
    return [
        Observation(stats.get("pool_size", 0),
            {"pool.name": "checkout-pg"}),
        Observation(stats.get("pool_available", 0),
            {"pool.name": "checkout-pg"}),
        Observation(stats.get("requests_waiting", 0),
            {"pool.name": "checkout-pg"}),
    ]

pool = AsyncConnectionPool(
    conninfo="postgresql://app:secret@db:5432/checkout",
    min_size=4,
    max_size=20,
    open=False,
)

@asynccontextmanager
async def lifespan(app):
    await pool.open()
    pool_gauge = meter.create_observable_gauge(
        "db.client.connections.usage",
        callbacks=[lambda o: observe_pool(pool)],
        unit="1",
        description="Active connections in the checkout database pool",
    )
    yield
    await pool.close()

# The FastAPI middleware handles the in-flight gauge
# automatically via opentelemetry-instrumentation-fastapi

For the async queue, the gauge is wired manually:

from opentelemetry.metrics import UpDownCounter

queue_depth = meter.create_up_down_counter(
    "asyncio.queue.depth",
    unit="1",
    description="Pending items in the asyncio event queue",
)

async def producer(queue: asyncio.Queue):
    while True:
        item = await fetch_item()
        await queue.put(item)
        queue_depth.add(1, {"queue.name": "events"})

async def consumer(queue: asyncio.Queue):
    while True:
        item = await queue.get()
        await process_item(item)
        queue_depth.add(-1, {"queue.name": "events"})

The increment is on put; the decrement is on task_done. The decrement must be in the finally block to handle exceptions.

2. The Collector pipeline

The Collector pipeline is the same shape as the RED pipeline. The cardinality is bounded by the number of pools, queues, and threads — typically under 20 per service.

# /etc/otelcol-contrib/config.yaml
processors:
  batch:
    timeout: 10s
service:
  pipelines:
    metrics:
      processors: [batch]

The saturation metrics are bulk-y in the chart shape (they oscillate with each request). The 10-second batch interval smooths the chart.

3. The Prometheus recording rule

The saturation view is promoted to a recording rule:

# /etc/prometheus/rules/saturation.rules
groups:
- name: saturation
  interval: 15s
  rules:
  - record: service:in_flight_requests:current
    expr: |
      sum by (service_name, http_route) (
        http_server_active_requests
      )

  - record: service:pool_usage:ratio
    expr: |
      sum by (service_name, pool_name) (
        db_client_connections_usage
      )
      /
      sum by (service_name, pool_name) (
        db_client_connections_max
      )

  - record: service:pool_pending_requests:current
    expr: |
      sum by (service_name, pool_name) (
        db_client_connections_pending_requests
      )

  - record: service:queue_depth:current
    expr: |
      sum by (service_name, queue_name) (
        asyncio_queue_depth
      )

The interval: 15s is shorter than the RED rule’s 30s; saturation gauges are more time-sensitive.

The promtool validator confirms the rule syntax:

# SEVERITY: READ-ONLY
promtool check rules /etc/prometheus/rules/saturation.rules

Expected output:

SUCCESS: /etc/prometheus/rules/saturation.rules

How to validate it

Validate that the saturation gauges are live with three checks.

1. The in-flight gauge is present.

# SEVERITY: READ-ONLY
curl -s 'http://prometheus:9090/api/v1/query?query=service:in_flight_requests:current{service_name%3D"checkout"}'

Expected output:

http_route
/checkout  12
/cart      3

The values are bounded; the gauge is reading the current number of in-flight requests.

2. The pool usage is recorded.

# SEVERITY: READ-ONLY
curl -s 'http://prometheus:9090/api/v1/query?query=service:pool_usage:ratio{service_name%3D"checkout"}'

Expected output:

pool_name
checkout-pg  0.45

A ratio of 0.45 is 45% — the pool is healthy. A ratio above 0.85 is a signal that the pool is close to exhaustion; the alert should fire on a ratio above 0.9 for 5 minutes.

3. The pending requests gauge is zero.

# SEVERITY: READ-ONLY
curl -s 'http://prometheus:9090/api/v1/query?query=service:pool_pending_requests:current{service_name%3D"checkout"}'

Expected output:

pool_name
checkout-pg  0

A non-zero value is a signal that the pool is exhausted and the application is queueing. The alert should fire on a non-zero value for 1 minute.

How it can fail

Four specific failure shapes appear regularly in saturation instrumentation:

1. Leaky counters. The middleware increments on entry but never decrements on exception. The gauge climbs monotonically. The dashboard is permanently red. The alert is silent because the gauge is expected to be high. The on-call concludes the service is saturated when the service is idle. Symptom: in-flight gauge climbs forever; CPU is 5%.
2. Missed decrements. The queue’s task_done callback is not wired. The queue depth gauge reports the same value forever. The dashboard shows the queue depth at 50; the underlying queue is empty. The on-call concludes the queue is saturated when the queue is idle. Symptom: the gauge is stuck at the previous value.
3. Queue draining. A burst of work arrives; the queue depth spikes to 80. The workers drain the queue over 30 seconds. The dashboard shows the spike; the on-call concludes the queue is permanently saturated. The dashboard must visualise the trend over the last 5 minutes, not the current value. Symptom: the gauge spikes and recovers; the alert fires on the spike; the queue is healthy.
4. Pool exhaustion. The pool has 20 connections and 50 concurrent requests. The pool is exhausted; the pending requests gauge is at 30. The application is queueing internally. The dependency is fine. The dashboard shows the dependency view as healthy; the pool view is the diagnostic. Symptom: dependency view normal; pool pending requests climbing.

How to troubleshoot it

When a saturation gauge says something the operator does not believe, the diagnostic order is:

1. Confirm the gauge is the current value, not the cumulative value. The up_down_counter and the gauge are current values. A counter that only increases is not the saturation signal.
2. Confirm the increment and decrement are paired. Inspect the code for the increment and decrement sites. The decrement must be in the finally block.
3. Confirm the queue is the queue the application is using. The async queue instrumentation may be wired to the wrong queue. The pool instrumentation may be wired to the wrong pool. The dashboard should match the gauge to the actual resource.
4. Confirm the alert is on the shape, not the value. A queue that spikes for 30 seconds is acceptable; a queue that stays at 50 for 5 minutes is not. The alert should fire on the sustained value, not the current value.

Security implications

The saturation gauges are bounded and the labels are structural (pool name, queue name, thread name). The risk is the pool name attribute: a pool named after a database may include the database hostname, which can leak topology. The dashboard should resolve the pool name to the canonical service name.

The in-flight gauge is not sensitive. The asyncio queue depth is not sensitive. The pool gauge is not sensitive.

The combination of pool name, queue depth, and pending requests is enough to identify the service’s workload shape. The view should be restricted to operators with a recorded purpose.

Performance implications

The saturation gauges are cheap to emit. The up_down_counter is a single increment per request. The observable gauge is a poll per scrape; the poll is a few microseconds.

The on-call cost is the dashboard. The histogram_quantile is not used for gauges; the gauge is read directly. The dashboard query is a single value per series.

The recording rule cost is the polling interval. The 15s interval is the right balance; 5s is too expensive for the value, 60s is too coarse.

Production guidance

• The saturation gauge is the on-call’s “before the incident” signal. The gauge catches the incident before the RED panel.
• Pair the gauge with the trend. A queue that spikes and recovers is fine; a queue that stays high is the alarm. The dashboard should show the 5-minute trend.
• The decrement must be in finally. The increment is on entry; the decrement is on exit. An exception path that skips the decrement leaks the gauge.
• The pool gauge is the dependency exhaustion signal. A dependency that is healthy but a pool that is exhausted is the application is slow because the application is queueing.
• The alert should fire on the sustained value, not the current value. A 5-minute for: clause filters transient spikes.

Verification

You should now be able to answer:

• What are the four most common queue shapes in production, and what is the canonical metric for each?
• Why must the saturation gauge’s decrement be in the finally block, and what does a leaky counter look like in the dashboard?
• How does the connection pool exhaustion failure mode differ from the dependency-saturation failure mode, and what is the diagnostic?
• Why is the queue draining failure mode a false positive, and how is it filtered?
• Why is the alert on the saturated value, not the current value, and what is the role of the for: clause?

Quiz