Dependency failures and cascading outages

When the database container stops, the API cannot serve requests. That much is unavoidable and not very interesting. What decides whether you have a five-minute blip or a ninety-minute incident is what the API does during those minutes — and the default behaviour of most libraries is the worst option available: retry immediately, forever, with no timeout.

This lesson is about the failure spreading, not the failure starting.

Blast radius: which dependencies are actually hard

The first thing to establish about any dependency is what happens without it, and the answer is usually not the one in the architecture diagram.

flowchart LR
  LB[Reverse proxy] --> API
  API --> DB[(PostgreSQL)]
  API --> Cache[(Redis)]
  API --> Queue[(RabbitMQ)]
  API --> Ext[Payment API]

The Redis rows are the point. The same container is a soft dependency and a hard one depending on what you put in it, and the classification lives in application code rather than anywhere an operator can see it. Writing this table down for your own stack is a twenty-minute exercise that reliably surprises somebody.

The second thing to establish is what a soft dependency does when it fails, because the common answer is “the same thing as a hard one”. Cache code written as value = cache.get(key) with no error handling raises on connection refused, and a service that was supposed to degrade to “slower” degrades to “down” instead. A soft dependency is only soft if somebody wrote the fallback.

The three failures Docker produces, and they are not the same

An application log line saying “could not connect to the database” is three different incidents. Inside Docker, they are unusually easy to tell apart, and the distinction points straight at the cause.

APP=api
DEP=postgres
DEP_PORT=5432

# 1. Do they share a network at all? If not, nothing below will work.
for c in "$APP" "$DEP"; do
printf '%s: ' "$c"
docker inspect -f '{{range $k, $v := .NetworkSettings.Networks}}{{$k}} {{end}}' "$c"
done

# 2. Does the name resolve from inside the app container?
docker exec "$APP" getent hosts "$DEP" || echo 'DNS FAILED'

# 3. Is anything listening on that port? (nc, not curl - this is not HTTP.)
docker exec "$APP" sh -c "nc -z -w3 $DEP $DEP_PORT" && echo 'PORT OPEN' || echo 'PORT CLOSED OR UNREACHABLE'

# 4. Does it answer as the protocol it claims to be?
docker exec "$DEP" pg_isready -h 127.0.0.1 -p "$DEP_PORT"

depends_on covers startup and nothing else

Compose’s depends_on has three long-form conditions:

services:
db:
  image: postgres:17
  restart: unless-stopped
  healthcheck:
    test: ["CMD-SHELL", "pg_isready -U postgres -d appdb"]
    interval: 5s
    timeout: 3s
    retries: 5
    start_period: 30s

migrate:
  image: example/api:2.4.1
  command: ["./manage.py", "migrate"]
  restart: "no"
  depends_on:
    db:
      condition: service_healthy

api:
  image: example/api:2.4.1
  restart: unless-stopped
  depends_on:
    db:
      condition: service_healthy
    migrate:
      condition: service_completed_successfully

service_started is the default for the short list form (depends_on: [db]), and it waits only for the container to exist. PostgreSQL takes several seconds past “running” before it accepts connections, so the short form buys you almost nothing — which is why so many people conclude depends_on “does not work”. It works; it was asked the wrong question.

Retry, with backoff and jitter

Retrying is correct. Retrying immediately is how a recovering dependency is kept down.

import random
import time

import requests


def fetch_with_retry(url, attempts=4, base=0.5, cap=8.0, timeout=(2.0, 5.0)):
    """GET with exponential backoff and full jitter.

    timeout is (connect, read). Both matter: a connect timeout bounds a
    dead peer, a read timeout bounds a wedged one.
    """
    last = None
    for attempt in range(attempts):
        try:
            response = requests.get(url, timeout=timeout)
            response.raise_for_status()
            return response
        except (requests.ConnectionError, requests.Timeout, requests.HTTPError) as exc:
            last = exc
            if attempt == attempts - 1:
                break
            # Full jitter: sleep a random amount in [0, min(cap, base * 2**n)].
            window = min(cap, base * (2 ** attempt))
            time.sleep(random.uniform(0, window))
    raise last

The random.uniform(0, window) is not a refinement; it is the part that works.

Timeouts, which are the setting nobody sets

Before any retry logic matters, every network call needs a deadline, and the defaults in most libraries are “none”.

The rule that keeps a system from cascading:

Every timeout must be shorter than the timeout of whatever is calling you.

If the proxy gives up at 30 seconds, the API’s total budget must be under 30 seconds, and the database call inside it must be under that. When the ordering is inverted — the API waits 60 seconds for a query while the proxy gave up at 30 — the API is holding a worker, a connection and a database session for a request nobody is waiting for any more. Under load that is how a slow dependency becomes an exhausted thread pool, which is how one slow dependency takes out endpoints that do not use it at all.

The circuit breaker

Retries with backoff handle a blip. A circuit breaker handles an outage: after enough failures it stops trying, fails immediately, and periodically tests whether the dependency is back.

import threading
import time


class CircuitOpenError(Exception):
    """Raised when the breaker is open and the call was not attempted."""


class CircuitBreaker:
    """Three states: closed (normal), open (failing fast), half-open (probing)."""

    def __init__(self, failure_threshold=5, reset_timeout=30.0):
        self.failure_threshold = failure_threshold
        self.reset_timeout = reset_timeout
        self._failures = 0
        self._opened_at = 0.0
        self._state = "closed"
        self._lock = threading.Lock()

    def call(self, fn, *args, **kwargs):
        with self._lock:
            if self._state == "open":
                if time.monotonic() - self._opened_at < self.reset_timeout:
                    raise CircuitOpenError("circuit open; not attempting call")
                self._state = "half-open"

        try:
            result = fn(*args, **kwargs)
        except Exception:
            with self._lock:
                self._failures += 1
                if self._state == "half-open" or self._failures >= self.failure_threshold:
                    self._state = "open"
                    self._opened_at = time.monotonic()
            raise

        with self._lock:
            self._failures = 0
            self._state = "closed"
        return result

Two details separate a working breaker from a decorative one, and both are easy to get wrong:

• The counter and the threshold must be different attributes. Storing the failure count in the same name as the configured threshold — a genuinely common bug — means the breaker either never opens or opens on the first call, and neither is visible until the day it is needed.
• A single failure in half-open must re-open the circuit. The probe request exists to answer “is it back?”; if the answer is no, the breaker must go straight back to open rather than counting up to the threshold again, which would let the full failure count of traffic through on every reset interval.

A breaker is worth adding for a partial dependency — the payment API, the recommendation service — where failing fast lets the rest of the request succeed. For a hard dependency it changes an error into a faster error, which is still worth something (it stops the thread pool filling) but is not a recovery.

Bulkheads, and why one slow dependency stops everything

This is the mechanism by which a partial dependency causes a total outage, and it has nothing to do with the dependency being important.

A container serves requests with a fixed pool of workers — threads, goroutines with a semaphore, or a fixed process count under a WSGI server. If the recommendation service starts taking 30 seconds instead of 50 milliseconds, every request that touches it occupies a worker for 30 seconds. With 20 workers and modest traffic, all 20 are soon blocked on recommendations, and requests to /health, /login and everything else queue behind them.

The service is now down, and the recommendation service — a nice-to-have — is the cause.

A bulkhead limits how many workers any one dependency may occupy:

import threading

# At most 5 of the 20 workers may be inside the recommendation call.
_recommendations = threading.Semaphore(5)


def get_recommendations(user_id):
    if not _recommendations.acquire(blocking=False):
        return []          # degrade, do not queue
    try:
        return recommendation_client.fetch(user_id, timeout=(1.0, 2.0))
    finally:
        _recommendations.release()

The blocking=False is the whole idea. Queueing for the semaphore just moves the pile-up; refusing immediately and returning a degraded result is what keeps the other fifteen workers available.

Verification: break it on purpose

Resilience that has never been tested is a comment in the code.

set -euo pipefail
APP=api
DEP=redis
URL=http://localhost:8080/

echo '--- baseline ---'
curl -sS -o /dev/null -w 'status=%{http_code} time=%{time_total}s\n' "$URL"

echo '--- dependency stopped ---'
docker stop "$DEP" >/dev/null
sleep 2
curl -sS -o /dev/null -w 'status=%{http_code} time=%{time_total}s\n' "$URL"
docker inspect -f 'app health={{.State.Health.Status}}' "$APP"

echo '--- dependency back ---'
docker start "$DEP" >/dev/null
sleep 10
curl -sS -o /dev/null -w 'status=%{http_code} time=%{time_total}s\n' "$URL"

echo '--- did the app log a retry storm? ---'
docker logs --since 60s "$APP" 2>&1 | grep -ci 'retry\|reconnect' || true

Read the three status lines against what you claimed:

• Redis is a cache and the middle status is 200, slower. The fallback exists and works.
• Redis is a cache and the middle status is 500. You have a hard dependency you thought was soft. That is the finding.
• The third status is still 500 after ten seconds. The connection pool is holding dead sockets. Fix the pool settings, not the retry logic.
• time_total in the middle line is 30 seconds. There is no connect timeout. Everything else is secondary to fixing that.

Repeating the drill with a slow dependency rather than a dead one is more revealing still, and docker network disconnect is the tool: disconnecting the dependency’s container from the network produces timeouts rather than connection refusals, which is the failure mode that finds missing bulkheads.

Which pattern for which dependency

If you do exactly one thing from this lesson, make it the first row. Timeouts are the only entry that is required everywhere, they are the cheapest to add, and their absence is what turns every other pattern into decoration.

Knowledge check

Where next

This part has covered what breaks and how it spreads. The observability part gives you the signals to see it happening — and the correlation that lets you follow one request through all five containers it touched.