Skip to main content
RunBook Academy

← All break/fix scenarios in Docker & Containers

intermediateContainer~15 min

Break/Fix 15: Container restart loop on dependency failure

Reported symptoms

  • ●Container is in `Restarting` state with a high RestartCount.
  • ●Logs show repeated connection refused / timeout errors to a dependency.
  • ●Dependency service is up but slow to respond.

Evidence

  • · `docker inspect CONTAINER --format "{{.State.Health.Log}}"`
  • · Application logs show the same exception every restart cycle.
  • · `docker stats CONTAINER` shows the container stops quickly each time.
Diagnosis and resolutionclick to reveal

Root cause

The application crashes on dependency failure. With `restart: always`, the daemon keeps restarting the container, which immediately fails again. The restart rate can mask the dependency issue.

Remediation

(1) Set `restart: on-failure` with a max retry count (in Compose `deploy.restart_policy.max_attempts`). (2) Add a startup probe / healthcheck that allows the application time to wait for the dependency. (3) Implement circuit breaker / retry-with-backoff in the application.

Verification

Container stabilises when the dependency is restored. RestartCount does not increment indefinitely.

Prevention

Design every service for partial failure: retries with exponential backoff, circuit breakers, and a graceful "I cannot reach my dependency yet" state. Restart policies are a defence-in-depth, not a substitute for resilient code.

Diagnosis

docker inspect my-app --format '{{.RestartCount}} {{.State.Health.Status}}'
docker logs --tail 50 my-app

The logs show the same connection error repeated across restarts. The restart policy is making things worse.

Fix

services:
  api:
    image: myorg/api:1.0.0
    restart: on-failure
    deploy:
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 5
        window: 60s

For Compose v2 with deploy, the daemon respects the bounded restart. For docker run, the equivalent is harder to enforce.

In the application, add retry-with-backoff:

import time
import requests

def fetch_with_retry(url, max_attempts=5):
    for attempt in range(max_attempts):
        try:
            return requests.get(url, timeout=2)
        except requests.exceptions.RequestException:
            time.sleep(2 ** attempt + 0.1)
    raise RuntimeError("dependency unreachable")