Break/Fix 8: Healthcheck tuning without believing the restart-loop myth

Reported symptoms

• Health.Status flips between healthy and unhealthy.
• Container stays Up the whole time.
• docker logs shows no application errors.
• No restart loop.

Evidence provided

$ docker inspect my-app --format '{{json .State.Health}}' | jq .
{
  "Status": "unhealthy",
  "FailingStreak": 1,
  "Log": [
    {
      "Output": "...",
      "ExitCode": 1,
      "Start": "2026-08-12T14:32:01Z",
      "End": "2026-08-12T14:32:03Z"
    },
    ...
  ]
}

$ docker logs my-app --tail 20
... normal application output, no errors ...

The probe exit code is non-zero; the application logs are clean.

Resolution path

1. Read the probe output.
2. docker inspect <id> --format "{{json .State.Health}}" | jq . and look at the recent Log entries. The Output field tells you exactly what the probe saw.
3. Time the probe command manually under load.
4. Run the test command inside the container: docker exec <id> <probe-command>. If the median response time is comparable to the healthcheck interval, the probe is racing itself.
5. Replace the probe path.
6. Probe /live or a similar path that does not touch downstream services. The probe should answer "is the process responsive", not "is the service end-to-end healthy".
7. **Give the application a start_period.**
8. During cold start the application is slower than steady state. Without a generous start_period, the probe fails on cold start and the daemon marks the container unhealthy before it has had a chance to come up.
9. Use the orchestrator for end-to-end health.
10. Kubernetes readiness/liveness probes, Swarm healthcheck, the load balancer health check - those are the layers that answer the service-level question. The Docker healthcheck is one input to those, not the source of truth.

Verification

1. Health.Status stays healthy under steady load.
2. Run a load test, watch the probe log, see no flips.
3. The probe command is fast.
4. time docker exec <id> <probe-command> reports a median under 100 ms.
5. End-to-end health is monitored elsewhere.
6. The load balancer and the metrics pipeline still see service-level symptoms that the Docker probe deliberately ignores.