Skip to main content
RunBook Academy

← All break/fix scenarios in Linux

advancedServices~60 min

Break/Fix: a service is down and the health check is failing

Reported symptoms

  • ●The load balancer health check for the service starts failing
  • ●Users report the site returns a connection error rather than an application error
  • ●The ServiceDown alert fires and does not clear on its own
  • ●Nothing was deployed and no configuration change is recorded

Evidence

  • · `systemctl status nginx` reports `inactive (dead)` rather than `failed`
  • · The unit has `Active: inactive (dead) since ...` with no preceding error
  • · `journalctl -u nginx -n 50` ends with a clean `Stopped ...` line and no crash
  • · `systemctl is-enabled nginx` may report `masked`, which is why a plain `start` returns "Unit is masked"
  • · `curl -sS -o /dev/null -w "%{http_code}" http://127.0.0.1/` returns 000 (connection refused), not 5xx
Diagnosis and resolutionclick to reveal

Root cause

The unit was stopped administratively, not by a crash. A stop leaves no error in the journal, which is exactly what distinguishes it from a failure: a crashed service is `failed` with an exit status, a stopped one is `inactive (dead)` with a clean `Stopped` line. If the unit was also masked, its symlink points at /dev/null and every start attempt is refused until it is unmasked.

Remediation

Unmask the unit if `systemctl is-enabled` reports `masked`, then start it and confirm it came up rather than assuming the start succeeded. Do not restart the host and do not reinstall the package: neither addresses a mask, and a reboot destroys the evidence that says who stopped it.

Verification

`systemctl is-active nginx` returns `active`, the health endpoint returns 200 from both the host itself and the load balancer, `journalctl -u nginx --since` the restart shows no new errors, and the alert clears without being silenced.

Prevention

Alert on `systemctl is-enabled` reporting `masked` for any unit under change control, not only on the service being down — a masked unit survives reboots and defeats automatic recovery. Record who stopped what by auditing `systemctl stop`/`mask` through sudo logging, and make the runbook's first question "was this stopped or did it die?" so the two are never conflated.

A service unavailable drill: stop a critical service, diagnose using the methodology, fix it, and document the runbook. The discipline is to apply the methodology in a controlled exercise.

Tasks

Task 1: Inject the failure

Stop a critical service on a test host:

sudo systemctl stop nginx
# Or
sudo systemctl mask nginx    # prevent restart

The service is now down. The health check fails. The alert fires.

Task 2: Diagnose

Apply the methodology:

  • Symptom: “the service is down”.
  • Impact: “all users are affected”.
  • Evidence: check the service status, the logs, the alerts.
sudo systemctl status nginx
sudo journalctl -u nginx -n 50

What does the evidence show?

  • Service is stopped.
  • No recent errors (it was just stopped).
  • Health check is failing.
  • Alert fired.

Task 3: Hypothesis

Based on the evidence, form a hypothesis. Example:

  • “I think the service was stopped manually, possibly by accident.”

Task 4: Fix

Based on the hypothesis, apply the fix:

sudo systemctl unmask nginx    # if masked
sudo systemctl start nginx

The service is back up. Verify it, rather than assuming the start worked — a masked unit fails the start and systemctl start says so only on stderr:

HOST=127.0.0.1                       # substitute the host you are testing

systemctl is-active nginx            # must print: active
curl -sS -o /dev/null -w '%{http_code}\n' "http://$HOST/"

Set HOST rather than pasting a <host> placeholder: in bash < is an input redirection operator, so a copy-pasted curl http://<host>/ fails with a redirection error instead of telling you to substitute a value.

Task 5: Document

Write the runbook:

RUNBOOK: <service> - Service Unavailable
====================================
Last updated: 2026-08-09
Owner: ops
Severity: SEV2

WHEN TO USE
- Service health check is failing
- Users report the service is unavailable
- Alert: ServiceDown

PRE-CHECKS
- Confirm with multiple users
- Check monitoring
- Check recent changes

PROCEDURE
1. SSH to the affected host
2. Check service status: systemctl status <service>
3. Check service logs: journalctl -u <service> -n 50
4. If stopped: try to start: systemctl start <service>
5. If masked: unmask first: systemctl unmask <service>
6. If start fails: check dependencies, configuration, logs
7. Verify: curl http://<host>/health
8. Document the incident

VERIFICATION
- Service health check passes
- Smoke test passes
- No new errors in logs

ROLLBACK
- If the change did not work, revert the configuration
- Document the failed attempt

ESCALATION
- If the service will not start, escalate to the service
  owner
- If the issue is in dependencies, escalate to the dependency
  owner

Task 6: Repeat with different failures

Repeat the drill with different failure modes:

  • Stop a database.
  • Block a network port.
  • Fill the disk.
  • Run out of memory.

For each, follow the methodology, fix, and document.

Knowledge check

Knowledge check · 3 questions

  1. Q1. `systemctl status nginx` reports `Active: inactive (dead)` and the journal ends with a clean `Stopped nginx` line. What does that tell you that `failed` would not?

  2. Q2. `sudo systemctl start nginx` returns immediately and the health check still fails. `systemctl is-enabled nginx` prints `masked`. What happened?

  3. Q3. Rebooting the host is a reasonable first response when a service is `inactive (dead)` with no error in its journal.

Passing score: 75%. Answers are checked in this browser.