Service Availability

The application on host app-prod-12 is responding to requests slowly. The CPU panel is green. The memory panel is green. The disk and network panels are green. The systemd unit that runs the application is active. The on-call engineer concludes the service is healthy. The application, in fact, is half-dead: the main process is up, the worker is stuck on a database connection, and the watchdog has not fired because it was not configured.

Service availability observability on Linux is the discipline of measuring whether the service is running and whether it is doing its job. The systemd unit state answers the first question: is the process alive? The health check and the watchdog answer the second: is the process responding to requests? The production discipline is to instrument both and to distinguish the two signals in the dashboard.

What it is

Service availability observability on Linux is the practice of exposing two classes of measurement for each service:

• Unit state - node_systemd_unit_state from the systemd collector, with the unit’s state (active, failed, inactive, activating, deactivating) and a name label. The metric is the kernel’s view of “is the service running”.
• Health check - the service’s own response to a liveness probe. The probe is HTTP (/healthz), TCP (connect to the listening port), or systemd Type=notify (sd_notify). The metric is the application’s view of “is the service doing its job”.

The two signals are not the same. A service can be active and unhealthy; a service can be inactive and serving requests from a previous instance. The production discipline is to expose both and to alert on the application-side signal.

The alternative is to alert on the unit state alone. The alert is cheap; the failure mode is that the service is alive and the application is dead. The lesson on health checks returns to this.

Why a sysadmin cares

The “service is up” dashboard is the most operationally expensive misleading metric in production. The unit is active; the network is up; the port is open. The user-facing requests are 500. The on-call engineer opens the dashboard, sees “active”, and the service is in fact broken in a way the unit state cannot detect.

A second reason: the lesson on systemd unit state is the entry point to the production alerting taxonomy. The “is the service running?” question is the first question; the “is the service doing its job?” question is the second; the “is the service fast enough?” question is the third. The three questions map to three metric families: unit state, health check, and request histogram. The discipline is to instrument all three.

A third reason: the watchdog is the bridge between the systemd unit state and the application’s view of itself. A watchdog configured with WatchdogSec=30s expects the application to call sd_notify(WATCHDOG=1) every 30 seconds; if the application fails, the kernel kills the process and the unit state moves to failed. The pattern is the production answer to the “alive but stuck” failure mode.

How it works

systemd exposes the unit state through the D-Bus interface. node_exporter reads the interface and emits one node_systemd_unit_state series per unit:

  systemd field        node_exporter metric
  -------------------  ----------------------------------------
  UnitFileState        node_systemd_unit_state{state="enabled"}
  ActiveState          node_systemd_unit_state{state="active"}
                       node_systemd_unit_state{state="failed"}
                       node_systemd_unit_state{state="inactive"}
                       ... (one per known state)
  SubState             node_systemd_unit_state{name="nginx.service"} 1
  LoadState            node_systemd_unit_state (per state)

The state label is the axis on which the operator pivots. A unit that is active and a unit that is failed are both reported as separate time series with the same name label and different state label values.

The health check is the application’s own signal. The canonical implementations:

                  Health Check Patterns
                  =====================

   1. systemd Type=notify
      - The service calls sd_notify(WATCHDOG=1) on a regular
        interval. systemd tracks the most recent notify.
      - If the service misses an interval, systemd kills it.
      - The metric is the unit state plus the watchdog state.

   2. HTTP /healthz
      - The service exposes a health endpoint. The scraper
        (Prometheus blackbox_exporter) probes the endpoint.
      - The metric is probe_success{job="..."}.
      - The cost is the dependency on the HTTP server.

   3. TCP connect
      - The scraper opens a TCP connection to the listening port.
      - The metric is probe_success{job="..."}.
      - The cost is the false positive on a port that is open
        but not serving.

   4. Active script
      - The service runs a script that exits 0 on health.
      - The metric is the exit code via pushgateway.
      - The cost is the script's own failure modes.

The watchdog is the canonical production-graded pattern. The service configures WatchdogSec= in the unit file and the service code calls sd_notify(WATCHDOG=1) on a regular interval. The kernel enforces the interval; missed intervals kill the process.

Under the hood

How to configure it

The systemd collector is enabled by default at runtime but requires the flag --collector.systemd in 1.8.x. The unit file for node_exporter should include the flag:

# /etc/systemd/system/node_exporter.service
[Service]
ExecStart=/opt/node_exporter/node_exporter \
  --web.listen-address=0.0.0.0:9100 \
  --collector.systemd \
  --collector.systemd.unit-include=^(nginx|postgresql|myapp|node_exporter|prometheus|grafana-server)\.service$ \
  --collector.processes

The --collector.systemd.unit-include regex is the cardinality control. Without it, the collector emits one series per state per unit on the host, which is hundreds of series on a modern host. The include form is the allow-list.

For the service itself, the production-grade unit file uses Type=notify, Restart=on-failure, and a watchdog:

# /etc/systemd/system/myapp.service
[Unit]
Description=My Application
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
User=myapp
Group=myapp
ExecStart=/opt/myapp/bin/myapp
ExecStartPre=/opt/myapp/bin/myapp-migrate
WorkingDirectory=/opt/myapp
Restart=on-failure
RestartSec=5s
WatchdogSec=30s
TimeoutStopSec=20s
EnvironmentFile=/etc/myapp/myapp.env
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true

[Install]
WantedBy=multi-user.target

The flags worth memorising:

• Type=notify - the service signals readiness via sd_notify(READY=1). systemd waits for the notification before declaring the unit active.
• Restart=on-failure - restart on non-zero exit, signal, or watchdog timeout. The production baseline.
• RestartSec=5s - the delay between restart attempts. The default is 100 ms; the production baseline is 5 s to avoid restart loops.
• WatchdogSec=30s - the maximum allowed interval between sd_notify(WATCHDOG=1) calls. The production baseline is 30 s. The service must call sd_notify at least as often as the watchdog interval.

The application’s code must call sd_notify on the right interval. The Python, Go, and Rust examples:

# Python: systemd.daemon.notify("WATCHDOG=1")
import systemd.daemon
import threading

def watchdog():
    while True:
        systemd.daemon.notify("WATCHDOG=1")
        threading.Event().wait(10)

threading.Thread(target=watchdog, daemon=True).start()

// Go: github.com/coreos/go-systemd/daemon
import "github.com/coreos/go-systemd/daemon"

ticker := time.NewTicker(10 * time.Second)
go func() {
    for range ticker.C {
        daemon.SdNotify("WATCHDOG=1")
    }
}()

For the HTTP /healthz pattern, the canonical Prometheus configuration uses blackbox_exporter:

# /etc/prometheus/configs/blackbox.yml
modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      valid_status_codes: [200]
      method: GET
      preferred_ip_protocol: ip4

# /etc/prometheus/prometheus.yml (excerpt)
scrape_configs:
  - job_name: 'blackbox_http'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
        - https://myapp.example.com/healthz
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter:9115

The metric is probe_success{job="blackbox_http"}. A value of 0 is the failure signal.

How to validate it

The first check is that the systemd collector is enabled and the metrics are present:

# SEVERITY: READ-ONLY
curl -s http://localhost:9100/metrics | grep '^node_systemd_unit_state' | head -10

Expected output (illustrative):

node_systemd_unit_state{name="myapp.service",state="active"} 1
node_systemd_unit_state{name="myapp.service",state="failed"} 0
node_systemd_unit_state{name="nginx.service",state="active"} 1
node_systemd_unit_state{name="node_exporter.service",state="active"} 1

A unit that is active has state="active" at 1 and all other states at 0. The query node_systemd_unit_state{state="failed"} == 1 returns the failing units.

The second check is the health check. For Type=notify:

# SEVERITY: READ-ONLY
systemctl show myapp.service | grep -E 'Type=|WatchdogSec=|ActiveState=|SubState='

Expected output (illustrative):

Type=notify
WatchdogSec=30s
ActiveState=active
SubState=running

For HTTP health:

# SEVERITY: READ-ONLY
curl -fsS http://myapp.example.com/healthz

A non-200 response is the failure signal.

The third check is the alert rule:

# /etc/prometheus/rules/service.rules.yml
groups:
- name: service.availability
  interval: 30s
  rules:
  - alert: SystemdUnitFailed
    expr: |
      node_systemd_unit_state{state="failed"} == 1
    for: 5m
    labels:
      severity: page
      team: platform
    annotations:
      summary: 'Unit {{ $labels.name }} on {{ $labels.instance }} is in failed state'
      description: 'systemd reports the unit as failed. Inspect journalctl -u <name>.'
      runbook_url: 'https://runbooks.example.com/host/unit-failed'

  - alert: HealthCheckFailing
    expr: |
      probe_success{job="blackbox_http"} == 0
    for: 5m
    labels:
      severity: page
      team: myapp
    annotations:
      summary: 'Health check on {{ $labels.instance }} failing'
      description: 'The /healthz endpoint is not returning 200. The service is unhealthy even if the unit is active.'
      runbook_url: 'https://runbooks.example.com/myapp/health-check'

The two alerts are the production baseline. The first is the unit state; the second is the health check.

How it can fail

Five failure modes appear repeatedly in production.

1. The unit is active but the application is unhealthy. The service is running; the worker is stuck on a database query. The systemd unit state is active. The user-facing requests are timing out. The fix is the health check or the watchdog.
2. The watchdog is not configured. The unit file does not have WatchdogSec=. The application is unhealthy; the unit state is active. The fix is to add WatchdogSec=30s and call sd_notify(WATCHDOG=1) on the right interval.
3. The restart loop masks the failure. The application crashes on every start. The systemd unit restarts it every 5 seconds. The unit state alternates between activating and active. The fix is to add StartLimitIntervalSec=300 and StartLimitBurst=5 to the unit file, and to coinvestigate the underlying failure.
4. The health check is incorrect. The /healthz endpoint returns 200 even when the database is unreachable. The check does not include the dependency. The fix is to extend the health check to include the critical dependencies.
5. The systemd collector is not enabled. The --collector.systemd flag is missing from the node_exporter unit file. The metrics are absent. The fix is to add the flag and restart the unit.

How to troubleshoot it

The diagnostic order when a service is reported as unhealthy:

1. Inspect node_systemd_unit_state{state="active"}. Is the unit active?
2. Inspect node_systemd_unit_state{state="failed"}. Is the unit in failed state?
3. Inspect systemctl status <unit>. What is the active state and sub-state?
4. Inspect journalctl -u <unit> -n 100. What does the log say?
5. Inspect systemctl show <unit> | grep -E 'Type=|WatchdogSec='. Is the watchdog configured?
6. Inspect the health check. Does /healthz return 200?
7. Inspect the application-side histogram. Is the request latency within SLO?
8. Inspect the dependency metrics. Is the database or the message queue reaching the host?

Each step confirms or rules out a layer. Steps 1-3 answer the “is the unit running?” question; step 4 is the application log; steps 5-6 answer the “is the unit doing its job?” question; step 7 answers the “is the unit fast enough?” question; step 8 is the dependency check.

Security implications

The systemd unit state is exposed by D-Bus and read by node_exporter. The collector does not expose the unit’s command line or the service’s environment variables; the information is the unit name and the state. The PII surface is low.

The systemd journal may contain sensitive information: the service’s command line, the application logs, the environment variables. The journal is a security boundary; the production baseline is to restrict the journal to the necessary users (adm, systemd-journal) and to ship the journal to the log platform with the right retention.

The Type=notify and watchdog mechanisms require the service to call sd_notify on the right socket. The socket is the NOTIFY_SOCKET environment variable. A misconfigured service that writes to a different socket is not detected. The fix is to verify the socket is set correctly for the service.

Performance implications

The systemd collector emits one node_systemd_unit_state series per unit name per state. The cardinality is the number of units times the number of states (typically 4). The metric is cheap.

The cost of reading the D-Bus interface is a few hundred microseconds per scrape. The scrape interval is not a performance concern.

The watchdog is a kernel-side mechanism. The cost is the application’s cycle to call sd_notify(WATCHDOG=1). The production baseline is 10 seconds, which is well within the 30-second WatchdogSec= interval.

The health check is application-side. The cost is the request handling overhead. The production baseline is to keep the health check fast (< 100 ms) and to avoid expensive dependencies.

Production guidance

• Use Type=notify and WatchdogSec=30s for any service that should be probed. The pattern is the production answer to the “alive but stuck” failure mode.
• Use Restart=on-failure and RestartSec=5s as the production baseline. Add StartLimitIntervalSec=300 and StartLimitBurst=5 to cap the restart rate.
• Alert on the unit state and the health check. The two alerts are the production baseline.
• Use blackbox_exporter for the HTTP /healthz pattern. The metric is the application-visible signal.
• Use the systemd collector’s --collector.systemd.unit-include regex to limit the cardinality. The default is unbounded.
• Test the health check by killing the database. The check must fail when the dependency is unreachable.

Verification

You should now be able to answer:

• What is the difference between node_systemd_unit_state{state="active"} and the application’s health check, and what question does each answer?
• Why is the watchdog the right answer to the “alive but stuck” failure mode, and how is it configured?
• What is the right pattern for the HTTP /healthz health check, and what is the production metric?
• What is the diagnostic order when a service is reported as unhealthy?
• Why is StartLimitIntervalSec=300 and StartLimitBurst=5 the production baseline for systemd restart limits?

Quiz