systemd and Process Management · Observability

A dashboard engineer reports “Grafana is down” the morning after a weekly maintenance window. systemctl status grafana-server shows active (running) for the last three days. The CPU is at 4%. The process has been busy-since 03:14 in a state the unit calls running but the application calls “deadlocked on the database pool.” The on-call engineer restarts the service. The symptom clears. The cause does not.

Grafana is one binary. The supervisor around it is the thing that decides when it runs, where it can write, who it can become, and what happens when it dies. The shipped unit is a starting point, not a production unit. The lessons in this module treat it as such.

What it is

A grafana-server.service unit is the declaration systemd uses to manage the lifecycle of the Grafana process: identity, invocation, dependencies, restart policy, resource limits, and a sandbox inside which the process is allowed to operate. The unit does not configure Grafana itself; that is grafana.ini. The unit governs where Grafana runs and under what rules.

   systemd
      |
      +-- grafana-server.service
              |
              +-- ExecStart=/usr/sbin/grafana-server
              |
              +-- User=grafana
              +-- Group=grafana
              +-- WorkingDirectory=/var/lib/grafana
              |
              +-- After=network-online.target mysql.service
              +-- Requires=mysql.service
              |
              +-- LimitNOFILE=65536
              +-- ProtectSystem=strict
              +-- ReadWritePaths=/var/lib/grafana
              |
              +-- Restart=on-failure
              +-- RestartSec=5s

The four target distributions — Ubuntu 24.04, Debian 12, RHEL 9 — all run systemd. One unit file describes the integration in full.

Why a sysadmin cares

The supervisor is the one thing that has to be right before Grafana is “up.” When the supervisor is wrong, Grafana is wrong in ways that look like Grafana bugs.

Without a unit, the process is a child of your shell. The first logout kills it. The first reboot orphans it. The first OOM is permanent.
Without sandboxing, a vulnerability in a plugin becomes a full-host compromise with the same privileges as the service.
Without a restart policy, a transient Postgres restart leaves Grafana running with stale connections and no recovery.
Without limit tuning, a busy install opens tens of thousands of file descriptors per process and runs into the systemd default ceiling of 1024 before the morning peak.

How it works: the lifecycle states

A Type=simple unit (the Grafana default) has three runtime states the operator observes:

   inactive (dead)            # unit loaded, nothing running
      |
      |  systemctl start
      v
   activating                 # process forked, started, opening ports
      |
      |  first listen on 3000
      v
   active (running)           # process alive, serving traffic
      |
      |  failed health check, OOM-kill, abnormal exit
      v
   active (running) -> failed  # process gone; unit is "failed"
      |
      |  systemd restart on-failure
      v
   activating -> active (running)

There is no intermediate “Serving” state in Grafana. The unit calls itself active (running) the moment grafana-server forks. A running process does not mean a serving process. The only truth about “is Grafana serving requests” is the local /api/health probe (lesson 06 covers this in detail).

The shipped Grafana unit on apt and rpm paths is Type=simple. Adding Type=notify would let Grafana signal “ready” with sd_notify(READY=1); upstream does not ship this out of the box, so a custom unit is required if you want a stronger signal.

How to configure it

A production hardening overlay

The apt and rpm packages ship /usr/lib/systemd/system/grafana-server.service with User=grafana, Restart=on-failure, and LimitNOFILE=4096. That is a starting point, not a production unit. The overlay below is a drop-in (lesson 04 of module I covers the drop-in pattern in detail):

# /etc/systemd/system/grafana-server.service.d/00-hardening.conf
[Service]
# Resource limits: dashboards, alert rules, image renders and
# datasource connections all want file descriptors. 1024 is too
# small past a few hundred concurrent sessions; 65536 is the
# Grafana team's ceiling.
LimitNOFILE=65536

# The number of processes the service may fork (plugin spawns,
# image rendering). 4096 covers a busy install.
LimitNPROC=4096

# Hardening: read-only filesystem, write only into the data dir.
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
NoNewPrivileges=true
ReadWritePaths=/var/lib/grafana /var/log/grafana
# Restrict the ambient Linux capabilities to the lowest useful set.
AmbientCapabilities=
CapabilityBoundingSet=

# Don't trust any incoming SUID or SGID bits on binaries the
# service might exec.
RestrictSUIDSGID=true

# Allow private /tmp (default), deny loading kernel modules, deny
# ptrace attach. These are belt and braces for a Go process that
# does not need any of them.
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectKernelLogs=true
ProtectControlGroups=true
RestrictNamespaces=true
LockPersonality=true
MemoryDenyWriteExecute=true

Dependency on the database

The Grafana systemd unit on Debian-family and RHEL-family distributions does not declare an ordering against the database. If Grafana is configured to talk to a local MySQL or PostgreSQL that runs under its own unit, the unit must declare the order:

# /etc/systemd/system/grafana-server.service.d/10-database.conf
[Unit]
# Start the database first; if it cannot start, neither can we.
After=network-online.target mysqld.service
Wants=network-online.target
Requires=mysqld.service

Requires= is a hard dependency: if mysqld.service fails to start, grafana-server is not started either. After= is the ordering without the hard dependency. Use both when Grafana genuinely cannot serve without the database. Use only After= when Grafana can start in read-only / disconnected mode for one boot to allow the database to recover.

A complete production unit, drop-in form

For reference, the entire production unit can be expressed as a drop-in over the shipped unit:

# /etc/systemd/system/grafana-server.service.d/99-production.conf
[Unit]
Description=Grafana instance (hardened drop-in)
Documentation=https://grafana.com/docs/grafana/latest/
After=network-online.target
Wants=network-online.target

[Service]
Environment=GRAFANA_OPTS=--config=/etc/grafana/grafana.ini
LimitNOFILE=65536
LimitNPROC=4096

ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
NoNewPrivileges=true
ReadWritePaths=/var/lib/grafana /var/log/grafana
RestrictSUIDSGID=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectKernelLogs=true
ProtectControlGroups=true
RestrictNamespaces=true
LockPersonality=true
MemoryDenyWriteExecute=true

Restart=on-failure
RestartSec=5s
TimeoutStopSec=20s

Reload the unit, then the service:

sudo systemctl daemon-reload
sudo systemctl edit --grafana-server   # opens the drop-in editor
sudo systemctl restart grafana-server

How to validate it

# READ-ONLY: the unit is loaded, active, and enabled at boot.
systemctl is-enabled grafana-server
# enabled
systemctl is-active grafana-server
# active

# READ-ONLY: the actual effective unit (overlay applied).
systemctl cat grafana-server
# Shows the base unit + every drop-in fragment, in order.

# READ-ONLY: the process under the right user, with the right
# number of file descriptors.
ps -fC grafana-server
ls -l /proc/$(pidof grafana-server)/fd | wc -l
# 65536 must be the cap, not the count of descriptors held.

# READ-ONLY: the limits are visible to the process itself.
cat /proc/$(pidof grafana-server)/limits \
  | grep -E 'open files|processes'

# READ-ONLY: the unit's last 200 lines, no pager.
journalctl -u grafana-server -n 200 --no-pager

# READ-ONLY: is the binary actually serving?
curl -fsS http://localhost:3000/api/health
# {"database":"ok","version":"11.3.0"}

A clean validation: the unit is enabled, active, driven by the correct user, with LimitNOFILE matching the planned concurrency, a database field of ok in /api/health, and no permission denied lines in the journal since the last restart.

How it can fail

The unit is one of the longer lists of failure modes in this module. The high-frequency shapes are listed below.

Unit enablement, but no start. A successful apt install on Debian-family distributions installs the unit but does not start it. The unit is loaded and enabled, the port is closed, /api/health is not reachable. The symptom is a fresh host where the binary is installed but the service is silent.
Hardened unit against a network filesystem. ProtectHome=true on a host where Grafana is configured to write to a path under /home (some legacy configs) blocks the write. The symptom is permission denied on first save and a /api/health of {"database":"ok"} followed by save failures.
LimitNOFILE left at the systemd default. A Grafana instance running for several months with the default ceiling hits too many open files during the morning alert-evaluation peak. The symptom is panel load failures on dashboards with many queries.
Drop-in shadowed by a syntax error. An [Install] fragment in a drop-in file is silently ignored; an [Service] typo (e.g. LimitNoFILE instead of LimitNOFILE) is also ignored, but the operator reads the unit and assumes the directive took effect. The symptom is “I added the directive; it does nothing.”
Requires= cycle. Adding Requires=grafana-server.service to mysqld.service (or vice versa) creates a dependency cycle that systemd refuses to start. The symptom is Failed to determine job dependencies on systemctl start and the dependency graph printed in the error.
Type=simple masking boot failure. A Type=simple unit calls itself active the moment the process exists. If Grafana fails to bind 3000 (port already in use, missing config file, bad database credentials), the unit is “active (running)” while Grafana is unusable. The symptom is systemctl status showing green while /api/health is unreachable.

How to troubleshoot it

The diagnostic order is “is the process running?”, “is the process serving?”, “what does the kernel think?”.

Is the unit loaded? systemctl cat grafana-server — if the file is empty or missing, the unit is not installed.
Is the unit enabled at boot? systemctl is-enabled grafana-server — if disabled, the unit will not start on the next reboot.
Is the process alive? ps -fC grafana-server — the unit can be active while the process is missing if the binary exited. The journal will say so.
Is the process serving? curl -fsS http://localhost:3000/api/health. If unreachable, the problem is upstream of the unit, not the unit itself.
What do the journal lines say? journalctl -u grafana-server -n 200 --no-pager. Look for panic lines, audit failures, and permission denied.
What does the kernel say? dmesg | grep -iE 'grafana|oom|segfault' for the last 1 000 lines. SELinux / AppArmor denials appear in the audit log on RHEL and Ubuntu respectively.

Security implications

Sandboxing directives reduce blast radius. ProtectSystem=strict, PrivateTmp=true, NoNewPrivileges=true, RestrictNamespaces=true, MemoryDenyWriteExecute=true are not theoretical; they are the difference between “a plugin vulnerability compromises Grafana” and “the same vulnerability compromises the host.”
Process identity. The unit must run as the grafana user with nologin as its shell. A misconfigured unit running as root makes every Grafana vulnerability a root-level issue.
AmbientCapabilities= and CapabilityBoundingSet=. The shipped unit has neither set; this is correct for a Go process that does not bind to a low port or change identity. Setting either to CAP_NET_BIND_SERVICE is required only if Grafana must bind 443 directly (lesson 04 covers the reverse-proxy-first design that avoids this).
Resource caps. MemoryMax=, CPUQuota=, and TasksMax= give the host a way to fence a runaway Grafana. Without them, a stuck alert-evaluation loop will eat the whole machine.

Performance implications

LimitNOFILE is the most operationally expensive limit. Every active dashboard panel holds one connection to its underlying data source. A hundred concurrent users on a Grafana with 200 dashboards hold tens of thousands of descriptors by the morning peak. The systemd default of 1024 is wrong; the upstream ceiling of 65536 is the practical upper bound.
LimitNPROC covers spawned helper processes — image rendering, plugin execution, query workers. The default is fine for a small Grafana; a busy one with image-render enabled will hit it before LimitNOFILE.
MemoryMax= caps RSS at a configured GiB count. Setting this too low causes OOM-kills of the unit; setting it too high removes the boundary that protects the rest of the host. Two GiB is a reasonable starting ceiling; tune from there.
CPUQuota= caps user-space CPU. Useful for noisy-neighbour multi-tenant boxes; not needed on a dedicated Grafana host.
TasksMax= caps the total process / thread count. The Go scheduler is happy with thousands of goroutines, but each one is a kernel task at some abstraction level; 4096 is comfortable for a Grafana box.

Production guidance

Adopt a drop-in pattern. The shipped unit is the base; the hardening directives live in /etc/systemd/system/grafana-server.service.d/. A re-install of the package does not overwrite the drop-in.
Size LimitNOFILE against cat /proc/<pid>/limits | grep 'open files' while the system is healthy, then add headroom. Picking 65536 because it is round is fine; verify the unit actually applied it.
Declare a hard ordering against the database only when Grafana genuinely cannot serve without it. Prefer After= to Requires= when the database is on a different host.
Use systemctl try-reload-or-restart grafana-server as the reload primitive. The unit treats SIGHUP as a configuration reload; the application surfaces part of that as /api/admin/provisioning/reload.

Verification

You should now be able to answer:

What is the difference between active (running) and “serving requests” in the context of a Grafana systemd unit?
Why does a hardened drop-in belong in /etc/systemd/system/grafana-server.service.d/ rather than as an edit to /usr/lib/systemd/system/grafana-server.service?
Which two Limit* directives are the operational ceiling you will tune first, and what each one covers?
When would you use Requires=mysqld.service versus After=mysqld.service in the drop-in, and what happens if you use the wrong one?