Skip to main content
RunBook Academy

← All labs in Linux

Lab · intermediate · ~60 min

Lab: Author a systemd service unit end-to-end

B · Nested virtualisationC · Simulation

Objectives

  • Write a service unit with resource controls and hardening
  • Use systemctl edit to add a drop-in override
  • Add a timer that triggers the service nightly
  • Verify the merged unit with systemctl cat and systemd-analyze

Prerequisites

This lab walks through the production workflow of authoring a hardened systemd service from scratch. The result is a unit that satisfies the principles covered in this part: scoped user, resource limits, hardening directives, drop-in override pattern, and timer activation.

Objective

By the end of this lab, you can write a service unit with resource controls and hardening, override it with a drop-in, attach a timer, and verify the result with the canonical diagnostic commands.

Architecture

flowchart LR
  USER[Unit authoring]
  RES[Resource controls]
  HARD[Hardening]
  DROP[Drop-in override]
  TIMER[Timer activation]
  VERIFY[Verify merged unit]
  USER --> RES --> HARD --> DROP --> TIMER --> VERIFY

Requirements

  • A Linux host (Ubuntu 24.04 LTS or Debian 12 preferred).
  • Root or sudo access.
  • The systemd-analyze tool (default on systemd systems).

Scenario

Your team needs to deploy myapp, a new application daemon. The deliverable: a production-grade systemd unit with hardening, resource limits, a drop-in override for environment-specific configuration, and a nightly timer for batch work.

The myapp binary lives at /opt/myapp/bin/myapp (use a stub script for the lab). The service should run as myapp:myapp, have a CPU quota of 200%, memory limit of 2 GB, and a number of hardening directives appropriate for a network service.

Tasks

Task 1: Write the full service unit

Create /etc/systemd/system/myapp.service:

[Unit]
Description=MyApp application daemon
Documentation=https://wiki.example.com/myapp
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp
EnvironmentFile=/etc/default/myapp
ExecStart=/opt/myapp/bin/myapp
Restart=on-failure
RestartSec=5
TimeoutStopSec=30

# Resource controls
CPUQuota=200%
MemoryMax=2G
MemoryHigh=1G
TasksMax=256

# Writable paths - required, because ProtectSystem=strict below
# makes everything else read-only. systemd creates these with the
# ownership from User=/Group= and exempts them from the read-only
# mount; RuntimeDirectory is removed again on stop.
# Names are relative: they become /var/lib/myapp, /var/log/myapp
# and /run/myapp. An absolute path here is rejected.
StateDirectory=myapp
LogsDirectory=myapp
RuntimeDirectory=myapp

# Hardening
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
PrivateDevices=true
ProtectKernelTunables=true
ProtectControlGroups=true
RestrictNamespaces=true
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
LockPersonality=true
MemoryDenyWriteExecute=true
SystemCallArchitectures=native

[Install]
WantedBy=multi-user.target

Create a stub for /opt/myapp/bin/myapp if it does not exist:

sudo mkdir -p /opt/myapp/bin
sudo tee /opt/myapp/bin/myapp >/dev/null <<'EOF'
#!/bin/bash
echo "myapp started at $(date -Is) with PID $$"
trap 'echo "myapp stopping"; exit 0' TERM INT
while true; do sleep 60; done
EOF
sudo chmod +x /opt/myapp/bin/myapp

Create the environment file:

echo "MYAPP_LOG_LEVEL=info" | sudo tee /etc/default/myapp

Task 2: Validate

sudo systemctl daemon-reload
sudo systemd-analyze verify /etc/systemd/system/myapp.service
sudo systemd-analyze security myapp.service

Both commands should report success. The security output will list findings; expect ~5-10 high-score items if you are running as a fresh user without /opt/myapp set up properly.

Task 3: Create the user and start the service

sudo useradd -r -m -d /opt/myapp -s /usr/sbin/nologin myapp
sudo systemctl enable --now myapp.service
sleep 2
systemctl status myapp

enable --now activates the unit and starts it immediately. Verify the active state is running and the PID is set.

Task 4: Add a drop-in override

sudo systemctl edit myapp.service

Add:

[Service]
Environment=MYAPP_REGION=us-east-1

Save and exit. systemd opens the editor and creates the file at /etc/systemd/system/myapp.service.d/override.conf.

Verify:

sudo systemctl daemon-reload
sudo systemctl cat myapp.service

The output should show both your unit file in /etc/systemd/system/ and the drop-in, in that order, with the drop-in’s directives applied last.

Task 5: Verify the environment variable

systemctl show myapp.service -p Environment
sudo systemctl restart myapp
sudo cat "/proc/$(systemctl show -p MainPID --value myapp.service)/environ" \
  | tr '\0' '\n' | grep MYAPP

The environment of the running process should contain MYAPP_REGION and MYAPP_LOG_LEVEL.

Two details in that command earn their keep. /proc/<pid>/environ is mode 0400 owned by the process UID — the unit runs as myapp, so reading it as your admin user without sudo fails with “Permission denied”. And systemctl show -p MainPID asks systemd which PID it is supervising, whereas pgrep -f can match several processes (or your own grep) and expand into a path with more than one PID in it, producing a confusing cat error rather than an answer.

Task 6: Add a nightly timer

A timer named X.timer activates X.service — that is the default when the [Timer] section has no explicit Unit=. So a myapp.timer would try to start myapp.service, which is a Type=simple daemon you enabled in Task 3 and which is already running. systemd would find it active, do nothing, and report success. The timer would fire every night forever and accomplish nothing.

Give the timer its own oneshot service instead. Create /etc/systemd/system/myapp-maintenance.service:

[Unit]
Description=Nightly myapp maintenance

[Service]
Type=oneshot
User=myapp
ExecStart=/opt/myapp/bin/maintenance.sh

and /etc/systemd/system/myapp-maintenance.timer:

[Unit]
Description=Nightly myapp maintenance task

[Timer]
OnCalendar=*-*-* 03:00:00
RandomizedDelaySec=15min
Persistent=true

[Install]
WantedBy=timers.target

Create a stub for the maintenance script the same way you did in Task 1:

sudo tee /opt/myapp/bin/maintenance.sh >/dev/null <<'EOF'
#!/bin/bash
echo "myapp maintenance ran at $(date -Is)"
EOF
sudo chmod +x /opt/myapp/bin/maintenance.sh
sudo systemctl daemon-reload
sudo systemctl enable --now myapp-maintenance.timer
systemctl list-timers myapp-maintenance.timer

Then prove it actually runs something, rather than trusting the schedule:

sudo systemctl start myapp-maintenance.service
systemctl show -p Result --value myapp-maintenance.service   # must be "success"
journalctl -u myapp-maintenance.service -n 20 --no-pager

Task 7: Audit

systemctl status myapp
systemctl is-enabled myapp
systemctl is-active myapp
systemd-analyze security myapp.service
systemd-analyze verify /etc/systemd/system/myapp.service

Confirm:

  • Active and enabled.
  • Security findings are reduced compared to the unhardened baseline.
  • Verify reports no syntax errors.

Task 8: Revert and confirm the discipline

sudo systemctl revert myapp.service
sudo systemctl daemon-reload
systemctl cat myapp.service

The drop-in should be gone, and your unit file in /etc/systemd/system/myapp.service should be active unchanged.

Note what revert did not do: it did not remove myapp.service itself. systemctl revert restores a unit to its vendor version, meaning it deletes drop-in directories and deletes an /etc unit only when that unit shadows a vendor-supplied one in /usr/lib/systemd/system/. myapp.service has no vendor counterpart — you wrote it from scratch in /etc — so there is nothing to revert it to and systemd leaves it alone. Reach for revert to undo local customisation of a packaged unit; it is not an uninstall command for units you authored yourself.

sudo systemctl edit myapp.service
# add the override again to demonstrate reversibility

Validation

The lab is complete when:

  • The myapp service is active and enabled.
  • A drop-in override exists at /etc/systemd/system/myapp.service.d/override.conf and survives a daemon-reload.
  • The nightly timer is enabled and scheduled, and it targets a Type=oneshot unit — not the always-running daemon.
  • systemctl start myapp-maintenance.service completed with Result=success and left a record in the journal. A timer that has never been proven to run anything is not a working timer.
  • systemctl cat shows the merged unit.
  • systemd-analyze security shows fewer findings than an unhardened baseline.
  • systemctl revert removes the drop-in cleanly.

Expected outcome

A production-ready service unit demonstrating all the patterns from this part: scoped user, resource limits, hardening directives, drop-in override, timer activation, and the diagnostic commands to verify each.

Troubleshooting

  • systemd-analyze verify reports an error — usually a syntax issue. Check the directive spelling; the error message points to the line.
  • enable --now reports “Unit myapp.service is masked” — another unit masks it. Run systemctl list-units --all | grep myapp and resolve the mask.
  • The service fails to start — check the journal: journalctl -xeu myapp. Common cause: /opt/myapp/bin/myapp is not executable or the user myapp cannot read it.
  • The security report still shows many findings — review each. Some findings are inherent (the service needs network, for example); document why those findings are intentional.

Cleanup

sudo systemctl disable --now myapp.service myapp-maintenance.timer
sudo rm /etc/systemd/system/myapp.service
sudo rm -rf /etc/systemd/system/myapp.service.d
sudo rm /etc/systemd/system/myapp-maintenance.timer /etc/systemd/system/myapp-maintenance.service
sudo userdel -r myapp
sudo rm -rf /opt/myapp /etc/default/myapp
sudo systemctl daemon-reload

What you learned

You can now write a hardened systemd service unit, attach a timer, override with a drop-in, and verify the result. The discipline is to use drop-ins for partial overrides, harden incrementally, and validate every change with systemd-analyze.

Deliverables

  • · /etc/systemd/system/myapp.service — full unit file
  • · /etc/systemd/system/myapp.service.d/override.conf — drop-in
  • · /etc/systemd/system/myapp-maintenance.service — oneshot maintenance unit
  • · /etc/systemd/system/myapp-maintenance.timer — timer unit
  • · Output of systemctl cat, systemctl status, and systemd-analyze security

Verification status

Last reviewed
2026-08-09
Executed end to end
not yet run on hardware

The commands and configuration here have been reviewed against the verified software versions, but nobody has run this lab start to finish on a system meeting its prerequisites. Treat the Expected Outcome as the intended result rather than an observed one, and keep the Cleanup section to hand.