Lab 1: Walk the Filesystem Hierarchy Standard

This lab trains the muscle memory of locating any Linux service’s binary, configuration, log, and unit file from scratch. You will apply the FHS lesson to two real services (sshd and chronyd) and to the kernel pseudo-filesystems.

Objective

By the end of this lab, you will be able to take an unfamiliar service on an unfamiliar distribution and answer:

• Where is its binary?
• Where is its configuration read from?
• Where are its logs?
• What systemd unit file(s) does it use, including overrides?

…in under two minutes, without consulting documentation.

Architecture

flowchart LR
  A[Service binary<br/>/usr/sbin/sshd] --> P[Package manager<br/>dpkg / rpm]
  P --> C[Config files<br/>/etc/ssh/sshd_config]
  P --> U[Unit files<br/>/lib/systemd/system + /etc/systemd/system overrides]
  U --> S[State files<br/>/run, /var/lib, /var/log]

You are investigating two services on the same host: the OpenSSH server (sshd) and the chrony time-sync daemon (chronyd).

Requirements

• A Linux host running Ubuntu 24.04 LTS or Debian 12 (Bookworm). Rocky 9 / RHEL 9 / AlmaLinux 9 also supported; commands called out.
• The openssh-server and chrony packages installed (both are present by default on most cloud images).
• Root or sudo access.

Scenario

You have just been handed a fresh Ubuntu 24.04 VM. Two services are running: sshd and chronyd. Without consulting documentation, you need to answer the four questions above for each service. You will use the FHS lesson’s tools: dpkg -L, dpkg -S, rpm -ql, rpm -qf, which, systemctl cat, journalctl, findmnt, /proc, and /sys.

Tasks

Task 1: Locate the sshd binary and its package

Use which to find the path to sshd. Then ask the package manager which package provides it.

which sshd
dpkg -S /usr/sbin/sshd   # Debian-family
rpm -qf /usr/sbin/sshd   # RHEL-family

Confirm the package and version:

dpkg -l openssh-server   # Debian-family
rpm -q openssh-server    # RHEL-family

Expected result: sshd is at /usr/sbin/sshd, provided by openssh-server, version 1:9.6p1-3ubuntu13 or similar.

Task 2: List every file the package installed

This shows you the complete surface area of the package — every binary, library, config, manpage, locale.

dpkg -L openssh-server | grep -E '/(s?bin|etc|lib)/'   # Debian
rpm -ql openssh-server | grep -E '/(s?bin|etc|lib)/'  # RHEL

Identify which lines are binaries, which are config files, which are unit files.

Task 3: Find the active sshd unit file and overrides

systemd reads unit files from a layered search path. The active unit, including any overrides, is what systemctl cat shows.

systemctl cat sshd
systemctl show sshd -p FragmentPath,DropInPaths

Expected result: A unit file that starts with [Unit], lists ConditionPathExists=!/etc/ssh/sshd_config.d/*.conf or similar guards, and points ExecStart at /usr/sbin/sshd -D or /usr/sbin/sshd -i. Look for a [Service] block with EnvironmentFile=-/etc/default/ssh.

Task 4: Locate the sshd configuration

The package query told you config paths. Now confirm what is being read at runtime:

sshd -T 2>&1 | grep -i 'configfile\|port\|pidfile'

sshd -T prints the effective configuration — what the daemon would use if started now. Look for the line configfile /etc/ssh/sshd_config.

Open /etc/ssh/sshd_config and identify the Include /etc/ssh/sshd_config.d/*.conf directive at the top.

Task 5: Find sshd logs

By default, sshd logs to syslog. Confirm and inspect:

journalctl -u ssh --since "1 hour ago" --no-pager | tail -20

Task 6: Repeat for chronyd

For chronyd, do the same five-task sequence in one go, without peeking at documentation.

which chronyd
dpkg -S $(which chronyd)        # Debian
rpm -qf $(which chronyd)        # RHEL

dpkg -L chrony | grep -E '/(s?bin|etc|lib)/'   # Debian
rpm -ql chrony | grep -E '/(s?bin|etc|lib)/'  # RHEL

systemctl cat chrony
systemctl show chrony -p FragmentPath,DropInPaths

ls -l /etc/chrony/chrony.conf   # or /etc/chrony.conf on RHEL

Expected result: chrony configures NTP servers; its unit file starts chronyd with -F 1 or similar; its logs are visible via journalctl -u chrony.

Task 7: Inspect /proc and /sys from the FHS lesson

Walk through each of these and note what they show:

cat /proc/version
cat /proc/cmdline
cat /proc/loadavg
free -h                # reads /proc/meminfo
nproc                  # reads /proc/cpuinfo
ls /sys/class/net/
cat /sys/class/net/$(ls /sys/class/net/ | grep -v lo | head -1)/speed
findmnt /run
findmnt /              # the full mount table

Validation

You have successfully completed this lab if you can show, for any service the instructor names:

• The path to its binary.
• The package that provides it.
• The active systemd unit file (with overrides).
• The configuration path actually read at runtime.
• Where its logs live.

Expected outcome

You now have an evidence-based map of /etc, /usr, /var, and /run for two real services. The same procedure works for any service on any Linux distribution.

Troubleshooting

• “command not found” — which may not be installed; use command -v or type -p instead.
• “Permission denied” when reading /proc or /sys — most files are world-readable; some kernel symbols require root.
• Package not installed — install with apt install openssh-server or dnf install openssh-server before continuing.
• sshd -T produces no output — the syntax may differ slightly between versions. sshd -T 2>&1 | head always produces something.

Cleanup

The lab is non-destructive. To return to the starting state:

sudo systemctl stop chrony   # if you started/stopped it during the lab
sudo systemctl start chrony

No files were created outside /tmp/.

What you learned

You can now triage an unfamiliar Linux service without documentation by walking the FHS, the package database, the systemd unit tree, and the journal. This is the foundation for every later lesson — every production diagnostic uses some subset of these same five steps.