high riskcluster affecting~60 min

Runbook: Production hardening - apply CIS baseline to a host

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Identify the target host and its role
· Confirm out-of-band access (cloud console, IPMI)
· Identify the application stack and dependencies
· Capture the baseline before changes

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Apply the sysctl hardening (network and kernel)
2Apply the mount options (/tmp, /dev/shm)
3Apply the SSH hardening (publickey, modern ciphers)
4Apply the auditd configuration
5Apply the password policy (PAM)
6Validate each change with smoke tests
7Document exceptions
8Run OpenSCAP to verify

4 · Verification

Confirm the procedure actually fixed the problem.

✓All Critical CIS recommendations are applied or documented as exceptions
✓Applications still work (smoke tests pass)
✓OpenSCAP PASS count is high
✓Exceptions are approved and reviewed

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Configuration management revert
↶Boot from previous snapshot (cloud)
↶Restore configuration files from backup
↶Revert one hardening control at a time, verifying the affected service between each - a bulk revert loses which control caused the breakage

6 · Escalation

When the runbook isn't enough, contact:

· If the hardening breaks an application, escalate to application owner
· If exceptions are required for security-sensitive controls, escalate to security team
· If compliance scan shows no improvement, escalate for review

This runbook applies a CIS baseline hardening to a production host. The procedure is staged so each change is reversible and applications continue to work.

When to use this runbook

Use this runbook when:

Onboarding a new host to a hardened baseline.
Bringing a legacy host up to current CIS.
After a security audit finding.
Quarterly as a regression test.

Inputs

Gather before starting:

The target host and its role (web, db, app).
The application stack and dependencies.
Out-of-band access (cloud console, IPMI).
The CIS profile to apply (Level 1 or Level 2).
Known exceptions from previous runs.

Procedure

Step 1: Capture the baseline

Read-only / Safesysctl

sysctl -a > /tmp/sysctl-before.txt
systemctl list-unit-files --state=enabled > /tmp/services-before.txt
ls -l /etc/ssh/sshd_config > /tmp/sshd-config-mtime.txt
sudo oscap xccdf eval \
--profile xccdf_org.ssgproject.content_profile_cis_level1_server \
--results baseline-scan.xml \
--report baseline-report.html \
/usr/share/xml/scap/content/ssg-ubuntu2204-ds.xml

The baseline scan report is the comparison point.

Step 2: Apply sysctl hardening

Configuration changeapply the reviewed sysctl baseline— Back up only if a previous file exists, then validate before applying. sysctl --system reports an unknown key rather than failing silently.

CONF=/etc/sysctl.d/99-hardening.conf

# Back up only if there is something to back up. A bare cp here fails on a
# first run, which is how this step used to abort before doing anything.
[ -f "$CONF" ] && sudo cp -a "$CONF" "$CONF.bak.$(date +%Y%m%d)"

# Place your reviewed baseline at $CONF now, then:
sudo sysctl -p "$CONF"          # applies just this file, and names any bad key
sudo sysctl --system | tail -20 # confirms nothing later in the load order won

Verify applications still work (curl, SSH from another host, application smoke tests).

Step 3: Apply mount options

Service impact possiblecp

sudo cp /etc/fstab /etc/fstab.bak

# Edit /etc/fstab for /tmp and /dev/shm
# /tmp was on /dev/sda1; change to tmpfs
# /dev/shm defaults but add nodev,nosuid,noexec

sudo mount -o remount /tmp
sudo mount -o remount /dev/shm

Test applications that use /tmp. If any fail, document as exception or move to /var/tmp.

Step 4: Apply SSH hardening

This is the step that locks people out

Everything here is arranged around one rule: do not close the session you are holding until a second, independent session has proved it works. The shell you are already in is the only recovery path you have if the change is wrong and the host has no console.

Two further hazards specific to this step:

An earlier revision overwrote /etc/ssh/sshd_config with a heredoc whose body was elided to # ... hardened sshd_config .... That replaces the file with a single comment. sshd -t then passes, because a comment-only config is syntactically valid, and the reload quietly drops the host to compiled-in defaults — the opposite of hardening, with no error to notice.
Version-gated keywords are fatal, not ignored. PerSourcePenalties needs OpenSSH 9.8+, PerSourceMaxStartups 9.5+; RHEL 9 ships 8.7p1. See linux-sshd-configuration before adding any of them.

Use a drop-in rather than rewriting the main file. Backing the change out then means deleting one file, not restoring a backup you hope is current.

Service impact possiblearm a safety net, then apply— The scheduled revert removes the drop-in and reloads sshd if you do not cancel it. If the change locks you out, the host repairs itself in ten minutes instead of needing a console.

DROPIN=/etc/ssh/sshd_config.d/50-hardening.conf

# ssh.service on Debian-family, sshd.service on RHEL-family. Resolve it.
SSHD_UNIT=$(systemctl list-unit-files --no-legend 'ssh.service' 'sshd.service' \
| awk 'NR==1{print $1}')

# Confirm the main file actually reads the drop-in directory first.
grep -i '^include' /etc/ssh/sshd_config || echo 'NO Include line - stop here'

# 1. Arm the revert BEFORE changing anything.
sudo bash -c "nohup sh -c 'sleep 600; rm -f $DROPIN; systemctl reload $SSHD_UNIT' >/dev/null 2>&1 &"

# 2. Place your reviewed baseline at $DROPIN now (see linux-sshd-configuration).

# 3. Validate before anything restarts. This exits non-zero on a bad file.
sudo sshd -t || echo 'CONFIG REJECTED - do not reload'

# 4. Reload only if step 3 passed.
sudo systemctl reload "$SSHD_UNIT"

Now open a second terminal and prove access before touching the first:

Read-only / Safeprove access from an independent session— Run these from another machine, not from the session you already hold. The last command must fail; if it succeeds, password authentication is still enabled.

USER_AT_HOST=admin@198.51.100.10

ssh "$USER_AT_HOST" 'echo reachable'

# Confirm the effective config is what you intended, not what you wrote.
ssh "$USER_AT_HOST" 'sudo sshd -T | grep -iE "permitrootlogin|passwordauthentication|maxauthtries"'

# This MUST be refused.
ssh -o PubkeyAuthentication=no -o PreferredAuthentications=password "$USER_AT_HOST" \
&& echo 'FAIL: password auth still accepted' || echo 'OK: password auth refused'

Only once that second session works, cancel the revert:

Configuration changestand down the safety net

sudo pkill -f 'sleep 600' || true
ls -l /etc/ssh/sshd_config.d/50-hardening.conf   # still present = revert cancelled

Step 5: Apply auditd

Service impact possibleapt install

sudo apt install auditd

# Apply CIS rules
sudo cp /etc/audit/rules.d/10-base-config.rules /etc/audit/rules.d/10-base-config.rules.bak

sudo tee /etc/audit/rules.d/10-base-config.rules <<EOF
# CIS recommendations
-w /etc/passwd -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/sudoers -p wa -k identity
-w /var/log/auth.log -p wa -k authlog
...
EOF

sudo augenrules --load
sudo systemctl restart auditd
sudo auditctl -l     # list loaded rules

Step 6: Apply PAM password policy

Configuration changeapt install

# Install pam_pwquality
sudo apt install libpam-pwquality

# Edit /etc/pam.d/common-password
# Add or modify the pam_pwquality.so line:
# password requisite pam_pwquality.so retry=3 minlen=14 dcredit=-1 ucredit=-1 lcredit=-1 ocredit=-1 difok=4

Test by attempting a weak password:

Read-only / Safe-k

sudo -k    # clear cached credentials
su - user
# Enter a weak password - should fail

Step 7: Disable unnecessary services

Configuration changesystemctl disable

sudo systemctl disable --now avahi-daemon
sudo systemctl disable --now bluetooth
sudo systemctl disable --now cups

Document any services that must stay.

Step 8: Re-scan

Read-only / Safeoscap xccdf

sudo oscap xccdf eval \
--profile xccdf_org.ssgproject.content_profile_cis_level1_server \
--results hardened-scan.xml \
--report hardened-report.html \
/usr/share/xml/scap/content/ssg-ubuntu2204-ds.xml

Compare the baseline and hardened reports. The PASS count should be significantly higher.

Step 9: Document

HARDENING REPORT
================
Host: <host>
Date: 2026-08-09
Profile: CIS Level 1

Baseline PASS: N
Hardened PASS: M (M > N)

Changes applied:
- sysctl hardening (file: /etc/sysctl.d/99-hardening.conf)
- mount options (/tmp tmpfs with nodev,nosuid,noexec)
- SSH hardening (publickey, modern ciphers)
- auditd enabled with CIS rules
- PAM password policy enforced
- Avahi, bluetooth, cups disabled

Exceptions:
- CIS 1.5.x: <app requires /tmp exec>; approved by <name>

Verification:
- Applications functional: nginx, postgresql, ssh all pass smoke test
- OpenSCAP PASS count increased
- No new MAC denials
- No service regressions

Common patterns

Issue	Resolution
Application needs /tmp exec	Document exception or move to /var/tmp
SSH breaks for one user	Check authorized_keys; verify shell access
auditd fills disk	Configure audit log rotation, max log size
PAM password policy too strict	Adjust minlen and complexity, document
Service won’t start after disabling	Check dependencies; the service may need another

Escalation

Escalate when:

The hardening breaks an application that has no fix.
Compliance scan shows no improvement.
An exception is needed for a security-sensitive control.
A second host is being hardened and patterns differ.

Bring: target host, baseline scan, hardened scan, exceptions, recommendation.