Capstone: Build a production Linux cluster from scratch

This capstone ties together everything in the course. You build a production-grade 3-node cluster from scratch, wire in the boring dependencies that take clusters down, then break it on purpose and prove you can put it back.

Objective

By the end you will have a running 3-node Pacemaker cluster whose fencing has been proven by an actual fence event, whose logs land in Loki, whose metrics land in Prometheus, and whose backup has been restored - not merely taken. You will also have a written record of five injected failures and how each one was detected and recovered.

Architecture

• Three cluster nodes: node1, node2, node3. Each has three NICs.
• Management VLAN 10, 10.0.0.0/24 - SSH, pcs, corosync ring 0.
• Application VLAN 20, 10.1.0.0/24 - service traffic and the VIP.
• Storage VLAN 30, 10.2.0.0/24 - iSCSI to the storage target.
• One infrastructure host, infra01 (10.0.0.5): DNS, NTP, internal CA, Prometheus, Loki, Grafana.
• One backup host, backup01 (10.0.0.6): the Borg repository. Deliberately not a cluster node.

The last line is the point of the architecture. If the backup lives on the cluster, the event that destroys the cluster destroys the backup with it.

Requirements

• Five VMs, Ubuntu 24.04 LTS or RHEL 9. Pick one family and stay on it - the package commands differ.
• Cluster nodes: 4 vCPU, 8 GB RAM, 50 GB disk, 3 NICs.
• infra01 and backup01: 2 vCPU, 4 GB RAM, 40 GB disk, 1 NIC.
• A BMC / iDRAC / virtual BMC per cluster node, each on its own address, reachable from the management VLAN.
• Root or full sudo on all five hosts.
• Roughly 4 hours. Do not start Task 9 with less than an hour left - fencing tests need time to settle.

Scenario

You have been handed five fresh VMs and a ticket that says “build the new cluster”. Nothing exists yet: no DNS, no time source, no certificates, no accounts. This is the ordinary starting position, and the order in which you build matters more than any single command.

Task 1: Provision and name the hosts

Addresses are per node. Write the table down before you touch a keyboard.

The service VIP is 10.1.0.100 on the application VLAN. It is not on the management VLAN and it is nowhere near the BMC range.

$ sudo hostnamectl set-hostname node1   # node2, node3 on their own hosts

Task 2: Configure the network

The single most common way to lose an afternoon here is to paste node1’s addresses onto all three nodes. Duplicate addresses on the management VLAN produce ARP conflicts and a corosync ring that will not form, and the symptom - “the cluster will not start” - points nowhere near the cause.

Set N once per node, then paste the same block everywhere.

$ # node1: N=10   node2: N=11   node3: N=12
N=10

sudo nmcli connection add type ethernet ifname eth0 con-name mgmt \
ipv4.method manual ipv4.addresses 10.0.0.$N/24 \
ipv4.gateway 10.0.0.1 ipv4.dns 10.0.0.5 ipv6.method disabled

sudo nmcli connection add type ethernet ifname eth1 con-name app \
ipv4.method manual ipv4.addresses 10.1.0.$N/24 ipv6.method disabled

sudo nmcli connection add type ethernet ifname eth2 con-name storage \
ipv4.method manual ipv4.addresses 10.2.0.$N/24 ipv6.method disabled

sudo nmcli connection up mgmt
sudo nmcli connection up app
sudo nmcli connection up storage
ip -br a

Confirm every node sees a different address before continuing.

$ for n in 10 11 12; do ping -c1 -W1 10.0.0.$n >/dev/null && echo "10.0.0.$n up"; done

10.0.0.10 up
10.0.0.11 up
10.0.0.12 up

Task 3: DNS and NTP before anything else

Clusters are time-sensitive and name-sensitive. Corosync tolerates clock skew poorly, Kerberos rejects it outright, and TLS certificates fail validation on a skewed host. Build this first.

$ sudo apt install -y bind9        # RHEL: sudo dnf install -y bind
sudo tee /etc/bind/db.lab.internal >/dev/null <<'EOF'
$TTL 300
@   IN SOA infra01.lab.internal. admin.lab.internal. ( 1 300 120 604800 300 )
@         IN NS  infra01.lab.internal.
infra01   IN A   10.0.0.5
backup01  IN A   10.0.0.6
node1     IN A   10.0.0.10
node2     IN A   10.0.0.11
node3     IN A   10.0.0.12
cluster   IN A   10.1.0.100
EOF
sudo named-checkzone lab.internal /etc/bind/db.lab.internal

$ sudo apt install -y chrony
echo 'server 10.0.0.5 iburst' | sudo tee /etc/chrony/sources.d/lab.sources
sudo systemctl restart chronyd
chronyc sources -v
chronyc tracking | grep -E 'Leap status|System time'

Do not continue until chronyc tracking reports Leap status: Normal and a system-time offset under 100 ms on all three nodes.

Task 4: Central identity

Administrators should not have local accounts on cluster nodes. Join each node to the directory so that access is revoked in one place.

$ sudo apt install -y sssd-ldap ldap-utils
sudo tee /etc/sssd/sssd.conf >/dev/null <<'EOF'
[sssd]
domains = lab.internal
services = nss, pam

[domain/lab.internal]
id_provider = ldap
auth_provider = ldap
ldap_uri = ldaps://infra01.lab.internal
ldap_search_base = dc=lab,dc=internal
ldap_tls_cacert = /etc/ssl/certs/lab-ca.pem
cache_credentials = true
EOF
sudo chmod 0600 /etc/sssd/sssd.conf
sudo systemctl enable --now sssd
id sysadmin1

cache_credentials = true matters here: if infra01 is down, a cached administrator can still log in to fix it. Without it, the directory outage and the cluster outage lock you out together.

Task 5: An internal CA and host certificates

$ sudo install -d -m 0700 /root/ca
sudo openssl req -x509 -newkey rsa:4096 -sha256 -days 3650 -nodes \
-keyout /root/ca/ca.key -out /root/ca/ca.pem \
-subj '/CN=lab.internal Internal CA'
sudo chmod 0400 /root/ca/ca.key

$ openssl req -newkey rsa:2048 -nodes -keyout node1.key -out node1.csr \
-subj '/CN=node1.lab.internal'
sudo openssl x509 -req -in node1.csr -CA /root/ca/ca.pem -CAkey /root/ca/ca.key \
-CAcreateserial -days 397 -sha256 \
-extfile <(printf 'subjectAltName=DNS:node1.lab.internal,DNS:cluster.lab.internal') \
-out node1.pem
openssl x509 -in node1.pem -noout -dates -ext subjectAltName

397 days is deliberate: it is the maximum public-trust lifetime, and using it internally forces you to build renewal into the runbook now rather than discovering you have no renewal process in a year. Record both expiry dates in the runbook library from Task 13.

Task 6: Secrets handling

Every credential in this build - the BMC password, the Borg passphrase, the directory bind password - is written to a 0400-mode file owned by root, never to a command line.

$ sudo install -d -m 0700 /etc/lab-secrets
umask 077
read -rs -p 'BMC password: ' PW; echo
printf '%s' "$PW" | sudo tee /etc/lab-secrets/bmc >/dev/null
unset PW
sudo chmod 0400 /etc/lab-secrets/bmc
sudo ls -l /etc/lab-secrets

Task 7: Shared storage

Present one iSCSI LUN from infra01 and make it visible on all three nodes over the storage VLAN. The cluster will mount it on exactly one node at a time.

$ sudo apt install -y open-iscsi
sudo iscsiadm -m discovery -t sendtargets -p 10.2.0.5
sudo iscsiadm -m node --login
lsblk --scsi | grep -i iscsi

$ # Confirm the device is the LUN and not a local disk before running this.
lsblk -o NAME,SIZE,MODEL /dev/sdb
sudo mkfs.xfs /dev/sdb

Task 8: Build the cluster

Package names and repositories differ by family. Pick your branch.

$ sudo apt install -y pacemaker pcs fence-agents

$ sudo dnf config-manager --set-enabled highavailability
sudo dnf install -y pcs pacemaker fence-agents-all

pcs host auth authenticates as the hacluster system account against the pcsd daemon. The package creates the account locked and leaves pcsd stopped, so both steps below are mandatory and both are the usual reason authentication fails.

$ sudo systemctl enable --now pcsd
read -rs -p 'hacluster password: ' PW; echo
echo "hacluster:$PW" | sudo chpasswd
unset PW

$ sudo pcs host auth node1 node2 node3 -u hacluster
sudo pcs cluster setup mycluster node1 node2 node3 --start --enable
sudo pcs status

Cluster name: mycluster
* 3 nodes configured
* 0 resource instances configured
Node List:
* Online: [ node1 node2 node3 ]

Stop here until all three nodes read Online. A two-of-three cluster will happily start and will then behave in ways that look like resource bugs.

Task 9: Fencing, proven before any resource exists

Three fence devices, one per node, each pointing at that node’s own BMC address. The password is read from the protected file written in Task 6, never passed as passwd=.

$ sudo tee /etc/lab-secrets/bmc.sh >/dev/null <<'EOF'
#!/bin/sh
cat /etc/lab-secrets/bmc
EOF
sudo chmod 0500 /etc/lab-secrets/bmc.sh

$ sudo pcs stonith create node1_ipmi fence_ipmilan \
pcmk_host_list="node1" ip="10.0.100.10" username="admin" \
password_script="/etc/lab-secrets/bmc.sh" lanplus=1

sudo pcs property set stonith-enabled=true

# Must print nothing: no cleartext password reached the CIB.
sudo pcs stonith config | grep 'passwd='

Now prove it. An unfenced cluster with a shared LUN is a data-loss incident waiting for its trigger, and a fence device that has never fired is a configuration, not a capability.

$ sudo pcs stonith fence node3 --off
sudo pcs status | grep -A3 'Node List'

Node List:
* Online: [ node1 node2 ]
* OFFLINE: [ node3 ]

Power node3 back on, confirm it rejoins, then repeat for node1 and node2 from another node. All three must be proven. A fence device that works for two of three nodes fails exactly when the untested node is the one that has gone bad.

Task 10: Resources and constraints

$ sudo pcs resource create vip ocf:heartbeat:IPaddr2 \
ip=10.1.0.100 cidr_netmask=24 op monitor interval=10s

sudo pcs resource create shared_fs ocf:heartbeat:Filesystem \
device=/dev/sdb directory=/srv/shared fstype=xfs \
op monitor interval=20s

sudo pcs resource create web systemd:nginx op monitor interval=15s

sudo pcs constraint colocation add web with vip INFINITY
sudo pcs constraint colocation add web with shared_fs INFINITY
sudo pcs constraint order vip then web
sudo pcs constraint order shared_fs then web
sudo pcs status resources

Colocation and order are independent statements. Colocation alone would place web beside the VIP but would let it start before the address is up; the order constraints are what make the start sequence correct.

Task 11: Monitoring

$ sudo apt install -y prometheus-node-exporter   # RHEL: dnf install -y node_exporter
sudo systemctl enable --now prometheus-node-exporter
curl -s localhost:9100/metrics | head -3

$ scrape_configs:
- job_name: cluster_nodes
  static_configs:
    - targets:
        - node1.lab.internal:9100
        - node2.lab.internal:9100
        - node3.lab.internal:9100

Add at least one alert rule that would have caught something in Task 15: a node that has stopped reporting, and a filesystem above 85 per cent. Then add Prometheus as a Grafana data source and build one host dashboard.

Task 12: Central logging

$ curl -fsSL https://keys.datadoghq.com/DATADOG_APT_KEY_CURRENT.public \
| sudo gpg --dearmor -o /usr/share/keyrings/vector.gpg
echo "deb [signed-by=/usr/share/keyrings/vector.gpg] \
https://apt.vector.dev/ stable vector-0" \
| sudo tee /etc/apt/sources.list.d/vector.list
sudo apt update && sudo apt install -y vector

The file is vector.yaml, so its contents must be YAML. Vector picks its parser from the extension, and a TOML body in a .yaml file fails to load - the daemon exits and ships nothing, silently, which is the worst possible failure mode for a logging pipeline.

$ sources:
journald:
  type: journald
  current_boot_only: false

sinks:
loki:
  type: loki
  inputs: [journald]
  endpoint: http://loki.lab.internal:3100
  encoding:
    codec: json
  labels:
    host: "{{ host }}"
    unit: "{{ _SYSTEMD_UNIT }}"
    job: journald

Validate before enabling. vector validate parses the file and checks that every sink’s inputs exist.

$ sudo vector validate /etc/vector/vector.yaml
sudo systemctl enable --now vector
logcli query '{job="journald", host="node1"}' --limit 5

Task 13: Backup that can actually be restored

$ sudo install -d -m 0700 /etc/borg
openssl rand -base64 32 | sudo tee /etc/borg/passphrase >/dev/null
sudo chmod 0400 /etc/borg/passphrase
export BORG_PASSCOMMAND='sudo cat /etc/borg/passphrase'

The repository lives on backup01, not on the cluster. Task 1’s architecture note becomes real here: a backup on the protected host is not a backup, it is a second copy that dies in the same incident.

$ borg init --encryption=repokey-blake2 borg@backup01:/srv/borg/cluster
borg key export borg@backup01:/srv/borg/cluster /etc/borg/repo-key.txt
sudo chmod 0400 /etc/borg/repo-key.txt
# Copy repo-key.txt and the passphrase to storage outside this lab.

$ borg create --stats --compression zstd \
borg@backup01:/srv/borg/cluster::'{hostname}-{now:%Y-%m-%dT%H:%M}' \
/etc /var/lib /srv

borg check --verify-data borg@backup01:/srv/borg/cluster
borg list borg@backup01:/srv/borg/cluster

Now the step that separates a backup from a belief. The task is not done until an archive has been extracted and compared.

$ ARCHIVE=$(borg list --short --last 1 borg@backup01:/srv/borg/cluster)
borg extract --dry-run --list \
"borg@backup01:/srv/borg/cluster::$ARCHIVE" etc/hostname

mkdir -p /var/tmp/restore && cd /var/tmp/restore
borg extract "borg@backup01:/srv/borg/cluster::$ARCHIVE" etc
diff -r /var/tmp/restore/etc /etc | head

Wrap borg create, borg check and borg prune in a systemd service plus a daily timer, and add a Prometheus alert that fires when the newest archive is older than 26 hours. A backup job that fails silently is indistinguishable from no backup.

Task 14: Security baseline

1. Run the CIS-aligned OpenSCAP scan from the self-audit lab and record the initial score.
2. SSH: PubkeyAuthentication yes, PasswordAuthentication no, PermitRootLogin no. Confirm with sshd -T | grep -E 'permitrootlogin|passwordauthentication'.
3. nftables: default-drop input; allow 22 from the management VLAN only, 2224 (pcsd) and 5405/udp (corosync) between nodes only, 9100 from infra01 only, 80/443 on the app VLAN.
4. SELinux enforcing (RHEL) or AppArmor enforce (Ubuntu). Confirm with getenforce or aa-status.
5. auditd rules for /etc/sudoers.d, /etc/shadow, /etc/pacemaker and /etc/lab-secrets.
6. Re-run the OpenSCAP scan and record the improvement. The delta is the deliverable, not the raw score.

Task 15: Configuration management

Everything above was done by hand once so that you understand it. Now make it repeatable. Write an Ansible role that reproduces Tasks 3, 4, 6, 11, 12 and 14 - the parts that are identical on every node - and run it against a fourth, freshly built node.

$ ansible-playbook -i inventory site.yml --check --diff
ansible-playbook -i inventory site.yml
ansible-playbook -i inventory site.yml --check --diff   # must report zero changes

The second --check run is the test. If it reports changes, the role is not idempotent and it will fight your hand-made state forever.

Task 16: Documentation

• Architecture diagram with every address from Task 1 on it, including the BMCs.
• Five runbooks minimum: planned failover, unplanned node loss, filesystem full, certificate renewal, restore from Borg.
• Three checklists minimum: pre-deployment, pre-maintenance, post-patching validation.
• A change-management record for the cluster itself, including the CIB backup location.

Task 17: Break it on purpose

Five injections. Each one states what you do, what you must observe, and what counts as a pass. Record the timings - they are your real RTO, not the one in the design document.

17.1 Planned failover

$ sudo pcs node standby node1
sudo pcs status resources
sudo pcs node unstandby node1

Pass: vip, shared_fs and web all report Started on node2 or node3, curl -sI http://10.1.0.100/ returns 200, and total unavailability from the standby command to the first successful curl is under 60 seconds.

17.2 Unplanned node loss and fencing

$ sudo pcs stonith fence node2 --off
sudo pcs status
journalctl -u pacemaker-fenced --since '-5 min' | grep -i 'operation.*off.*ok'

Pass: the fence action is logged as successful, node2 shows OFFLINE, resources restart elsewhere, and shared_fs mounts on exactly one node. Confirm the last point explicitly - two mounts is the failure this whole design exists to prevent.

17.3 Restore from backup

Delete /srv/shared/testfile, then restore only that path from the newest archive and diff it against what you deleted. Pass: byte identical, and you did it from the runbook without improvising.

17.4 Monitoring fires

$ sudo systemctl stop prometheus-node-exporter
# Wait for the alert rule's "for" duration plus one scrape interval.
sudo systemctl start prometheus-node-exporter

Pass: the alert moves pending, then firing, reaches the receiver, and resolves after the exporter returns. An alert that fires but reaches nobody has failed the test.

17.5 Dependency failure, not cluster failure

Stop named on infra01 for five minutes. Pass: you can state from evidence which of Loki ingestion, SSSD lookups, Borg over SSH and Pacemaker itself degraded, and which survived - and your incident notes name DNS as the cause rather than the cluster.

Validation

$ sudo pcs status | grep -E 'Online|Started|Failed'
sudo pcs stonith config | grep -c 'fence_ipmilan'
chronyc tracking | grep 'Leap status'
curl -s -o /dev/null -w '%{http_code}\n' http://10.1.0.100/
curl -sG http://infra01.lab.internal:9090/api/v1/query \
--data-urlencode 'query=up{job="cluster_nodes"}' | grep -o '"value"' | wc -l
logcli query '{job="journald"}' --limit 1
borg list --last 1 borg@backup01:/srv/borg/cluster

Every line must pass:

• pcs status: three nodes Online, three resources Started, zero Failed actions.
• Exactly 3 fence_ipmilan devices, and all three have been fired at least once.
• chronyc: Leap status Normal on all three nodes.
• The VIP returns HTTP 200 from a client on the application VLAN.
• Prometheus returns up==1 for all three node targets.
• logcli returns at least one line for each of the three hosts.
• borg list shows an archive from within the last 24 hours, and it has been extracted and diffed.

Expected Outcome

Three nodes online with fencing proven on each. One VIP, one shared filesystem mounted on exactly one node, one service, correctly ordered. Metrics and logs from every node arriving centrally. A Borg repository on a host that is not part of the cluster, holding an archive you have restored from. A hardened baseline with a before-and-after scan. An Ansible role that reproduces the repeatable parts idempotently. Five injections recorded with timings.

Troubleshooting

Cleanup

Run in this order. Stopping the cluster before unmounting leaves the shared filesystem in an unclean state.

$ sudo pcs resource disable web shared_fs vip
sudo pcs resource delete web
sudo pcs resource delete shared_fs
sudo pcs resource delete vip
sudo pcs stonith delete node1_ipmi node2_ipmi node3_ipmi
sudo pcs cluster stop --all
sudo pcs cluster destroy --all

$ sudo systemctl disable --now vector prometheus-node-exporter sssd chronyd
sudo iscsiadm -m node --logout
sudo rm -rf /etc/lab-secrets /etc/borg /var/tmp/restore
sudo nmcli connection delete mgmt app storage

Delete the Borg repository on backup01 only when you are certain you no longer need the restore evidence, then destroy the five VMs.

What You Learned

• Objective 1: you built three nodes with distinct addresses and can explain why a pasted address block breaks corosync.
• Objective 2: you built DNS, NTP, identity, TLS and secrets before the cluster, and Task 17.5 showed you what each one takes down when it fails.
• Objective 3: fencing was proven on all three nodes by an actual power-off before any shared-storage resource existed.
• Objective 4: metrics, logs and an offsite backup all validated by query, and the backup validated by extraction rather than by the exit status of borg create.
• Objective 5: five injections, each with observed evidence and a recorded recovery time.