The Linux baseline for a Ceph host

Why this matters in production

Ceph runs in containers on a fairly ordinary Linux host, so the baseline is short. The items that matter are the ones where the default is wrong for a storage server rather than for a general-purpose machine.

Distribution and kernel

Use a distribution the Ceph release supports and that your team already operates. The container runtime insulates Ceph from most distribution differences, so the choice is largely an operational one.

Kernel version matters for one thing specifically: the kernel RBD and CephFS clients. A host that mounts Ceph filesystems or maps RBD images needs a kernel new enough for the features you use; a host that only runs OSD containers does not.

uname -r
cat /etc/os-release

The settings that matter

File descriptors. OSDs open many files and sockets:

# /etc/security/limits.d/ceph.conf
*  soft  nofile  1048576
*  hard  nofile  1048576

# for containerised daemons, the systemd unit matters more
[Service]
LimitNOFILE=1048576
LimitNPROC=1048576

PID limits. Dense OSD hosts run many threads:

sysctl -w kernel.pid_max=4194304
sysctl -w kernel.threads-max=2097152

Memory behaviour.

sysctl -w vm.swappiness=10
sysctl -w vm.min_free_kbytes=4194304
sysctl -w vm.vfs_cache_pressure=50

Low swappiness matters because an OSD swapped out becomes a latency outlier that affects every PG it hosts.

Network buffers, for high-throughput hosts:

sysctl -w net.core.rmem_max=268435456
sysctl -w net.core.wmem_max=268435456
sysctl -w net.ipv4.tcp_rmem="4096 87380 268435456"
sysctl -w net.ipv4.tcp_wmem="4096 65536 268435456"

Persist these in /etc/sysctl.d/ rather than setting them at runtime.

What does not matter much

• Root filesystem choice — the daemons store data on their own devices; ext4 or XFS on root is equally fine
• Most tuned profiles — throughput-performance is reasonable and the difference from the default is modest
• CPU governor — worth setting to performance on latency-sensitive clusters, marginal otherwise

Verifying

ulimit -n
sysctl vm.swappiness kernel.pid_max
systemctl show ceph-osd@12 -p LimitNOFILE
free -h

Check the limits as the daemon sees them, not as the shell sees them — containerised daemons take theirs from the systemd unit.

Quiz

Production discipline

Verify limits as the daemon sees them rather than as the shell does; containerised daemons take theirs from the systemd unit and the difference surfaces only under load. Persist sysctl settings in /etc/sysctl.d/ so a reboot does not quietly revert the host to defaults.

Cross-course references

• Kubernetes: container resource limits differ from host limits in the same confusing way
• Linux: this is ordinary storage-server tuning with Ceph-specific emphasis