Skip to main content
RunBook Academy

CephXCV · Maintenance FlagsMaintenance Flags

The noout flag in depth

Intermediate⏱ ~17 minceph

What you'll learn

  • Explain the mechanism noout suppresses
  • Choose between cluster-wide and scoped variants
  • Apply it for the right duration
  • Recognise when it is the wrong tool

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Not yet marked complete on this device.

Why this matters in production

noout suppresses one specific automatic behaviour, and knowing exactly which one determines when it helps.

The mechanism it suppresses

An OSD stops responding
  → peers report it down after osd_heartbeat_grace
  → the monitors mark it down
  → after mon_osd_down_out_interval, the monitors mark it out
  → CRUSH recomputes without it
  → recovery begins

noout suppresses only the fourth step. The OSD is still marked down, PGs are still degraded, and clients still experience the failure detection window.

ceph config get osd mon_osd_down_out_interval
ceph osd set noout
ceph osd dump | grep flags
noout prevents:  the automatic out marking, and therefore the recovery
noout does not:  prevent the down marking, prevent degradation,
                 keep clients from noticing, or stop I/O to that OSD failing

Cluster-wide and scoped variants

# cluster-wide
ceph osd set noout

# a single OSD
ceph osd add-noout osd.44
ceph osd rm-noout osd.44

# a whole host
ceph osd add-noout ceph-03
ceph osd dump | grep -E 'flags|noout'
ceph osd tree | grep noout

The scoped variants are almost always better: a cluster-wide flag suppresses the automatic recovery for every OSD including ones that fail genuinely during the window.

Duration

WorkDurationApproach
A rebootminutesscoped noout on the host
A kernel upgradetens of minutesscoped noout, or maintenance mode
Hardware workan hour or moreceph orch host maintenance enter
A permanent removal—not noout; drain properly
ceph orch host maintenance enter ceph-03
ceph orch host maintenance exit ceph-03

Maintenance mode sets the appropriate flags, stops the daemons, and records the host’s state, which is why it is preferable beyond a reboot.

When it is the wrong tool

SituationCorrect tool
An OSD that is not coming backlet it be marked out
Draining for removalceph orch osd rm
Reducing recovery impactthrottles, not suppression
A flapping OSDfix the cause
An extended outage of unknown durationlet recovery run
noout during an outage of unknown duration means the cluster stays degraded
with no recovery, indefinitely.

Quiz

Knowledge check · 4 questions

  1. Q1. Which step in the failure sequence does `noout` suppress?

  2. Q2. A cluster-wide `noout` during maintenance on one host is equivalent to a scoped one.

  3. Q3. Choose the right suppression for a maintenance window.

    A two-hour hardware maintenance is planned on one host in a 96-OSD cluster. The team plans to set cluster-wide noout.

  4. Q4. Why is `noout` wrong for an outage of unknown duration?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Use ceph osd add-noout on the specific host rather than the cluster-wide flag — a cluster-wide suppression leaves an unrelated failure elsewhere unrecovered and unnoticed. Remember it suppresses the recovery, not the degradation, so it is wrong for an outage of unknown length.

Cross-course references

  • Kubernetes: scoped tolerations beat cluster-wide scheduler changes for the same reason
  • Linux: narrowly scoped suppressions leave the rest of the system protected