Skip to main content
RunBook Academy

← All break/fix scenarios in Ceph

intermediateceph-net-degraded~22 min

Network degraded - bond member down on multiple OSD hosts

Reported symptoms

  • ●OSD latency p99 increases
  • ●ceph daemonperf <osd> shows a sub-percent of the cluster affected
  • ●No OSDs are down

Evidence

  • · ip -s link show <bond> shows one member in NO-CARRIER
  • · ceph daemonperf <osd> shows latency correlated with the bond member loss
  • · /var/log/messages shows link flap on the bond
Diagnosis and resolutionclick to reveal

Root cause

One bond member is down on multiple OSD hosts; traffic moves to the surviving member; latency increases.

Remediation

Identify the affected bond member; replace the physical layer if necessary; the bond recovers; latency returns to baseline.

Verification

Bond member is up; ip -s link show <bond> shows both members up; ceph daemonperf returns to baseline latency; cluster is HEALTH_OK.

Prevention

LACP with redundant switches; SNMP / port-counter monitoring; bond failure alerts.

Architecture

The student examines the broken architecture diagram and identifies the responsible component. The cluster is running Ceph Tentacle (20.2.x); the incident is one a production operator must diagnose from the evidence presented.

Symptoms

  • OSD latency p99 increases
  • ceph daemonperf &lt;osd&gt; shows a sub-percent of the cluster affected
  • No OSDs are down

Evidence

  • ip -s link show <bond> shows one member in NO-CARRIER
  • ceph daemonperf &lt;osd&gt; shows latency correlated with the bond member loss
  • /var/log/messages shows link flap on the bond

Student investigation

The student follows the methodology: define the symptom, determine the impact, gather evidence, identify the component, form a hypothesis, test safely, restore, validate.

Progressive hints

  • Hint 1: check the cluster state with ceph -s first.
  • Hint 2: read ceph health detail and identify the affected PGs / OSDs / daemons.
  • Hint 3: use ceph osd tree, ceph pg dump, or ceph mds stat as the next-level diagnostic.
  • Hint 4: the root cause is documented in the frontmatter root_cause field.

Validation

The student runs the verification steps and confirms the symptom cleared.

Root cause

See the frontmatter root_cause field.

Remediation

Apply the fix from the frontmatter remediation field.

Prevention

Apply the prevention measures from the frontmatter prevention field.