Skip to main content
RunBook Academy

← All break/fix scenarios in Ceph

advancedceph-rgw-multisite~22 min

RGW multisite sync falls behind - replication lag

Reported symptoms

  • ●radosgw-admin sync status shows increasing lag
  • ●Objects written to the master zone do not appear in the secondary zone
  • ●The lag metric increases steadily

Evidence

  • · radosgw-admin sync status lists the lag per zone
  • · ceph -s reports the master zone as HEALTH_OK
  • · radosgw-admin log show lists the sync errors
Diagnosis and resolutionclick to reveal

Root cause

The RGW sync process is behind. Possible causes: secondary zone is slow; sync worker is overwhelmed; network between zones is congested.

Remediation

Identify the cause; tune rgw_zone_sync_thread_count; verify network bandwidth; restart the sync workers.

Verification

radosgw-admin sync status reports lag: 0s; the secondary zone catches up; objects are visible in the secondary zone.

Prevention

Monitor sync lag; tune sync workers; rehearse sync lag incident.

Architecture

The student examines the broken architecture diagram and identifies the responsible component. The cluster is running Ceph Tentacle (20.2.x); the incident is one a production operator must diagnose from the evidence presented.

Symptoms

  • radosgw-admin sync status shows increasing lag
  • Objects written to the master zone do not appear in the secondary zone
  • The lag metric increases steadily

Evidence

  • radosgw-admin sync status lists the lag per zone
  • ceph -s reports the master zone as HEALTH_OK
  • radosgw-admin log show lists the sync errors

Student investigation

The student follows the methodology: define the symptom, determine the impact, gather evidence, identify the component, form a hypothesis, test safely, restore, validate.

Progressive hints

  • Hint 1: check the cluster state with ceph -s first.
  • Hint 2: read ceph health detail and identify the affected PGs / OSDs / daemons.
  • Hint 3: use ceph osd tree, ceph pg dump, or ceph mds stat as the next-level diagnostic.
  • Hint 4: the root cause is documented in the frontmatter root_cause field.

Validation

The student runs the verification steps and confirms the symptom cleared.

Root cause

See the frontmatter root_cause field.

Remediation

Apply the fix from the frontmatter remediation field.

Prevention

Apply the prevention measures from the frontmatter prevention field.