Skip to main content
RunBook Academy

← All runbooks in Ceph

low riskservice affecting~15 min

Troubleshoot a RGW client

1 · Prerequisites

Confirm every item is in place before any state change.

  • An RGW client (S3) is failing or slow
  • Bucket access is denied or slow

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

  • · radosgw-admin zone get shows the data pool is healthy
  • · radosgw-admin user list shows the user exists
  • · RGW daemons are healthy
  • · Load balancer health is healthy

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

  1. 1Run an S3 client test: aws s3 ls s3://<bucket>
  2. 2Inspect radosgw-admin log show
  3. 3Check the bucket policy
  4. 4Check the user caps
  5. 5Check the data pool utilisation

4 · Verification

Confirm the procedure actually fixed the problem.

  • The S3 client connects
  • The bucket is accessible
  • I/O latency is acceptable

5 · Rollback

If verification fails, undo the procedure in reverse order.

  • If auth fails: re-issue keys
  • If the data pool is full: add capacity
  • If the daemon is slow: restart, check the load

6 · Escalation

When the runbook isn't enough, contact:

  • · Storage team

Troubleshoot a RGW client

This runbook covers Troubleshoot a RGW client on a Ceph Tentacle cluster.

When to use

  • Active production scenario matching the impact (service-affecting)
  • The prerequisites are met

Prerequisites

Before starting this runbook, verify each prerequisite has been met. If a prerequisite is not met, resolve it first or escalate.

  • An RGW client (S3) is failing or slow
  • Bucket access is denied or slow

Pre-checks

Collect evidence before taking action. Evidence is the proof the runbook worked.

# radosgw-admin zone get shows the data pool is healthy
# radosgw-admin user list shows the user exists
# RGW daemons are healthy
# Load balancer health is healthy

Procedure

Step 1

Run an S3 client test: aws s3 ls s3://<bucket>

Step 2

Inspect radosgw-admin log show

Step 3

Check the bucket policy

Step 4

Check the user caps

Step 5

Check the data pool utilisation

Verification

After completing the procedure, verify the result:

# The S3 client connects
# The bucket is accessible
# I/O latency is acceptable

Rollback

If the procedure does not produce the expected verification, roll back:

  • If auth fails: re-issue keys
  • If the data pool is full: add capacity
  • If the daemon is slow: restart, check the load

Escalation

If you cannot roll back or the rollback fails, escalate:

  • Storage team

References

  1. Ceph documentation