Troubleshoot a RGW client
1 · Prerequisites
Confirm every item is in place before any state change.
- An RGW client (S3) is failing or slow
- Bucket access is denied or slow
2 · Pre-checks
Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.
- · radosgw-admin zone get shows the data pool is healthy
- · radosgw-admin user list shows the user exists
- · RGW daemons are healthy
- · Load balancer health is healthy
3 · Procedure
Execute each step in order. Verify the expected output of a step before moving to the next.
- 1Run an S3 client test: aws s3 ls s3://<bucket>
- 2Inspect radosgw-admin log show
- 3Check the bucket policy
- 4Check the user caps
- 5Check the data pool utilisation
4 · Verification
Confirm the procedure actually fixed the problem.
- ✓The S3 client connects
- ✓The bucket is accessible
- ✓I/O latency is acceptable
5 · Rollback
If verification fails, undo the procedure in reverse order.
- ↶If auth fails: re-issue keys
- ↶If the data pool is full: add capacity
- ↶If the daemon is slow: restart, check the load
6 · Escalation
When the runbook isn't enough, contact:
- · Storage team
Troubleshoot a RGW client
This runbook covers Troubleshoot a RGW client on a Ceph Tentacle cluster.
When to use
- Active production scenario matching the impact (
service-affecting) - The prerequisites are met
Prerequisites
Before starting this runbook, verify each prerequisite has been met. If a prerequisite is not met, resolve it first or escalate.
- An RGW client (S3) is failing or slow
- Bucket access is denied or slow
Pre-checks
Collect evidence before taking action. Evidence is the proof the runbook worked.
# radosgw-admin zone get shows the data pool is healthy
# radosgw-admin user list shows the user exists
# RGW daemons are healthy
# Load balancer health is healthy
Procedure
Step 1
Run an S3 client test: aws s3 ls s3://<bucket>
Step 2
Inspect radosgw-admin log show
Step 3
Check the bucket policy
Step 4
Check the user caps
Step 5
Check the data pool utilisation
Verification
After completing the procedure, verify the result:
# The S3 client connects
# The bucket is accessible
# I/O latency is acceptable
Rollback
If the procedure does not produce the expected verification, roll back:
- If auth fails: re-issue keys
- If the data pool is full: add capacity
- If the daemon is slow: restart, check the load
Escalation
If you cannot roll back or the rollback fails, escalate:
- Storage team