CephXLVIII · RGW TroubleshootingRGW Troubleshooting
S3 authentication failures and what each means
What you'll learn
- Map each S3 auth error to its cause
- Diagnose signature failures systematically
- Handle endpoint and region mismatches
- Verify credentials end to end
Prerequisites
None — start here.
Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18
Why this matters in production
The four common authentication errors have four unrelated causes, and the one everyone assumes — a wrong secret key — is not the most frequent. The code tells you which investigation to run.
The four errors
| Error | Means | Usual cause |
|---|---|---|
InvalidAccessKeyId | the access key is unknown | wrong key, wrong cluster, or the user was removed |
SignatureDoesNotMatch | the key exists, the signature does not verify | wrong secret, clock skew, or endpoint mismatch |
AccessDenied | authenticated, not authorised | policy, ACL, or suspended user |
RequestTimeTooSkewed | timestamp outside the tolerance | clock skew, explicitly |
SignatureDoesNotMatch is the one that misleads: the secret is often
correct and something else about the signed request differs.
Diagnosing SignatureDoesNotMatch
The signature covers the HTTP method, the canonical URI, the query string, selected headers, the payload hash, and a timestamp. Anything that alters one of those between the client and the gateway breaks it.
# 1. clock — check both ends
date -u
ssh rgw-01 date -u
chronyc tracking
# 2. endpoint — the host header is signed
aws --endpoint-url https://rgw.example.com s3 ls # signed for this host
# a proxy rewriting the Host header breaks the signature
# 3. the credential itself
radosgw-admin user info --uid=analytics | jq '.keys'
# 4. signature version
aws configure set default.s3.signature_version s3v4
| Cause | Check |
|---|---|
| Clock skew | compare client and gateway clocks |
| Host header rewritten by a proxy | test directly against a gateway |
| Path-style versus virtual-hosted addressing | try --endpoint-url with s3.addressing_style |
| Wrong secret | compare against user info |
| Signature version mismatch | force s3v4 |
The addressing style question
# virtual-hosted: bucket.rgw.example.com — needs wildcard DNS
aws configure set default.s3.addressing_style virtual
# path-style: rgw.example.com/bucket — no DNS requirement
aws configure set default.s3.addressing_style path
RGW supports both, but the host header differs between them and the header is signed. A client using virtual-hosted addressing against a deployment without wildcard DNS fails in a way that presents as a signature error.
ceph config get client.rgw rgw_dns_name
Verifying a credential end to end
export AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=...
aws --endpoint-url https://rgw.example.com s3 ls
aws --endpoint-url https://rgw.example.com s3 ls s3://known-bucket/
A successful s3 ls proves the credential authenticates. A failure at the
second step with success at the first is authorisation, not
authentication.
Quiz
Knowledge check · 4 questions
Q1. An S3 client receives SignatureDoesNotMatch. Which cause is most common?
Q2. Testing directly against a gateway, bypassing any reverse proxy, is an effective way to diagnose signature failures.
Q3. Diagnose signature failures affecting only some clients.
Some S3 clients receive SignatureDoesNotMatch while others using the same credentials succeed. The failing clients are all newer SDK versions. The deployment is behind a reverse proxy.
Q4. Why does an AccessDenied error indicate a different problem from SignatureDoesNotMatch?
Passing score: 75%. Answers are checked in this browser.
Production discipline
Read the specific error code before investigating; the four common S3 auth errors have unrelated causes and the code selects the investigation. Check clocks and test directly against a gateway before concluding a credential is wrong — replacing a correct credential leaves the real cause untouched.
Cross-course references
- Kubernetes: distinguishing authentication from authorisation failures follows the same discipline
- Linux: TLS and Kerberos failures are similarly sensitive to clock skew and hostname mismatches