Skip to main content
RunBook Academy

CephXLVIII · RGW TroubleshootingRGW Troubleshooting

S3 authentication failures and what each means

Intermediate⏱ ~17 minawsradosgw-admincurl

What you'll learn

  • Map each S3 auth error to its cause
  • Diagnose signature failures systematically
  • Handle endpoint and region mismatches
  • Verify credentials end to end

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Not yet marked complete on this device.

Why this matters in production

The four common authentication errors have four unrelated causes, and the one everyone assumes — a wrong secret key — is not the most frequent. The code tells you which investigation to run.

The four errors

ErrorMeansUsual cause
InvalidAccessKeyIdthe access key is unknownwrong key, wrong cluster, or the user was removed
SignatureDoesNotMatchthe key exists, the signature does not verifywrong secret, clock skew, or endpoint mismatch
AccessDeniedauthenticated, not authorisedpolicy, ACL, or suspended user
RequestTimeTooSkewedtimestamp outside the toleranceclock skew, explicitly

SignatureDoesNotMatch is the one that misleads: the secret is often correct and something else about the signed request differs.

Diagnosing SignatureDoesNotMatch

The signature covers the HTTP method, the canonical URI, the query string, selected headers, the payload hash, and a timestamp. Anything that alters one of those between the client and the gateway breaks it.

# 1. clock — check both ends
date -u
ssh rgw-01 date -u
chronyc tracking

# 2. endpoint — the host header is signed
aws --endpoint-url https://rgw.example.com s3 ls        # signed for this host
# a proxy rewriting the Host header breaks the signature

# 3. the credential itself
radosgw-admin user info --uid=analytics | jq '.keys'

# 4. signature version
aws configure set default.s3.signature_version s3v4
CauseCheck
Clock skewcompare client and gateway clocks
Host header rewritten by a proxytest directly against a gateway
Path-style versus virtual-hosted addressingtry --endpoint-url with s3.addressing_style
Wrong secretcompare against user info
Signature version mismatchforce s3v4

The addressing style question

# virtual-hosted: bucket.rgw.example.com — needs wildcard DNS
aws configure set default.s3.addressing_style virtual

# path-style: rgw.example.com/bucket — no DNS requirement
aws configure set default.s3.addressing_style path

RGW supports both, but the host header differs between them and the header is signed. A client using virtual-hosted addressing against a deployment without wildcard DNS fails in a way that presents as a signature error.

ceph config get client.rgw rgw_dns_name

Verifying a credential end to end

export AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=...
aws --endpoint-url https://rgw.example.com s3 ls
aws --endpoint-url https://rgw.example.com s3 ls s3://known-bucket/

A successful s3 ls proves the credential authenticates. A failure at the second step with success at the first is authorisation, not authentication.

Quiz

Knowledge check · 4 questions

  1. Q1. An S3 client receives SignatureDoesNotMatch. Which cause is most common?

  2. Q2. Testing directly against a gateway, bypassing any reverse proxy, is an effective way to diagnose signature failures.

  3. Q3. Diagnose signature failures affecting only some clients.

    Some S3 clients receive SignatureDoesNotMatch while others using the same credentials succeed. The failing clients are all newer SDK versions. The deployment is behind a reverse proxy.

  4. Q4. Why does an AccessDenied error indicate a different problem from SignatureDoesNotMatch?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Read the specific error code before investigating; the four common S3 auth errors have unrelated causes and the code selects the investigation. Check clocks and test directly against a gateway before concluding a credential is wrong — replacing a correct credential leaves the real cause untouched.

Cross-course references

  • Kubernetes: distinguishing authentication from authorisation failures follows the same discipline
  • Linux: TLS and Kerberos failures are similarly sensitive to clock skew and hostname mismatches