Working an AccessDenied to its source

Why this matters in production

AccessDenied is emitted by four independent mechanisms, and they interact. Checking them in a fixed order — cheapest and most decisive first — resolves it in minutes instead of by trial and error.

The four mechanisms

The order to check

1. Is the user suspended?

radosgw-admin user info --uid=analytics | jq '.suspended'

One field, and it overrides everything. Check it first.

2. Is there an explicit Deny in the bucket policy?

aws --endpoint-url $EP s3api get-bucket-policy --bucket data | jq -r '.Policy' | jq .

Look for "Effect": "Deny" and check whether its conditions match the failing request. A deny cannot be overridden.

3. Does the policy grant what is being attempted?

Check both the action and the resource ARN. ListBucket needs the bucket ARN; GetObject needs the object ARN.

4. Do the ACLs permit it?

aws --endpoint-url $EP s3api get-bucket-acl --bucket data
aws --endpoint-url $EP s3api get-object-acl --bucket data --key file.txt

Objects carry their own ACL, so an object uploaded by a different user may be inaccessible even where the bucket permits it.

5. Is the credential a subuser with a limiting access level?

radosgw-admin user info --uid=analytics | jq '.subusers'

The cross-user object case

aws --endpoint-url $EP s3api put-object --bucket shared --key f.txt --body f.txt \
    --acl bucket-owner-full-control

When user A uploads to user B’s bucket, the object is owned by A by default and B cannot read it despite owning the bucket. The bucket-owner-full-control ACL at upload time is the fix, and a bucket policy can require it:

{"Effect": "Deny", "Principal": "*", "Action": "s3:PutObject",
 "Resource": "arn:aws:s3:::shared/*",
 "Condition": {"StringNotEquals": {"s3:x-amz-acl": "bucket-owner-full-control"}}}

Verifying a grant

# as the granted credential, attempt the specific operation
AWS_ACCESS_KEY_ID=$K AWS_SECRET_ACCESS_KEY=$S \
  aws --endpoint-url $EP s3api get-object --bucket data --key file.txt /tmp/out

# and confirm the denial still holds where it should
AWS_ACCESS_KEY_ID=$K AWS_SECRET_ACCESS_KEY=$S \
  aws --endpoint-url $EP s3api get-object --bucket other --key file.txt /tmp/out

Quiz

Production discipline

Check the four denial mechanisms in order — suspension, explicit deny, policy grant, ACLs — rather than guessing; the order is by cost and decisiveness. Require bucket-owner-full-control on shared buckets through a policy, since object ownership is the mechanism that most often produces confusing denials.

Cross-course references

• Kubernetes: RBAC plus admission plus PodSecurity all denying independently is the same layered model
• Linux: POSIX permissions plus ACLs plus MAC labels each denying separately follows the same pattern