AWS Storage

The Complete Amazon S3 403 Access Denied Playbook: Every Cause, Decoded

It is 02:00, an on-call page just fired, and a Fargate task that has read the same bucket for a year is now returning 403 AccessDenied on GetObject. You own the bucket. You wrote the IAM policy. The Policy Simulator says Allow. And S3 still says no. Every S3 engineer meets this wall, and the reflex — widen the identity policy, add s3:*, maybe make the bucket public “just to test” — is almost always wrong, because the layer that denied you is usually a layer you never touched. The real skill is not writing S3 policies; it is locating, in under two minutes, which of nine independent policy sources across four services returned the deny — and then fixing that one, at least privilege, not carpet-bombing the whole stack.

A single S3 request is adjudicated by more moving parts than any other AWS API. Before S3 ever consults its own bucket policy, the request can be killed by a Service Control Policy or Resource Control Policy at the Organizations boundary, by a VPC endpoint policy on the network path, or by Block Public Access. Then the caller’s IAM identity policy must allow it — but so must any permission boundary on that principal and any session policy stapled on at AssumeRole, because your effective permission is the intersection of all three. If the object is encrypted with a customer-managed key, a completely different service — KMS — gets a veto through its key policy, its grants, or the key’s own state. And even with every policy perfect, Object Lock, a legal hold, requester-pays, cross-account object ownership, or a mis-scoped presigned URL can each produce a 403 that looks identical to a permissions problem but isn’t one. Nine sources, four services, one three-digit status code.

This article is the cross-service decision tree that collapses those nine sources back to one. It is the diagnostic companion to Securing Amazon S3: Bucket Policies, Block Public Access and the 403 Playbook, which teaches how to set up the access-control stack; here you learn how to decode a denial under pressure — read the 2023-era detailed AccessDenied message that now names the offending policy type, pull the exact denied API call and principal from CloudTrail data events, run the IAM Policy Simulator while knowing precisely what it cannot see, and walk a request path badge by badge until the guilty layer confesses. The general IAM evaluation mechanics live in Debugging IAM ‘Access Denied’: Policy Evaluation Logic, SCPs, Boundaries and a Playbook; this is that model applied to the one service where it hurts most. Read the prose once; keep the master table open at 02:00.

What problem this solves

The problem is time and blast radius. A misdiagnosed 403 costs an afternoon of widening the wrong policy, and it costs security posture: the industry-standard “fix” of attaching AdministratorAccess or slapping "Principal": "*" on a bucket converts a five-minute layer diagnosis into a permanent audit finding and, in the worst case, a data-leak headline. The denial was always precisely locatable. What was missing was a method — a way to look at the shape of the error and the CloudTrail record and say “that is the KMS key policy” or “that is a session policy from the SSO broker” without touching anything else first.

What breaks without this method is repeatable and expensive. Cross-account data-lake reads half-configured so one door is open and one is shut. SSE-KMS rollouts where the S3 permissions are flawless and the key denies everyone. VPC-only architectures where a bucket works from a laptop and 403s from every EC2 instance. Compliance guardrails — force-TLS, force-SSE, org-perimeter SCPs — that deny correctly but with a message the on-call engineer can’t parse. Object Lock in Compliance mode returning 403 on a delete that no one, not even the account root, can perform. Each of these is a different layer with a different fix, and treating them all as “the IAM policy is too narrow” is how teams end up over-permissioned and still broken.

Who hits this hardest: anyone running cross-account buckets (the AND rule, plus a third door on the KMS key), encrypted workloads (the KMS gate that masquerades as an S3 problem), VPC-only or PrivateLink designs (endpoint policies and aws:SourceVpce conditions), regulated estates with SCPs/RCPs and Object Lock, and anyone who has ever handed a presigned URL to a browser and watched it expire mid-upload. The fix is never “open it wider until it works.” The fix is “read the denial, name the layer, correct that layer.” This article makes that reflex automatic.

Learning objectives

By the end of this article you can:

Prerequisites & where this fits

You should be comfortable creating buckets, uploading and downloading objects, and reading an ARN, a JSON policy and a sts:AssumeRole flow — if not, do the hands-on Amazon S3 Fundamentals Hands-On: Buckets, Objects, Versioning and Presigned URLs first. You should understand the difference between an identity-based policy (attached to a principal) and a resource-based policy (attached to the bucket or the KMS key), and know that a role session is what actually calls S3 after an assume-role. This article assumes the setup knowledge from Securing Amazon S3: Bucket Policies, Block Public Access and the 403 Playbook — how BPA, Object Ownership, bucket policies and condition keys are configured — and turns it around to diagnosis.

Where it fits: this is the S3-specific application of the general authorization model in Debugging IAM ‘Access Denied’: Policy Evaluation Logic, SCPs, Boundaries and a Playbook. That article owns the universal evaluation logic (how Allow/Deny/boundary/SCP/session combine across any service); this one owns the parts that are unique to S3 and its neighbours — the KMS gate, Object Lock, requester-pays, ownership, presigned URLs, the ListBucket/GetObject split, the S3 detailed message and S3 data events. For the audit trail you will lean on throughout, the mechanics of trails, data events and Config live in AWS CloudTrail and Config: Audit and Compliance at Scale. On the certification map, this is core to SCS-C02 (Security Specialty) and SOA-C02 (SysOps), and heavily tested in SAA-C03 and DVA-C02.

Core concepts

Five ideas make every later diagnosis mechanical. Learn them once and the master table stops being a lookup and becomes a reflex.

One request, up to nine deny sources, four services. An S3 API call is not authorized by “the S3 policy.” It is authorized by a pipeline that stitches together policies from Organizations (SCP, RCP), VPC networking (endpoint policy), S3 (Block Public Access, bucket policy, ACL/ownership, Object Lock, requester-pays), IAM (identity policy, permission boundary, session policy) and KMS (key policy, grants, key state). Any one of them can return the deny, and the fix lives in whichever one it was — not necessarily the one you control.

Explicit Deny wins, everywhere, always. The evaluation starts at an implicit deny (nothing is allowed until something allows it). An Allow lifts it. But a single explicit Deny in any policy in scope — SCP, RCP, bucket policy, identity policy, boundary, session, VPC endpoint policy — ends the evaluation with a denial that no number of Allows can overturn. When “adding another Allow changes nothing,” you are looking at an explicit Deny; stop adding Allows and go find it.

Same-account is OR; cross-account is AND; SSE-KMS adds a third door. Inside one account, GetObject is allowed if either the identity policy or the bucket policy allows it. Across accounts, both must allow — independently. And if the object is SSE-KMS with a customer-managed key, a third independent grant is required on the KMS key. The majority of cross-account 403s are simply “one of the two-or-three doors is shut.”

The effective identity permission is an intersection, not a union. People think of an IAM policy as the whole story. It isn’t. The caller’s effective permission is identity policy ∩ permission boundary ∩ session policy (each present layer narrows it further). An action allowed by the identity policy is still denied if the boundary doesn’t also allow it, or if a session policy passed at AssumeRole doesn’t. These two — boundary and session — are invisible in the bucket, invisible in most dashboards, and the cause of the nastiest “but my policy allows it!” 403s.

Read the message first, then confirm with CloudTrail. Since 2023 the S3 AccessDenied message increasingly names the policy type that denied you — “with an explicit deny in a service control policy,” “in a permissions boundary,” “in a resource-based policy,” or “because no identity-based policy allows.” That one clause collapses nine layers to one. CloudTrail data events then give you the ground truth: the exact eventName, the errorCode, the real userIdentity (often not who you assumed), the sourceIPAddress and the vpcEndpointId. Message narrows it; CloudTrail proves it.

Every source of a deny — the master map

Pin the whole field before the deep sections. This is the diagnostic index; every row gets its own section below.

Deny source Lives in (service) Grants or only denies? Signature in the error / behaviour First thing to confirm
Service Control Policy (SCP) Organizations Only caps (Deny) …explicit deny in a service control policy Read the account’s OU SCPs
Resource Control Policy (RCP) Organizations Only caps (Deny) …explicit deny in a resource control policy Read RCPs on the resource’s OU
VPC endpoint policy VPC networking Only caps Fails only from inside the VPC describe-vpc-endpoints + its policy
Block Public Access S3 Only vetoes public A “public” grant still 403s get-public-access-block (acct + bucket)
Bucket policy (explicit Deny) S3 Grants + Deny …explicit deny in a resource-based policy get-bucket-policy; read the conditions
Bucket policy (missing principal) S3 Grants Cross-account 403; no resource-based policy allows get-bucket-policy; is the role named?
IAM identity policy IAM Grants + Deny no identity-based policy allows the … action simulate-principal-policy
Permission boundary IAM Only caps Denied though identity policy allows get-role/get-user; boundary attached?
Session policy IAM/STS Only caps Denied only for the assumed session Inspect the AssumeRole call / broker
KMS key policy / grant KMS Grants (key use) 403 names kms:Decrypt + a key ARN get-key-policy + list-grants
KMS key state KMS Blocks KMSInvalidStateException / disabled describe-keyKeyState
Object Ownership / ACL S3 Grants (legacy) Owner 403s on another account’s object get-object-acl shows a foreign owner
Object Lock / legal hold S3 Blocks write/delete 403 only on overwrite/delete of a version get-object-retention / -legal-hold
Requester Pays S3 Blocks 403 without x-amz-request-payer get-bucket-request-payment
Presigned URL failure S3/SigV4 403 Request has expired / SignatureDoesNotMatch Inspect expiry, method, signer

The evaluation pipeline as a diagnostic order

Work the layers in the order S3 effectively resolves them, because the early ones short-circuit. If an SCP or a bucket-policy Deny is killing the request, nothing you do to the identity policy or the KMS key will ever matter — so rule the short-circuiting layers out first.

Order Layer Question If it denies, the tell is… Grants or only caps
0 Implicit deny Does anything allow this? no identity-based policy allows… Baseline
1 Explicit Deny (any policy) Does any policy Deny this? Adding Allows changes nothing Caps (absolute)
2 SCP / RCP Does the org boundary allow it? …in a service/resource control policy Caps
3 VPC endpoint policy Allowed through the endpoint? Fails only from inside the VPC Caps
4 Block Public Access Is it “public,” and is BPA on? A public grant is inert Caps (never grants)
5 Object Ownership / ACL Does an ACL grant (if ACLs on)? Owner 403s on foreign-written object Grants (legacy)
6 Bucket policy Does the resource policy allow? …in a resource-based policy / not named Grants + Deny
7 IAM identity ∩ boundary ∩ session Does the caller’s effective set allow? no identity-based policy allows… Grants + Deny/caps
8 KMS key policy / grant / state May the principal use the key? 403 names kms:* + a key ARN Grants (key use)
9 Object condition (Lock / requester-pays) Any object-level block? 403 only on write/delete, or needs a header Blocks

The OR / AND / third-door truth table

Most cross-account 403s are one of these seven rows. Memorise it.

Scenario Identity allows? Bucket policy allows? KMS key allows? Explicit Deny anywhere? Result
Same account, unencrypted Yes (any) n/a No Allow (OR)
Same account, unencrypted No Yes n/a No Allow (OR)
Same account, unencrypted No No n/a No Deny (implicit)
Any account (any) (any) (any) Yes Deny (Deny wins)
Cross account, unencrypted Yes No n/a No Deny (one door shut)
Cross account, unencrypted Yes Yes n/a No Allow (both doors)
Cross account, SSE-KMS Yes Yes No No Deny (third door — the key)

The vocabulary in one table

These terms recur; the glossary repeats them for lookup.

Term One-line definition Role in a 403
Implicit deny The default: nothing allows it Base state; message no … policy allows
Explicit Deny A statement with Effect: Deny Beats every Allow; ends evaluation
Permission boundary Managed policy capping a principal’s max Intersects the identity policy — hidden cap
Session policy Policy passed at AssumeRole/federation Intersects further — invisible in the bucket
SCP Org guardrail on member accounts Caps every principal in the account
RCP Org guardrail on resources (S3/KMS/…) Caps access to the resource, org-wide
VPC endpoint policy Policy on the S3 gateway/interface endpoint Caps calls that traverse the endpoint
KMS key policy Resource policy on the encryption key Authoritative for kms:Decrypt/GenerateDataKey
KMS grant Programmatic delegation of key ops An alternate path to key permission
Object Lock WORM retention / legal hold on a version Blocks overwrite/delete, not read
Object Ownership Who owns written objects; ACLs on/off Owner-403 on cross-account writes
Requester Pays Requester pays for request + transfer 403 without the request-payer header
Presigned URL Time-limited signed link, acts as signer Expiry/method/signer failures 403

The IAM identity layer — action, resource, boundary, session

Start where most people start and most people stop too early: the caller’s identity permissions. An identity policy can deny you four different ways, and only the first is the one everyone checks.

Way the identity layer denies What it looks like Confirm Fix
Missing action no identity-based policy allows the s3:GetObject action simulate-principal-policy --action-names s3:GetObject Add the exact action (not a near-neighbour)
Wrong Resource ARN ListBucket works, GetObject 403s (or reverse) Read Resource per statement Bucket ARN for bucket actions, bucket/* for objects
Explicit Deny in the policy explicit deny in an identity-based policy Grep attached policies for Effect:Deny Scope or remove the Deny
A condition evaluated false Denied under some contexts only Inspect the statement’s Condition Meet the condition (MFA, tag, region, …)

The Resource-ARN split is the single most common self-inflicted S3 403, and it deserves its own reference because bucket actions and object actions need different ARNs — mixing them denies you with a perfectly reasonable-looking policy.

Action Level Resource ARN it needs Wrong ARN that 403s
s3:ListBucket Bucket arn:aws:s3:::b arn:aws:s3:::b/*
s3:GetObject Object arn:aws:s3:::b/* arn:aws:s3:::b
s3:PutObject Object arn:aws:s3:::b/* arn:aws:s3:::b
s3:DeleteObject Object arn:aws:s3:::b/* arn:aws:s3:::b
s3:GetObjectVersion Object arn:aws:s3:::b/* arn:aws:s3:::b
s3:ListBucketVersions Bucket arn:aws:s3:::b arn:aws:s3:::b/*
s3:GetBucketPolicy Bucket arn:aws:s3:::b arn:aws:s3:::b/*
s3:GetObjectTagging Object arn:aws:s3:::b/* arn:aws:s3:::b
s3:PutObjectRetention Object arn:aws:s3:::b/* arn:aws:s3:::b
s3:ListAllMyBuckets Account * any bucket ARN
Prefix scope Object arn:aws:s3:::b/logs/* arn:aws:s3:::b/logs

The two invisible caps — boundary and session

Here is where the identity layer gets genuinely hard, and why the Policy Simulator’s green “Allow” lies to you. The identity policy is necessary but not sufficient. The real answer is the intersection of every cap on the principal.

Layer Attached to Effect on the effective permission Where a 403 hides
Identity policy User / role Grants (the union of attached policies) Missing action, wrong ARN, explicit Deny
Permission boundary User / role Caps — final permission is identity ∩ boundary Boundary omits s3:GetObject, so it’s denied
Session policy The role session (passed at AssumeRole) Caps further… ∩ session Broker/SSO scoped the session down
Resource (bucket) policy The bucket Grants (and cross-account is mandatory) Not named; explicit Deny; wrong ARN

A permission boundary is a managed policy that sets the maximum permissions a principal can ever have; the effective set is the intersection of the identity policy and the boundary. If your developer role has a boundary that grants read-only S3 and someone adds s3:PutObject to the identity policy, the PutObject is still denied — the boundary never allowed it. The tell: the identity policy visibly grants the action, yet you’re denied, and get-role shows a PermissionsBoundary.

A session policy is even sneakier because it exists only for the life of one role session. When code calls sts:AssumeRole with --policy (an inline session policy) or --policy-arns, the resulting credentials are capped to role permissions ∩ session policy. SSO brokers, CI systems and “scoped credential” tools routinely pass a session policy that grants far less than the role. Nothing in the bucket, the role’s attached policies, or the Policy Simulator reveals it — you have to inspect the actual AssumeRole call. When a role works from one entry point and 403s from another with identical role ARNs, suspect a session policy.

# Reproduce a session-policy cap: assume a role but scope the session to ListBucket only.
aws sts assume-role \
  --role-arn arn:aws:iam::111122223333:role/app-reader \
  --role-session-name scoped \
  --policy '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":"s3:ListBucket","Resource":"arn:aws:s3:::my-bucket"}]}'
# Now GetObject with these creds 403s — even though role/app-reader allows GetObject —
# because the SESSION POLICY intersection removed it.

The bucket policy layer — explicit Deny and condition-triggered denials

The bucket policy grants cross-account access and, far more often in a 403 investigation, denies through a guardrail. A hardened bucket carries Deny statements for TLS, encryption and data-perimeter, and every one of them will happily deny a legitimate caller who trips the condition. When the message says with an explicit deny in a resource-based policy, this is the layer — and the Condition block tells you exactly what you violated.

Condition-triggered Deny Denies when… Error signature / how it feels Confirm Fix (satisfy the condition)
aws:SecureTransport (Bool false) Request not over TLS Old SDK / http:// endpoint 403s get-bucket-policy; look for the Deny Use https://; upgrade SDK to SigV4
s3:x-amz-server-side-encryption PutObject without the right SSE header PUT 403s “explicit deny” Inspect the PUT headers Send --sse aws:kms or rely on bucket default
s3:x-amz-server-side-encryption-aws-kms-key-id Wrong KMS key on write PUT 403s naming the key condition Compare the key id you sent Send the exact allowed key ARN
aws:SourceIp (NotIpAddress) Caller’s public egress IP not in range Works from office, 403s from cloud Check your public (NAT) IP Add the egress CIDR; prefer VPCe over IP
aws:SourceVpce Not the named VPC endpoint 403s only from inside a VPC Compare the vpce-… id Match the endpoint or add it to the list
aws:PrincipalOrgID Caller not in the org Cross-account partner 403s Compare the caller’s org id Add the account to the org / widen
s3:x-amz-content-sha256 Unsigned payload where signing is required Streaming upload 403s Check for UNSIGNED-PAYLOAD Send a signed payload
s3:DataAccessPointArn Not via the required access point Direct bucket call 403s Check you’re calling the AP ARN Call through the access point
aws:SourceArn / aws:SourceAccount Confused-deputy guard mismatch CloudFront/logging 403s Decode the source service ARN Match the distribution/service ARN

Two distinct bucket-policy failures live here and must not be conflated. The first is an explicit Deny you tripped (the table above) — the message contains explicit deny in a resource-based policy. The second is a missing grant in a cross-account read — the bucket policy simply never named the external principal, so there is no Deny, just no Allow on the resource side; the message reads because no resource-based policy allows the s3:GetObject action. The confirm command is the same (get-bucket-policy), but the fix is opposite: for the first you relax or satisfy a condition, for the second you add the principal (with both the bucket ARN and the bucket/* ARN).

# The condition-triggered Deny confirm: pull the policy and read the Condition blocks.
aws s3api get-bucket-policy --bucket my-bucket --query Policy --output text | jq '.Statement[] | select(.Effect=="Deny")'

Block Public Access, ACLs and Object Ownership

These three S3-native layers cause a specific, recognisable class of 403 — and because setup is covered in depth in the S3 security companion, here the focus is purely the diagnostic signature of each.

Block Public Access (BPA) never grants; it only vetoes grants S3 deems public. The signature is unmistakable: you deliberately made something public — a public-read object, a "Principal":"*" policy — and it still 403s. The trap is that a *-principal statement scoped only by aws:SourceIp is still public to BPA (an IP allow-list is not authentication), so PutBucketPolicy is rejected or the grant is inert.

BPA setting Blocks Signature when it bites
BlockPublicAcls New public ACLs (rejects the call) PutObjectAcl public-read fails
IgnorePublicAcls Existing public ACLs (ignores them) Old public-read object 403s anonymously
BlockPublicPolicy New public bucket policies (rejects) PutBucketPolicy “public policies blocked”
RestrictPublicBuckets Effect of an existing public policy Anonymous/cross-account use of a * policy 403s

Object Ownership / ACLs cause the cross-account ownership 403, one of the three nastiest cases (full prose later). The signature: the bucket owner gets 403 on GetObject for objects that another account wrote, and only those objects. Confirm with get-object-acl — the Owner is a canonical ID that isn’t yours.

Ownership setting ACLs Who owns a written object 403 it causes
BucketOwnerEnforced Disabled Always the bucket owner None (this is the fix)
BucketOwnerPreferred Enabled Owner if writer sends bucket-owner-full-control Owner-403 if writer omits the ACL
ObjectWriter Enabled The writing account Owner-403 on every cross-account write

A companion signature: once ACLs are disabled (BucketOwnerEnforced), any request that still sends an ACL header fails with AccessControlListNotSupported (HTTP 400, not 403) — a different error that means “stop sending --acl,” not “you lack permission.”

The “it’s actually KMS” layer

This is the deny that wastes the most senior-engineer hours, because the 403 comes from S3 but the cause is in a different console entirely. If an object is encrypted with a customer-managed KMS key, reading it needs s3:GetObject and kms:Decrypt; writing it needs s3:PutObject and kms:GenerateDataKey. The S3 permissions can be flawless and the key denies you — and the tell is that the error names a kms:* action and a key ARN, not S3.

S3 operation (SSE-KMS) S3 permission KMS permission 403 signature if KMS is missing
GetObject s3:GetObject kms:Decrypt …is not authorized to perform: kms:Decrypt on … key/…
PutObject s3:PutObject kms:GenerateDataKey names kms:GenerateDataKey + key ARN
CopyObject s3:GetObject + s3:PutObject kms:Decrypt + kms:GenerateDataKey whichever key op is missing
UploadPart (multipart) s3:PutObject kms:GenerateDataKey names kms:GenerateDataKey
Cross-account read s3:GetObject + bucket policy kms:Decrypt in key policy and caller IAM key policy didn’t name the external principal

The key can deny through several distinct mechanisms, and you must check each — a missing key-policy statement is common, but so is a key that delegates to IAM but the caller’s IAM lacks kms:Decrypt, or a grant that should exist and doesn’t, or a kms:ViaService/encryption-context condition that excludes the call, or simply a key state of disabled or pending deletion.

KMS deny mechanism What went wrong Confirm Fix
Key policy omits the principal The key policy never grants the caller kms:Decrypt aws kms get-key-policy --key-id k --policy-name default Add a statement granting the principal the key op
No IAM delegation Key policy delegates to IAM (Enable IAM policies) but caller IAM lacks kms:Decrypt Check both key policy and identity policy Add kms:Decrypt to the caller’s IAM policy
Cross-account: key policy External principal not named on the key get-key-policy on the owner account’s key Add the external account/role to the key policy
Cross-account: caller IAM Key policy allows the account, but caller’s IAM doesn’t allow kms:Decrypt Check the caller-account identity policy Grant kms:Decrypt on the caller side too
Missing grant A service/role relied on a KMS grant that was retired aws kms list-grants --key-id k Recreate the grant (create-grant)
kms:ViaService condition Key restricted to S3 in region X; call came via region Y Read the key-policy Condition Match the s3.<region>.amazonaws.com value
Encryption-context condition Key policy requires an encryption context S3 didn’t send Read kms:EncryptionContext* conditions Relax the condition (S3 sets object ARN as context)
Key state disabled/pending The key is Disabled or PendingDeletion aws kms describe-key --key-id k --query KeyMetadata.KeyState Enable the key / cancel deletion

The KMS error codes are worth memorising because they separate policy from state — a policy problem you fix in the key policy, a state problem you fix by enabling the key.

KMS-related code HTTP Meaning Fix in
AccessDenied naming kms:Decrypt + key ARN 403 Key policy/grant/IAM denies key use Key policy / grant / caller IAM
AccessDenied naming kms:GenerateDataKey 403 Same, on write Key policy / grant
KMSInvalidStateException 400 Key is disabled or pending deletion Enable / cancel deletion
KMS.DisabledException 400 Key explicitly disabled Re-enable the key
KMSInvalidKeyUsageException 400 Wrong key spec/usage for the op Use a symmetric encryption key
KMS.NotFoundException 400 Key id/alias doesn’t exist (or wrong region) Fix the key id / region
AccessDeniedException from KMS on cross-account 403 Key policy didn’t name the external principal Add to the key policy

The perimeter layers — SCP, RCP and VPC endpoint policy

Above the account sit two Organizations guardrails that most bucket owners forget they’re subject to, and on the network path sits a third. All three are caps — they can only deny, never grant — and all three produce a 403 that no amount of bucket-or-IAM editing will fix, because the layer that denied you is not one you were editing.

Perimeter layer Scope Applies to Error signature Where to look
SCP Org OU → member account IAM principals in the account (not the resource) …explicit deny in a service control policy Organizations → the account’s OU SCPs
RCP Org OU → resources Access to S3/KMS/etc. resources, org-wide …explicit deny in a resource control policy Organizations → RCPs on the resource OU
VPC endpoint policy The gateway/interface endpoint Any call that traverses the endpoint Fails only from inside the VPC describe-vpc-endpointsPolicyDocument

The SCP is the classic “widening the IAM policy changes nothing” 403: an OU-level Deny s3:* (perhaps guarded by “unless from our VPCE” or “unless region = ap-south-1” or “unless MFA present”) caps every principal in the account, so the fix is at the OU, not the bucket. The RCP (a 2024 addition) is the resource-side mirror: an org can attach a policy that denies access to its S3 buckets unless the principal is in the org — a partner who used to work suddenly 403s with a resource-control-policy clause. The VPC endpoint policy is the one with the cleanest fingerprint: the identical call succeeds from your laptop and 403s from an EC2/Fargate task, because the task’s traffic goes through a gateway endpoint whose policy caps it — and it often pairs with a bucket-side aws:SourceVpce condition, so you must read both ends of the perimeter.

# Confirm a VPC endpoint policy is the culprit: read the endpoint's own policy.
aws ec2 describe-vpc-endpoints \
  --filters Name=service-name,Values=com.amazonaws.ap-south-1.s3 \
  --query 'VpcEndpoints[].{id:VpcEndpointId,policy:PolicyDocument}' --output json

Object Lock, legal hold and retention — the write/delete-only 403

Every layer so far can deny a read. This one cannot — it only denies overwrites and deletes, which is exactly why it fools people: GetObject works perfectly, so “it can’t be permissions,” yet DeleteObject on a specific version returns 403 that no IAM change fixes. S3 Object Lock enforces WORM (write-once-read-many) through a retention mode with a retain-until date, and/or an independent legal hold.

Protection Retain-until date? Who can shorten / remove it Delete/overwrite of the version Permission/header to override
Governance retention Yes A principal with the bypass permission 403 unless bypassed s3:BypassGovernanceRetention + header x-amz-bypass-governance-retention:true
Compliance retention Yes No one — not even the account root 403 until the date passes None — you wait it out
Legal hold No (indefinite) A principal with s3:PutObjectLegalHold 403 while hold is ON Turn the hold OFF first

The diagnostic signatures, and why each is distinct from a permissions problem:

Symptom It’s… Confirm Fix
DeleteObject/overwrite 403s, GetObject fine, Governance mode Governance retention aws s3api get-object-retention --bucket b --key k --version-id v Add s3:BypassGovernanceRetention + send the bypass header
Same, but even root 403s and bypass header does nothing Compliance retention get-object-retention shows Mode=COMPLIANCE Wait until RetainUntilDate; nothing overrides it
Delete 403s though retention expired/absent Legal hold ON aws s3api get-object-legal-hold --bucket b --key k --version-id v put-object-legal-hold --legal-hold Status=OFF (needs the permission)
Lifecycle can’t expire locked objects Any lock Lifecycle rule silently skips protected versions Expected — locks outrank lifecycle
# The tell: a delete that 403s where GetObject succeeds — check for a lock.
aws s3api get-object-retention  --bucket my-wormbucket --key ledger/2026.json --version-id abc123
aws s3api get-object-legal-hold --bucket my-wormbucket --key ledger/2026.json --version-id abc123
# Governance-mode delete WITH bypass (needs s3:BypassGovernanceRetention):
aws s3api delete-object --bucket my-wormbucket --key ledger/2026.json --version-id abc123 \
  --bypass-governance-retention

Requester Pays, presigned URLs and the ListBucket-vs-GetObject trap

Three more S3-specific 403 classes that have nothing to do with your IAM policy.

Requester Pays. When a bucket owner enables Requester Pays, the requester must be authenticated and must acknowledge the charge with x-amz-request-payer: requester on every call; anonymous requests and requests missing the header get 403.

Requester-pays fact Implication
Header required Missing x-amz-request-payer: requester → 403
Anonymous blocked Only authenticated principals may request
Requester needs s3:GetObject The normal grant still applies on top
Confirm aws s3api get-bucket-request-payment --bucket b shows Payer=Requester
aws s3api get-object --bucket open-data-requester-pays --key big.parquet out.parquet \
  --request-payer requester        # omit --request-payer and it 403s

Presigned URLs. A presigned URL performs one operation as the signer, and it fails in ways that look like permissions but aren’t. Crucially, a presigned URL created by a role session is only valid while that session’s temporary credentials are — so a URL “signed for 7 days” dies when the role session expires, often within the hour.

Presigned-URL failure Error Root cause Fix
Link used after expiry 403 Request has expired X-Amz-Expires window elapsed Re-sign; use short, honest TTLs
Signed for GET, used with PUT 403 SignatureDoesNotMatch HTTP method is part of the signature Re-sign for the exact method
Extra/missing headers 403 SignatureDoesNotMatch Signed headers must match sent headers Sign exactly the headers you send
Worked, now 403s 403 AccessDenied Signer’s role session expired or permission revoked Re-sign with a live, permitted signer
Wrong region 403 SignatureDoesNotMatch / AuthorizationHeaderMalformed Region is in the SigV4 scope Sign with the bucket’s region
SSE-KMS object 403 naming kms:Decrypt Signer lacks kms:Decrypt on the key Grant the signer the key op

The s3:ListBucket vs s3:GetObject trap. These are different actions on different resources, and the confusion runs deeper than the ARN split — it changes what a missing object returns. S3 deliberately hides object existence from principals who cannot list the bucket: if you have s3:ListBucket, a GetObject on a non-existent key returns 404 NoSuchKey; if you do not have s3:ListBucket, that same missing key returns 403 AccessDenied. So a “permissions” 403 can actually be a typo in the key name — the object never existed, and S3 masked the 404 as a 403 to avoid leaking that fact.

You have… Key exists? Result What it really means
s3:GetObject + s3:ListBucket Yes 200 Normal read
s3:GetObject + s3:ListBucket No 404 NoSuchKey The key genuinely doesn’t exist
s3:GetObject only (no ListBucket) No 403 AccessDenied Masked 404 — the key may just be missing
s3:ListBucket only (no GetObject) Yes 403 on Get; ListBucket works Can enumerate, can’t read
Neither (any) 403 on both True permissions deny

The operational rule: when you get a 403 on GetObject and you lack s3:ListBucket, do not assume it’s a policy problem — first prove the key exists (grant s3:ListBucket temporarily, or head-object from an admin principal). Half of these “denials” are a wrong prefix or a missing trailing character in the key.

Reading the denial — detailed message, CloudTrail data events, Policy Simulator

You now know every layer. The last skill is extracting, from the failure itself, which layer — and that comes from three instruments.

The 2023+ detailed AccessDenied message

Modern S3 AccessDenied messages name the policy type that denied you. Map the clause straight to the layer.

Reason clause in the message Layer that denied Fix
no identity-based policy allows the … action IAM identity (implicit) — or a boundary/session that removed it Add the action + right ARN; check boundary/session
with an explicit deny in an identity-based policy IAM identity Deny Find and scope the Deny
with an explicit deny in a resource-based policy Bucket policy Deny (condition) Satisfy the condition (TLS/SSE/VPCe)
because no resource-based policy allows the … action Bucket policy (cross-account, missing principal) Add the principal to the bucket policy
with an explicit deny in a service control policy SCP Adjust the OU’s SCP
with an explicit deny in a resource control policy RCP Adjust the OU’s RCP
with an explicit deny in a VPC endpoint policy VPC endpoint policy Add the action/resource to the endpoint policy
with an explicit deny in a permissions boundary Permission boundary Widen the boundary
with an explicit deny in a session policy Session policy (assume-role) Widen the session policy at AssumeRole
names kms:Decrypt/kms:GenerateDataKey + a key ARN KMS key policy/grant Grant the key op on the key

The one nuance to internalise: the message names the policy type only when the deny is explicit. An implicit deny from a boundary or session policy (it simply doesn’t grant the action) still reads no identity-based policy allows — which is why you must actively check for a boundary and a session policy whenever the identity policy looks sufficient.

CloudTrail data events — the ground truth

S3 object-level calls (GetObject, PutObject, DeleteObject, list-objects) are data events, and — unlike management events — they are not logged by default. You must enable them on a trail or CloudTrail Lake event data store with an advanced event selector for AWS::S3::Object. Once on, a denied call is captured with everything you need to place the deny.

CloudTrail field What it tells you in a 403 investigation
eventName The exact API (GetObject, PutObject, DeleteObjects, …)
errorCode AccessDenied, NoSuchKey, KMS.DisabledException, …
errorMessage The full detailed reason clause (the table above)
userIdentity.type / .arn Who actually called — often not the role you assumed
userIdentity.sessionContext The role, session name, and whether MFA was present
requestParameters.bucketName / .key The exact bucket and object key (catches typos)
resources The bucket/object/KMS-key ARNs in play
sourceIPAddress The caller’s IP — public egress vs a VPC address
vpcEndpointId The vpce-… if it traversed an endpoint (VPCe denies)
additionalEventData SSE algorithm, x-amz-id-2, cipher — flags SSE-KMS calls
readOnly Read vs mutate — narrows Object Lock cases

The critical operational gotcha: aws cloudtrail lookup-events only searches management events, so it will never return your denied GetObject. To find a data-event 403 you query the delivered logs (Athena over the CloudTrail S3 bucket) or CloudTrail Lake (SQL over an event data store).

Instrument Covers data events? How you query Latency
cloudtrail lookup-events No (management only) CLI, last 90 days Minutes
CloudTrail → S3 log files + Athena Yes (if enabled) SQL over partitioned logs ~5–15 min delivery
CloudTrail Lake Yes (if the EDS captures them) SQL, aws cloudtrail start-query ~Minutes
S3 server access logs Yes (separate mechanism) Parse text logs / Athena Best-effort, delayed
-- CloudTrail Lake: find who was denied on which key in the last day.
SELECT eventtime, useridentity.arn, errorcode, errormessage,
       element_at(requestparameters, 'bucketName') AS bucket,
       element_at(requestparameters, 'key')        AS key,
       sourceipaddress, vpcendpointid
FROM   <event_data_store_id>
WHERE  eventsource = 's3.amazonaws.com'
  AND  errorcode  = 'AccessDenied'
  AND  eventtime > timestamp '2026-07-13 00:00:00'
ORDER  BY eventtime DESC;

The IAM Policy Simulator — and its blind spots

aws iam simulate-principal-policy evaluates a principal’s identity policies (and can factor in permission boundaries and SCPs) against an action and resource. It is excellent for the identity layer — and dangerous if you trust its “allowed” past that layer, because it is blind to most of the S3 stack. This is exactly why “the simulator says Allow” is a trap, not an all-clear.

Layer Does the Policy Simulator evaluate it? Consequence
Identity policy Yes Reliable for missing action / wrong ARN
Permission boundary Yes (if you include it) Good, if you remember to
SCP Yes (simulate-principal-policy with org data) Good for org denies
Session policy No A scoped session still 403s — simulator says Allow
Resource (bucket) policy Partial (--resource-policy), unreliable cross-account Cross-account AND logic not modelled well
KMS key policy No SSE-KMS 403 invisible to the simulator
Block Public Access No A BPA-vetoed public grant looks allowed
VPC endpoint policy No Endpoint denies invisible
Object Lock / requester-pays No Write/delete blocks invisible
# Use it for what it's GOOD at: the identity layer, with the boundary included.
aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::111122223333:role/app-reader \
  --action-names s3:GetObject s3:ListBucket \
  --resource-arns arn:aws:s3:::my-bucket/data/report.csv arn:aws:s3:::my-bucket \
  --query 'EvaluationResults[].{action:EvalActionName,decision:EvalDecision}' --output table
# A green "allowed" here means the IDENTITY layer is fine — nothing more.

Architecture at a glance

The diagram traces one S3 request left to right through every layer that can return a 403, with a numbered badge on each deny origin and a legend that narrates symptom · confirm · fix. On the left, the caller — usually an assumed-role session, not a user — makes a SigV4-signed request. It first meets the org and network perimeter: a Service Control Policy or Resource Control Policy (badge 1) can cap every principal in the account, and a VPC endpoint policy (badge 2) caps anything traversing the endpoint — the deny that bites only from inside the VPC. The request then reaches the bucket gate: Block Public Access (badge 3) vetoes any public grant, and the bucket policy (badge 4) can explicit-Deny on a TLS/SSE/IP/VPCe condition or simply fail to name a cross-account principal. Next, identity: the caller’s IAM identity policy must allow the exact action on the exact bucket-vs-bucket/* ARN (badge 5), narrowed by any permission boundary and session policy — the intersection that the Policy Simulator can’t fully see. Finally the key and object layer: for an SSE-KMS object the KMS key must grant kms:Decrypt (badge 6), and even then Object Lock, a legal hold, or cross-account object ownership can 403 a write or delete. Only a request that collected an Allow (identity OR bucket policy same-account; both, plus the key, cross-account) and passed every perimeter with no explicit Deny reaches the object; anything else exits as 403 AccessDenied, which you decode back to the badge that stopped it.

Amazon S3 request path left to right through every access-control layer that can produce a 403: a caller as an assumed-role session, the org and network perimeter of SCP/RCP and VPC endpoint policy, the bucket gate of Block Public Access and the bucket policy, the identity layer of IAM policy plus permission boundary and session policy, and the key-and-object layer of the KMS key plus Object Lock and ownership, with six numbered badges marking each AccessDenied origin and a legend narrating symptom, confirm and fix per layer

Real-world scenario

Aureus Analytics runs a 30-account AWS Organization with a shared data-lake bucket, aureus-lake-prod-111122223333, in the data-prod account, encrypted with a customer-managed KMS key and fronted by a compliance guardrail: an org SCP denies s3:* unless the caller is in the org, an RCP denies access to lake buckets from outside the org, and every object over a year old is under Object Lock in Governance mode. Three teams read the lake. Over one launch week, four different 403s land on the same on-call engineer, and the value of the decision tree is that each is a different layer — treating them all as “the IAM policy is wrong” would have failed four times.

Monday — the analytics reader 403s cross-account. The lake-reader role in account 444455556666 has a clean identity policy allowing s3:GetObject on the lake, and simulate-principal-policy returns Allow. That green result is the trap. CloudTrail Lake shows the denied GetObject with errorMessage = …with an explicit deny in a resource-based policy and a populated vpcEndpointId. The lake bucket policy carried a data-perimeter Deny requiring aws:SourceVpce to match data-prod’s endpoint — but the reader came through the analytics account’s endpoint. Fix: rewrite the perimeter Deny to accept either endpoint via a multi-value aws:SourceVpce list. Ten minutes, because they read the message instead of the simulator.

Tuesday — a 403 that names a key. The same lake-reader, unchanged, now 403s — and the CloudTrail errorMessage references kms:Decrypt on a key ARN, not S3. A platform rotation had swapped in a new customer-managed key whose key policy granted kms:Decrypt only to a named admin role, and did not restore Enable IAM policies delegation. The S3 permission never changed; the key started denying. Fix at the key, not the data service: add lake-reader (and the cross-account analytics principal) to the key policy with kms:Decrypt, and turn on an S3 Bucket Key to blunt the now-higher KMS bill. Runbook line: an S3 403 that mentions a key ARN is a KMS problem.

Wednesday — a delete that even admins can’t do. A data-engineering cleanup job 403s on DeleteObject for last year’s partitions, though the role has s3:DeleteObject and the Policy Simulator is green. GetObject on the same keys works. get-object-retention shows Mode=GOVERNANCE with a future RetainUntilDate. The job lacked s3:BypassGovernanceRetention and wasn’t sending the bypass header. Fix: grant the narrow bypass permission to the cleanup role and add --bypass-governance-retentionnot widening S3 permissions, which would have done nothing.

Thursday — the “missing” object that was a typo. A new microservice 403s on GetObject for curated/2026/06/summary.json. Its role has s3:GetObject but deliberately not s3:ListBucket (least privilege). The team spent an hour widening the bucket policy before someone head-object’d the key from an admin and got 404 — the service was requesting curated/2026/6/summary.json (no zero-padding); the object never existed, and because the role lacked s3:ListBucket, S3 masked the 404 as a 403. Fix: correct the key in the caller, and add a lint check. The engineer’s note: a GetObject 403 without s3:ListBucket might just be a wrong key. Four pages, four layers, one method.

Advantages and disadvantages

The multi-layer, deny-by-default model is what makes S3 safe to run at scale across thousands of accounts — and what makes a 403 a skill rather than a lookup. The trade-off is explicit.

Advantages of the layered model Disadvantages / costs
Defence in depth — one misconfigured layer can’t leak data More layers = more places a request silently dies
Central teams can guarantee limits (SCP/RCP/BPA) no owner can exceed The denying layer is often not the one you control
The detailed message now names the policy type Only for explicit denies; implicit ones stay vague
CloudTrail data events give exact ground truth Off by default; cost money; not in lookup-events
Explicit Deny gives hard, unbypassable guardrails “Adding an Allow” wastes hours against a Deny
KMS adds an auditable second authorization An SSE-KMS 403 masquerades as an S3 problem
Object Lock enforces true WORM 403s a delete that no permission can perform

The layering pays off most in regulated, multi-account estates, where an SCP/RCP guarantee and Object Lock compliance are worth the diagnostic overhead. In a single-account side project, most layers are dormant — no SCP, no cross-account AND, no VPCe, no KMS — and a 403 collapses to “identity policy or bucket policy,” a two-minute check. The cost is entirely in the middle: teams that grew into multi-account and cross-account patterns without learning the decision tree, who fight each new 403 from scratch.

Hands-on lab

You will reproduce three distinct 403s, diagnose each with CloudTrail data events and the IAM Policy Simulator, fix each at the correct layer, then tear everything down. Free-tier-safe except a few cents of S3/KMS/CloudTrail and the ~$1/month KMS key (deleted at the end). Use a sandbox account. Replace 111122223333 with your account id; the cross-account step needs a second account 444455556666 you control (or read it as a walkthrough).

Step 0 — Confirm who you are, and turn on S3 data events. Half of all “impossible” 403s are “you’re not the principal you think,” and you can’t investigate data-event denials you never logged.

aws sts get-caller-identity
# Expected: { "Account": "111122223333", "Arn": "arn:aws:iam::111122223333:user/you" }

REGION=ap-south-1
BUCKET=kv-403-lab-$RANDOM
TRAILBUCKET=kv-403-trail-$RANDOM
aws s3api create-bucket --bucket $BUCKET      --region $REGION --create-bucket-configuration LocationConstraint=$REGION
aws s3api create-bucket --bucket $TRAILBUCKET --region $REGION --create-bucket-configuration LocationConstraint=$REGION

Create a trail with an advanced event selector for S3 object data events (management events alone will not capture GetObject denials). Give CloudTrail permission to write to the trail bucket first (console does this automatically; via CLI attach the standard CloudTrail bucket policy), then:

aws cloudtrail create-trail --name kv-403-trail --s3-bucket-name $TRAILBUCKET
aws cloudtrail put-event-selectors --trail-name kv-403-trail \
  --advanced-event-selectors '[{
     "Name":"S3 object data events",
     "FieldSelectors":[
       {"Field":"eventCategory","Equals":["Data"]},
       {"Field":"resources.type","Equals":["AWS::S3::Object"]}]}]'
aws cloudtrail start-logging --name kv-403-trail

⚠️ Data events bill at ~$0.10 per 100,000 events. This lab generates a handful; the cost is fractions of a cent. Stop logging at teardown.

Step 1 — Reproduce 403 #1: a bucket-policy VPC-endpoint condition. Attach a bucket policy that denies everything except calls from a VPC endpoint that your CLI (running over the internet) is not using — so you get denied, exactly as a mis-scoped perimeter would deny a legitimate caller.

cat > /tmp/vpce-deny.json <<JSON
{ "Version":"2012-10-17","Statement":[
  { "Sid":"OnlyFromOurVpce","Effect":"Deny","Principal":"*","Action":"s3:*",
    "Resource":["arn:aws:s3:::$BUCKET","arn:aws:s3:::$BUCKET/*"],
    "Condition":{"StringNotEquals":{"aws:SourceVpce":"vpce-0nonexistent000"}} } ]}
JSON
aws s3api put-bucket-policy --bucket $BUCKET --policy file:///tmp/vpce-deny.json

echo "hello" > /tmp/o.txt
aws s3api put-object --bucket $BUCKET --key o.txt --body /tmp/o.txt
# Expected: An error occurred (AccessDenied) ... with an explicit deny in a resource-based policy

Diagnose #1. The message already names the layer (resource-based policy). Prove it with CloudTrail after a minute, and confirm the Policy Simulator (identity layer) is green — the classic mismatch that tells you the deny is elsewhere.

aws iam simulate-principal-policy \
  --policy-source-arn "$(aws sts get-caller-identity --query Arn --output text)" \
  --action-names s3:PutObject --resource-arns arn:aws:s3:::$BUCKET/o.txt \
  --query 'EvaluationResults[].EvalDecision' --output text
# Expected: allowed   <-- identity layer is fine; the deny is the bucket policy condition

aws s3api get-bucket-policy --bucket $BUCKET --query Policy --output text | jq '.Statement[]|select(.Effect=="Deny").Condition'
# Expected: the aws:SourceVpce StringNotEquals condition you tripped

Fix #1. Satisfy the condition or scope the Deny. Here, remove the perimeter Deny (in production you’d correct the vpce-… id or add yours to a multi-value list).

aws s3api delete-bucket-policy --bucket $BUCKET
aws s3api put-object --bucket $BUCKET --key o.txt --body /tmp/o.txt   # Expected: succeeds

Step 2 — Reproduce 403 #2: SSE-KMS missing kms:Decrypt. Encrypt an object with a customer-managed key whose policy grants you Encrypt/GenerateDataKey but not Decrypt, then try to read it — the S3 permission is perfect, the key denies.

KEYID=$(aws kms create-key --description kv-403-lab \
  --query KeyMetadata.KeyId --output text)
ME=$(aws sts get-caller-identity --query Arn --output text)
ACCT=$(aws sts get-caller-identity --query Account --output text)

# Key policy: allow admin + GenerateDataKey/Encrypt to you, but NOT Decrypt.
cat > /tmp/key-policy.json <<JSON
{ "Version":"2012-10-17","Statement":[
  { "Sid":"Admin","Effect":"Allow","Principal":{"AWS":"arn:aws:iam::$ACCT:root"},"Action":"kms:*","Resource":"*"},
  { "Sid":"NoDecryptForCaller","Effect":"Deny","Principal":{"AWS":"$ME"},"Action":"kms:Decrypt","Resource":"*"} ]}
JSON
aws kms put-key-policy --key-id $KEYID --policy-name default --policy file:///tmp/key-policy.json

aws s3api put-object --bucket $BUCKET --key enc.txt --body /tmp/o.txt \
  --server-side-encryption aws:kms --ssekms-key-id $KEYID     # Expected: succeeds (GenerateDataKey allowed)
aws s3api get-object --bucket $BUCKET --key enc.txt /tmp/back.txt
# Expected: An error occurred (AccessDenied) ... is not authorized to perform: kms:Decrypt on ... key/...

Diagnose #2. The error names kms:Decrypt and a key ARN — that alone places the layer. Note the Policy Simulator is useless here (it can’t see KMS), so go straight to the key policy and list-grants.

aws kms get-key-policy --key-id $KEYID --policy-name default --query Policy --output text | jq '.Statement[]|select(.Effect=="Deny")'
aws kms list-grants --key-id $KEYID --query 'Grants[].{grantee:GranteePrincipal,ops:Operations}' --output json
# Expected: the explicit Deny on kms:Decrypt for your ARN; no compensating grant

Fix #2. Grant kms:Decrypt on the key (not S3). Remove the Deny by restoring an allow-all-to-account key policy.

cat > /tmp/key-policy2.json <<JSON
{ "Version":"2012-10-17","Statement":[
  { "Sid":"Admin","Effect":"Allow","Principal":{"AWS":"arn:aws:iam::$ACCT:root"},"Action":"kms:*","Resource":"*"} ]}
JSON
aws kms put-key-policy --key-id $KEYID --policy-name default --policy file:///tmp/key-policy2.json
aws s3api get-object --bucket $BUCKET --key enc.txt /tmp/back.txt   # Expected: succeeds

Step 3 — Reproduce 403 #3: cross-account object ownership. With ACLs enabled (ObjectWriter), an object written by another account is owned by that account, and you — the bucket owner — 403 on it. If you have a second account, have its role PutObject here; otherwise read the diagnosis. First flip ownership to the legacy mode that allows the trap:

aws s3api put-bucket-ownership-controls --bucket $BUCKET \
  --ownership-controls 'Rules=[{ObjectOwnership=ObjectWriter}]'
# (From account 444455556666, its role writes without granting bucket-owner-full-control:)
#   aws s3api put-object --bucket kv-403-lab-XXXX --key foreign.txt --body f.txt
# Now as the BUCKET OWNER:
aws s3api get-object --bucket $BUCKET --key foreign.txt /tmp/f.txt
# Expected (for the foreign-written object): An error occurred (AccessDenied)

Diagnose #3. The tell is that the owner 403s on specific objects — the ones another account wrote. get-object-acl shows a foreign canonical owner.

aws s3api get-object-acl --bucket $BUCKET --key foreign.txt \
  --query 'Owner' --output json
# Expected: an Owner ID / DisplayName that is NOT your account — that's the whole bug

Fix #3. Disable ACLs going forward (BucketOwnerEnforced) so the bucket owner owns every future object, and re-own existing foreign objects with an in-place copy.

aws s3api put-bucket-ownership-controls --bucket $BUCKET \
  --ownership-controls 'Rules=[{ObjectOwnership=BucketOwnerEnforced}]'
# Re-own an existing foreign object by copying it in place (now you own the copy):
aws s3api copy-object --bucket $BUCKET --key foreign.txt \
  --copy-source $BUCKET/foreign.txt --metadata-directive COPY
aws s3api get-object --bucket $BUCKET --key foreign.txt /tmp/f.txt   # Expected: succeeds

Step 4 — Terraform equivalent (reproducible guardrails). The same three layers, as code, so a team can stand up (and destroy) the scenario deterministically.

resource "aws_s3_bucket" "lab" { bucket = "kv-403-lab-tf-111122223333" }

# Layer: Object Ownership (BucketOwnerEnforced disables the cross-account trap)
resource "aws_s3_bucket_ownership_controls" "lab" {
  bucket = aws_s3_bucket.lab.id
  rule { object_ownership = "BucketOwnerEnforced" }
}

# Layer: bucket-policy VPC-endpoint perimeter Deny (repro #1)
resource "aws_s3_bucket_policy" "lab" {
  bucket = aws_s3_bucket.lab.id
  policy = jsonencode({ Version = "2012-10-17", Statement = [{
    Sid = "OnlyFromOurVpce", Effect = "Deny", Principal = "*", Action = "s3:*",
    Resource  = [aws_s3_bucket.lab.arn, "${aws_s3_bucket.lab.arn}/*"],
    Condition = { StringNotEquals = { "aws:SourceVpce" = "vpce-0abc123def456" } }
  }]})
}

# Layer: SSE-KMS default (repro #2 lives on the KEY policy, managed separately)
resource "aws_kms_key" "lab" { description = "kv-403-lab"; enable_key_rotation = true }
resource "aws_s3_bucket_server_side_encryption_configuration" "lab" {
  bucket = aws_s3_bucket.lab.id
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm     = "aws:kms"
      kms_master_key_id = aws_kms_key.lab.arn
    }
    bucket_key_enabled = true
  }
}

# Layer: CloudTrail S3 data events (so denials are actually logged)
resource "aws_cloudtrail" "lab" {
  name                          = "kv-403-trail"
  s3_bucket_name                = aws_s3_bucket.trail.id
  include_global_service_events = false
  advanced_event_selector {
    name = "S3 object data events"
    field_selector { field = "eventCategory"; equals = ["Data"] }
    field_selector { field = "resources.type"; equals = ["AWS::S3::Object"] }
  }
}

Step 5 — Teardown (avoid lingering charges).

aws cloudtrail stop-logging --name kv-403-trail
aws cloudtrail delete-trail --name kv-403-trail
aws s3 rm s3://$BUCKET      --recursive
aws s3 rm s3://$TRAILBUCKET --recursive
aws s3api delete-bucket --bucket $BUCKET
aws s3api delete-bucket --bucket $TRAILBUCKET
aws kms schedule-key-deletion --key-id "$KEYID" --pending-window-in-days 7   # ⚠️ CMK: 7–30 day window

⚠️ A customer-managed KMS key bills ~$1/month while it exists and can only be scheduled for deletion. Schedule it now so it stops billing.

Common mistakes & troubleshooting

This is the playbook — the reason to keep this article open at 02:00. Work it top-down: rule out the short-circuiting layers (explicit Deny, SCP, BPA) first, and always read the errorMessage before touching anything — it names the layer. Start with the 30-second triage, then the master decision tree, then the two reference tables.

30-second triage — the pattern maps to the layer

If you see… It’s probably… Do this first
Broadening the IAM policy changes nothing An explicit Deny (bucket / SCP / RCP) Read the reason clause; stop adding Allows
The same call works from your laptop, 403s from EC2/Fargate VPC endpoint policy or aws:SourceVpce Read the endpoint policy + bucket condition
A deliberately public read is blocked Block Public Access get-public-access-block (account + bucket)
403 names a KMS key ARN The KMS gate Fix the key policy/grant, not S3
simulate-principal-policy says Allow but you’re still denied A layer the simulator can’t see (session/KMS/BPA/VPCe/resource) Go to CloudTrail; read the message
Owner 403s on objects another account wrote Object Ownership (ACLs on) get-object-acl → foreign owner
GetObject 403s and you lack s3:ListBucket Possibly a missing key masked as 403 head-object from an admin; check the key
403 only on delete/overwrite, read is fine Object Lock / legal hold get-object-retention / -legal-hold
403 on a bucket you don’t own, needs a header Requester Pays Add --request-payer requester
SignatureDoesNotMatch / RequestTimeTooSkewed Not IAM — clock / region / creds Fix NTP / region / credentials

The master decision tree

# Exact error / symptom Layer Root cause Confirm (CloudTrail / IAM policy sim / console) Fix
1 no identity-based policy allows the s3:GetObject action IAM identity (implicit) Missing action or wrong ARN simulate-principal-policy --action-names s3:GetObject Add the exact action + correct ARN
2 ListBucket works, GetObject 403s (or reverse) IAM / bucket ARN Bucket-vs-bucket/* ARN mixup Read Resource per statement Bucket ARN for bucket actions, bucket/* for objects
3 Identity policy visibly allows it, still 403 Permission boundary Boundary omits the action (intersection) get-role/get-userPermissionsBoundary Add the action to the boundary
4 Works from one entry point, 403s from another (same role) Session policy AssumeRole passed a scoped session policy Inspect the AssumeRole call / SSO broker Widen the session policy
5 with an explicit deny in an identity-based policy IAM identity Deny A Deny statement in an attached policy Grep attached policies for Effect:Deny Scope or remove the Deny
6 with an explicit deny in a resource-based policy Bucket policy Deny Condition tripped (TLS/SSE/IP/VPCe) get-bucket-policy; read Condition Satisfy the condition (HTTPS/SSE/right VPCe)
7 Cross-account 403, no resource-based policy allows… Bucket policy (missing principal) External role not named on the resource get-bucket-policy; is the role ARN there? Add the principal (bucket ARN + bucket/*)
8 with an explicit deny in a service control policy SCP Org guardrail caps the account Organizations → the account’s OU SCPs Adjust the SCP at the OU
9 with an explicit deny in a resource control policy RCP Org resource perimeter denies external access Organizations → RCPs on the resource OU Adjust the RCP
10 Same call 403s only from inside a VPC VPC endpoint policy Endpoint policy caps the call describe-vpc-endpointsPolicyDocument Add the action/ARN to the endpoint policy
11 A “public” read still returns 403 Block Public Access BPA vetoes the public grant get-public-access-block (account + bucket) Front with CloudFront/OAC or presign — don’t disable BPA
12 PutBucketPolicy rejected “public policies are blocked” BlockPublicPolicy Principal:"*" scoped only by aws:SourceIp Check the policy for * + SourceIp De-publicise (VPCe / OrgID / named principal)
13 Owner 403s on objects another account wrote Object Ownership ObjectWriter — the writer owns them get-object-acl shows a foreign owner BucketOwnerEnforced; re-copy to re-own
14 AccessControlListNotSupported (400) ACLs disabled Call sends an ACL header The request has --acl/x-amz-acl Remove the ACL header
15 403 names kms:Decrypt + a key ARN KMS key policy Key doesn’t grant the caller get-key-policy + list-grants Grant kms:Decrypt on the key
16 PutObject 403 names kms:GenerateDataKey KMS Missing GenerateDataKey on the key get-key-policy Add kms:GenerateDataKey to the caller on the key
17 Cross-account SSE-KMS read 403 KMS (cross-account) Key policy missing external principal, or caller IAM missing kms:Decrypt Read key policy AND caller IAM Grant on the key policy AND the caller IAM (both doors)
18 KMSInvalidStateException / KMS.DisabledException KMS key state Key disabled or pending deletion describe-key --query KeyMetadata.KeyState Enable the key / cancel deletion
19 DeleteObject/overwrite 403s, read fine Object Lock (Governance) Retention with a retain-until date get-object-retentionMode=GOVERNANCE Grant s3:BypassGovernanceRetention + send the bypass header
20 Same, but even root 403s, bypass no help Object Lock (Compliance) Compliance retention — nothing overrides get-object-retentionMode=COMPLIANCE Wait until RetainUntilDate
21 Delete 403s though retention expired Legal hold Legal hold is ON get-object-legal-holdStatus=ON put-object-legal-hold Status=OFF
22 403 on someone else’s bucket needing a header Requester Pays Missing x-amz-request-payer get-bucket-request-paymentPayer=Requester Add --request-payer requester
23 Presigned URL 403 Request has expired Presigned (expiry) X-Amz-Expires elapsed or signer session ended Inspect the URL’s expiry / signer TTL Re-sign with a live, permitted signer
24 Presigned URL 403 SignatureDoesNotMatch Presigned (method/header) Wrong HTTP method or header mismatch Compare method/headers to what was signed Re-sign for the exact method + headers
25 GetObject 403, no s3:ListBucket, key uncertain Masked 404 Object doesn’t exist; S3 hides it as 403 head-object from an admin; add ListBucket Fix the key; the “deny” was a typo
26 SignatureDoesNotMatch / RequestTimeTooSkewed / PermanentRedirect Not IAM Clock skew, wrong region, wrong key aws --debug; check clock/region Fix NTP / use the bucket’s region / verify creds
27 AllAccessDisabled (403) Account-level Billing/legal hold on the account Check account/billing status Resolve with AWS Support/billing

The condition-key deny reference

When the action and ARN are right but a condition evaluated false, the layer is a policy Condition. This is the quiet-denier lookup.

Condition key Denies when… Confirm Common trap
aws:SecureTransport Request not over TLS Retry with https:// Old SDK / HTTP endpoint
s3:x-amz-server-side-encryption PutObject without the SSE header Inspect PUT headers Rely on bucket default or send --sse aws:kms
s3:x-amz-server-side-encryption-aws-kms-key-id Wrong KMS key on write Compare the key id sent Sent the default key, policy wants a specific one
aws:SourceVpce Not the named VPC endpoint Compare the vpce-… id Second endpoint; use …IfExists for non-VPC calls
aws:SourceVpc Wrong VPC Compare the vpc-… id Peered VPC not in the list
aws:SourceIp Public egress IP not in range Check the public (NAT) IP It’s the NAT/egress IP, not the private VPC IP
aws:PrincipalOrgID Caller not in the org Compare the org id Hard-coded old org id after a migration
aws:PrincipalTag / aws:RequestTag ABAC tag mismatch Compare principal/request tags Missing/typo’d tag on the role or request
s3:x-amz-acl Required canned ACL absent Check the write’s ACL Only relevant while ACLs enabled
aws:SourceArn / aws:SourceAccount Confused-deputy source mismatch Decode the source service ARN CloudFront OAC / logging service principal
s3:DataAccessPointArn Not via the required access point Check you called the AP ARN Called the bucket directly
aws:MultiFactorAuthPresent MFA required, absent Was the session MFA’d? Console vs CLI session difference
kms:ViaService Key call not via the allowed service/region Read the key-policy condition Wrong region in s3.<region>.amazonaws.com
s3:x-amz-content-sha256 Unsigned payload where signing required Check for UNSIGNED-PAYLOAD Streaming client sends unsigned payload

The three nastiest, in prose

1. The SSE-KMS 403 that is really a KMS problem. This is the single most misdiagnosed S3 denial. Everything about the error says S3 — it’s a GetObject, the status is 403, the console is S3 — but the cause is a different service. The one reliable tell is the errorMessage: if it names kms:Decrypt (or kms:GenerateDataKey) and a key ARN, stop looking at S3 entirely. Your s3:GetObject is fine; the key is refusing to decrypt. There are five sub-cases, and you must check them in order: (a) the key policy doesn’t name your principal; (b) the key delegates to IAM (Enable IAM policies) but your identity policy lacks kms:Decrypt; © cross-account — the key policy must name the external principal and the caller’s IAM must allow it (two doors on the key, on top of the two on S3, for a total of four grants for a cross-account SSE-KMS read); (d) a grant that a service relied on was retired (list-grants); (e) a kms:ViaService or encryption-context condition excludes the call. The fix is always at the key, never at S3 — and the deepest trap is spending an hour widening S3 permissions that were never the problem.

2. The cross-account object-ownership trap. You own the bucket. You have s3:GetObject on bucket/*. You get 403 — but only on some objects, the ones a different account uploaded. With legacy ObjectWriter ownership (ACLs enabled), the writer owns the object it creates, so unless that writer sent a bucket-owner-full-control ACL, the bucket owner has no permission to its own bucket’s object. get-object-acl is the smoking gun: the Owner is a canonical ID that isn’t yours. The reason this is nasty is that it’s intermittent by object — the bucket “works” for everything you wrote and 403s for everything a partner wrote, which reads like flaky permissions rather than an ownership model. The fix is Object Ownership = BucketOwnerEnforced, which disables ACLs so the bucket owner owns every future object, plus a one-time in-place copy-object to re-own the existing foreign objects. Set BucketOwnerEnforced on every new bucket and this entire class disappears.

3. The s3:ListBucket-vs-s3:GetObject confusion. Two failure shapes hide here. First, the ARN split: s3:ListBucket is a bucket action needing arn:aws:s3:::bucket, while s3:GetObject is an object action needing arn:aws:s3:::bucket/* — grant one ARN and the other action 403s, which looks like a partial outage. Second, and far subtler: S3 masks a missing object as a 403 for principals who lack s3:ListBucket. If you have s3:ListBucket, a GetObject on a key that doesn’t exist returns 404 NoSuchKey (honest). If you don’t have s3:ListBucket — the common least-privilege posture for a read-only app role — that same non-existent key returns 403 AccessDenied, because S3 refuses to confirm or deny the key’s existence to someone who can’t list. The result: a wrong prefix or a missing zero-pad in the key name presents as a permissions denial, and teams widen policies for an hour chasing a typo. The rule: on a GetObject 403 where the role lacks s3:ListBucket, prove the key exists (head-object from an admin principal) before assuming the policy is wrong.

Best practices

Security notes

The debugging table and the security posture are two sides of one coin: every layer that can deny you is a layer that protects the data, and the temptation under a 403 page is to weaken it.

Principle Why it matters in a 403 How
Least-privilege on the fix The denial is a chance to grant exactly what’s needed Add the specific action + ARN, never s3:* or public
Never disable BPA to “fix” a 403 BPA rarely is the deny, and disabling it opens a leak Front with CloudFront/OAC or presign
Keep explicit-Deny guardrails Force-TLS/SSE/perimeter Denies are working when they 403 you Satisfy the condition; don’t delete the guardrail
KMS as a second authorization layer The key is authoritative for the key Scope kms:Decrypt/GenerateDataKey to real consumers only
Boundaries and SCPs are guarantees They deny because central policy says so Change them centrally, with review — not by local override
Object Lock is integrity, not a bug A Compliance-mode 403 is the control working Design bypass roles narrowly; never loosen the mode
Log the denials CloudTrail data events are the audit trail and the debugger Enable on sensitive buckets; alarm on AccessDenied spikes
Don’t leak object existence The 403-masks-404 behaviour is deliberate Keep least-privilege reads without s3:ListBucket where apt

Cost & sizing

The diagnostic capabilities cost almost nothing; the two line items to watch are CloudTrail data events and the KMS requests that SSE-KMS drives.

Item What drives cost Rough cost Note / free-tier
IAM Policy Simulator Free Always free
get-bucket-policy / get-key-policy / describe-key Free Standard API calls
CloudTrail management events Free (first copy) On by default
CloudTrail data events Per event logged ~$0.10 / 100k events The lever — scope selectors to buckets that matter
CloudTrail Lake Ingest + scan ~$2.50/GB ingest; query per GB scanned Cheaper than Athena for ad-hoc if already ingesting
Athena over CloudTrail logs Per TB scanned ~$5/TB scanned Partition by date to cut scan cost
S3 server access logs Storage of logs S3 storage rates Free-ish but noisy; prefer data events
SSE-KMS requests Decrypt/GenerateDataKey per object op ~$0.03 / 10k requests Enable S3 Bucket Keys — up to ~99% fewer calls
Customer-managed KMS key Per key/month ~$1/key/mo AWS-managed key is free; CMK is ~$1

The practical guidance: scope data-event selectors to the buckets you actually need to audit rather than “all S3,” because on a high-traffic estate “log every object event everywhere” is the surprise line on the CloudTrail bill. When SSE-KMS is in play, always turn on S3 Bucket Keys — at any real object volume the KMS request cost dwarfs the ~$1/month key, and Bucket Keys cut it by up to 99% while also reducing the CloudTrail KMS-event volume you pay to store. For ad-hoc denial hunts, partition your CloudTrail-log Athena table by date so a “who was denied yesterday” query scans megabytes, not terabytes.

Interview & exam questions

Q1. List the policy sources that can produce an S3 403 AccessDenied. SCP, RCP, VPC endpoint policy, Block Public Access, bucket policy (explicit Deny or missing principal), IAM identity policy, permission boundary, session policy, KMS key policy/grant/state, and object-level blocks (Object Lock, requester-pays, cross-account ownership). Any explicit Deny among them wins. (SCS-C02, SAA-C03)

Q2. The IAM policy allows s3:GetObject and the Policy Simulator says Allow, but you’re denied. Name three layers the simulator can’t see. Session policies (passed at AssumeRole), KMS key policies (SSE-KMS objects), Block Public Access, VPC endpoint policies, and cross-account resource-policy logic. The simulator validates only the identity layer. (SCS-C02, DVA-C02)

Q3. A GetObject 403 message names kms:Decrypt and a key ARN. Which layer, and where’s the fix? The KMS key. The object is SSE-KMS with a key that doesn’t grant the caller kms:Decrypt — fix the key policy (or add a grant), not the S3 or IAM policy. Cross-account needs the key policy to name the external principal and the caller’s IAM to allow kms:Decrypt. (SCS-C02)

Q4. Why can a GetObject on an existing-looking key return 403 instead of 404? If the caller lacks s3:ListBucket, S3 masks a non-existent object as 403 AccessDenied (rather than 404 NoSuchKey) to avoid leaking object existence. So the “denial” may be a wrong key. Grant s3:ListBucket or head-object from an admin to confirm. (DVA-C02, SCS-C02)

Q5. Same-account vs cross-account: how many policies must allow an S3 GetObject? Same-account: one (identity OR bucket policy). Cross-account: both (identity AND bucket policy). SSE-KMS adds a third — the KMS key policy — and cross-account SSE-KMS effectively needs four grants. (SAA-C03, SCS-C02)

Q6. A DeleteObject 403s though the role has s3:DeleteObject and GetObject works. Likely cause? Object Lock. Governance-mode retention needs s3:BypassGovernanceRetention plus the bypass header; Compliance-mode retention blocks everyone (even root) until the retain-until date; a legal hold blocks deletes until turned off. (SCS-C02, SOA-C02)

Q7. What does the reason clause with an explicit deny in a service control policy tell you, and where do you fix it? An SCP at the Organizations OU denies the action for every principal in the account. The fix is at the OU (Organizations), not the bucket or IAM — widening those does nothing. (SCS-C02)

Q8. How do you find the exact denied S3 API call and principal, and why won’t cloudtrail lookup-events show it? Enable S3 data events and query the delivered logs via Athena or CloudTrail Lake — lookup-events returns only management events, so object-level denials never appear there. (SOA-C02, SCS-C02)

Q9. A bucket works from your laptop but 403s from an EC2 instance in a VPC. Prime suspect? The VPC endpoint policy (or a bucket-side aws:SourceVpce condition) — the instance’s traffic traverses the S3 gateway endpoint, whose policy caps the call. Read both the endpoint policy and the bucket condition. (SAA-C03, ANS-C01)

Q10. Bucket owner 403s on objects another account uploaded. What setting fixes it and how? Object Ownership = BucketOwnerEnforced, which disables ACLs so the bucket owner owns every object going forward; re-own existing foreign objects with an in-place copy-object. (SAA-C03, SCS-C02)

Q11. What is a permission boundary, and how does it produce a 403 that a wider identity policy won’t fix? A managed policy that caps a principal’s maximum permissions; the effective set is identity ∩ boundary. If the boundary omits the action, it’s denied no matter how permissive the identity policy — you must widen the boundary. (SCS-C02, DVA-C02)

Q12. A presigned URL that worked yesterday now 403s. Two likely causes? The X-Amz-Expires window elapsed, or the signer’s temporary role-session credentials expired (a URL “signed for 7 days” is only valid while the signer’s session is). Re-sign with a live, still-permitted signer. (DVA-C02)

Quick check

  1. A 403 message ends with with an explicit deny in a resource-based policy. Which layer, and is the fix to add an Allow or satisfy a condition?
  2. The Policy Simulator says Allow but you’re still denied on an SSE-KMS object. What’s the most likely layer, and why is the simulator silent?
  3. You have s3:GetObject but not s3:ListBucket, and a GetObject returns 403. Before editing any policy, what must you rule out?
  4. Which S3 403s appear only on write or delete, never on read? Name two.
  5. Why won’t aws cloudtrail lookup-events help you find a denied GetObject?

Answers

  1. The bucket policy, via an explicit Deny — almost always a Condition (TLS, SSE, aws:SourceIp, aws:SourceVpce). The fix is to satisfy the condition (use HTTPS/SSE/the right endpoint) or scope the Deny; adding an Allow elsewhere cannot beat an explicit Deny.
  2. The KMS key policy/grant. The object is SSE-KMS and the caller lacks kms:Decrypt on the key. The Policy Simulator evaluates only identity policies (and boundaries/SCPs), never KMS key policies — so it shows Allow while the key denies.
  3. That the object actually exists. Without s3:ListBucket, S3 returns 403 for a missing key (masking the 404). head-object from an admin, or grant s3:ListBucket temporarily — the “denial” may be a wrong key.
  4. Object Lock (Governance/Compliance retention) and legal hold — both block overwrite/delete of a protected version while reads succeed. (Cross-account object ownership can also 403 reads, but the lock/hold class is write/delete-only.)
  5. Because lookup-events searches only management events. Object-level GetObject is a data event, which is off by default and, once enabled, is read from the delivered logs (Athena) or CloudTrail Lake — never from lookup-events.

Glossary

Next steps

AWSS3403 AccessDeniedIAMKMSCloudTrailObject LockTroubleshooting
Need this built for real?

Vinod is a Senior Cloud Architect (22+ yrs) — available for Azure / AWS / GCP architecture, landing zones, and migrations.

Work with me

Comments

Keep Reading