Debugging IAM 'Access Denied': Policy Evaluation Logic, SCPs, Boundaries & a Playbook

The API call comes back the same way it always does when it’s this kind of day: An error occurred (AccessDenied) when calling the GetObject operation. You wrote the identity policy yourself. You can see the Allow. And it is still denied. This is the single most common — and most misdiagnosed — failure in all of AWS, because AccessDenied does not mean “you have no permission.” It means “at least one of seven independent gates said no, and I’m not going to tell you which one unless you know how to ask.” The trap is that engineers stare at the identity policy for an hour when the real denier is a service control policy two OUs up, a permission boundary attached by the platform team, a KMS key policy that never named you, or a resource policy in another account that has no idea you exist.

This article is the diagnostic playbook. The depth anchor is the thing almost nobody holds fully in their head: AWS’s complete policy-evaluation logic — the order in which explicit Deny, SCPs/RCPs, resource-based policies, identity-based policies, permission boundaries, session policies and VPC endpoint policies are combined, and the one iron rule underneath it: every request is denied by default; an Allow is needed to override the default; and that Allow must survive every gate — while a single Deny anywhere ends the evaluation immediately. Once that model is in your hands, “Access Denied” stops being a wall and becomes a decision tree with exactly one true answer.

Then we make it operational. You will learn to read the AccessDenied message itself — the newer messages name the failing policy type in plain English — to decode the encoded authorization failure message with aws sts decode-authorization-message, and to drive the three tools that tell the truth: the IAM Policy Simulator, IAM Access Analyzer, and CloudTrail (where every denied call is logged with its errorCode and errorMessage). Because this is a reference you will open mid-incident, the evaluation gates, the message reasons, the condition keys, the ARN traps and the playbook itself are all laid out as scannable tables. Read the prose once; keep the tables open at 02:00.

What problem this solves

IAM is deny-by-default and additive-with-caps, and that combination is genuinely hard to reason about under pressure. A permission you need can be granted in one place and quietly removed in another. A cross-account call needs two policies to agree. An encrypted object needs a third policy — on the key — that lives in a different blade entirely. The failure surface is not “did I write an Allow” but “did my Allow survive an SCP, a boundary, a session policy, a resource policy and a key policy, and did none of them contain a Deny.” Six or seven things must all be true; the error tells you only that something wasn’t.

What breaks without this knowledge is time and trust. On-call engineers widen a policy to "Action": "*" and it still fails (because the denier was an SCP, and * can’t beat an allow-list gap or a Deny), which teaches the terrifying lesson that “IAM is random.” Others grant AdministratorAccess to “unblock” a workload — turning a five-minute permissions fix into a standing security finding — and it still fails because the real gate was a KMS key policy or a permission boundary that AdministratorAccess cannot exceed. The actual cause is always precisely diagnosable; the skill is knowing which of the seven gates to interrogate and with which command.

Who hits this: everyone, but hardest in AWS Organizations environments (SCPs and RCPs add gates most builders never see), in cross-account designs (assume-role trust policies, resource policies, external IDs), on encrypted workloads (the KMS gate), and on any team that uses permission boundaries to delegate IAM safely (the boundary silently caps every role the delegated admins create). The fix is almost never “add more permissions” — it’s “find the gate that is saying no and make that specific gate say yes.”

To frame the whole field before the deep dive, here is every denial class this article covers, what is actually saying no, and the first place to look.

Learning objectives

By the end of this article you can:

• Recite and apply AWS’s complete policy-evaluation order — explicit Deny → SCP/RCP → resource-based → identity-based → permission boundary → session policy → VPC endpoint policy — and state the deny-by-default rule that ties them together.
• Read an AccessDenied message and map its exact wording to the gate that denied you, distinguishing an implicit deny (“no identity-based policy allows”) from each explicit-deny variant (“with an explicit deny in a service control policy / permission boundary / resource-based policy / VPC endpoint policy”).
• Decode the encoded authorization failure message with aws sts decode-authorization-message and read allowed, explicitDeny, matchedStatements and failures.
• Drive the IAM Policy Simulator (simulate-principal-policy / simulate-custom-policy) and know its blind spots (it does not evaluate SCPs), use Access Analyzer for external-access and unused-access findings and policy checks, and query CloudTrail for the denied call by errorCode.
• Diagnose the nine root-cause families: missing Action, wrong Resource ARN, Condition mismatch (SourceIp / MFA / PrincipalOrgID), SCP/RCP deny, permission boundary too tight, KMS key policy, cross-account trust policy, resource policy missing the principal, and wrong region/partition.
• Explain why an encrypted resource needs a KMS key policy grant in addition to the data-service permission, and why cross-account needs both an identity allow and a resource allow.
• Work a real AccessDenied to root cause in minutes using a structured symptom → root cause → confirm → fix playbook.

Prerequisites & where this fits

You should already know IAM’s building blocks: a principal (an IAM user, an IAM role session, or the account root), a policy (a JSON document of Statements each with Effect, Action, Resource and optional Condition), the difference between an identity-based policy (attached to a user/group/role) and a resource-based policy (attached to a resource like an S3 bucket or an SQS queue), and how a role is assumed via sts:AssumeRole. You should be comfortable running the aws CLI, reading JSON, and reading an ARN. If those are shaky, start with AWS Organizations & IAM Foundations and the hands-on IAM Users, Groups, Roles & Policies, then come back.

This sits at the top of the Security & Identity track: it is the debugging layer over everything those foundations build. It pairs tightly with AWS Organizations SCPs & Multi-Account Guardrails (SCPs are the gate most people forget) and IAM Cross-Account Roles & AssumeRole (trust policies, external IDs and session policies are three of the seven gates). Everything here maps to certification objectives: the evaluation order and SCPs are core to SCS-C02 (Security Specialty) and SAP-C02 (Solutions Architect Professional), and appear on SAA-C03 and DVA-C02.

A quick map of who owns which gate, so during an incident you page the right person fast:

Core concepts

Five mental models make every later diagnosis obvious.

Deny by default; an Allow overrides the default; a Deny overrides every Allow. Nothing is permitted until something explicitly allows it. That is the base state — an implicit deny. An Allow in an identity-based or resource-based policy lifts the implicit deny. But an explicit Deny in any policy that applies to the request ends the evaluation with a denial, no matter how many Allows exist. There is no “score”; Deny is not outvoted. Root user is the one near-exception — the account root has implicit full access that SCPs can still restrict.

The request is evaluated against a set of policies, and the caps only subtract. When you call an API, AWS assembles every policy in scope — identity policies on the principal, any resource policy on the target, the permission boundary on the principal, the SCPs/RCPs on the account’s OU path, any session policy from the assume-role, and any VPC endpoint policy on the path. SCPs, RCPs, permission boundaries and session policies never grant anything — they are maximum-permission filters. They can only turn an Allow into a deny; they cannot create access. So the effective permission set is (identity ∪ resource allows) ∩ SCP ∩ RCP ∩ boundary ∩ session ∩ endpoint, minus any explicit Deny anywhere.

Cross-account needs both doors open. Inside one account, an action is allowed if either the identity policy or the resource policy allows it (for services that have resource policies). Across accounts it is stricter: the caller’s identity policy in account A must allow the action and the resource’s resource policy in account B must allow the caller — both, independently. Assuming a role is the canonical case: your identity needs sts:AssumeRole on the role, and the role’s trust policy (a resource-based policy) must name your principal.

Encryption adds a whole extra gate. If a resource is encrypted with a customer-managed KMS key, reading or writing it requires the data-service permission and a KMS permission (kms:Decrypt, kms:GenerateDataKey, …) that the KMS key policy allows for your principal. The key policy is authoritative for the key: unless it either names your principal or delegates to IAM (the default Enable IAM User Permissions statement), no identity policy in the world grants key access. This is why an S3 GetObject with a perfectly good bucket permission still 403s — the denier is the key, in a different console entirely.

The message is diagnostic — if you read it precisely. AWS’s AccessDenied messages evolved to name the reason. “no identity-based policy allows” is an implicit deny (you never granted it). “with an explicit deny in a service control policy” is an SCP Deny. “with an explicit deny in a permissions boundary” is the boundary. For services that support it (EC2 and others), the error carries an encoded authorization failure message you decode to see the matched statements. Learning to read the exact wording collapses the search space from “seven gates” to “this gate.”

The vocabulary in one table

Pin down every moving part before the deep sections; the glossary repeats these for lookup.

The complete policy-evaluation logic

This is the depth anchor. When a principal makes a request, AWS runs a fixed evaluation to a single boolean. The order matters because the first rule short-circuits everything: any explicit Deny ends the evaluation immediately. After that, each cap must permit the action, and at least one Allow must exist. Here is the full order, top to bottom, exactly as AWS combines them within an account (plus the VPC endpoint gate when the request traverses one).

Read the flow as a gauntlet: the request starts denied, an Allow (from identity or resource) lifts it, and then it must pass through every cap without hitting a Deny. The truth table below is the entire model compressed — memorise it and you can predict any outcome.

Three subtleties that trip up even experienced engineers, stated precisely so you don’t over- or under-claim them:

Gate 1 — Explicit Deny (always wins)

An explicit Deny in any policy in scope ends the evaluation with a denial. There is no override — not AdministratorAccess, not root, not a resource policy allow. Because a Deny can live in six different places, “who wrote the Deny” is the whole question. This is the gate people fight for hours because they keep adding Allows, which cannot help.

A frequent real Deny is the data-perimeter pattern — an SCP that denies everything unless aws:PrincipalOrgID matches, or aws:SourceIp is in the corporate range. A developer on home Wi-Fi trips the IP condition and reads it as “my policy is wrong.” It isn’t; the guardrail is doing its job.

Gate 2 — SCPs and RCPs (the Organizations cap)

Service control policies are Organizations guardrails attached to the management account, an OU, or an account. They cap the maximum permissions of every principal in member accounts — and they never grant. Two failure modes: an allow-list gap (the SCP uses Allow on a specific list and your action isn’t on it, so it’s implicitly denied) and an explicit Deny. The management account is not restricted by SCPs, which is exactly why “it works in the org’s management account but fails in the member account” is the SCP fingerprint. Resource control policies (RCPs) are the newer sibling: they cap what resource-based policies can grant across the org (e.g. force aws:PrincipalOrgID on every bucket), and they surface as “explicit deny in a resource control policy.”

Confirm the effective SCP for the account you’re denied in (run from the management or a delegated-admin account):

# The union/intersection of every SCP on this account's OU path
aws organizations describe-effective-policy \
  --policy-type SERVICE_CONTROL_POLICY \
  --target-id 123456789012 \
  --query 'EffectivePolicy.PolicyContent' --output text | jq .

The classic allow-list-gap SCP — everything below it is denied by omission the moment you leave FullAWSAccess:

{
  "Version": "2012-10-17",
  "Statement": [
    { "Sid": "AllowedServicesOnly", "Effect": "Allow",
      "Action": ["s3:*", "ec2:*", "cloudwatch:*", "logs:*"],
      "Resource": "*" }
  ]
}

If your action is dynamodb:GetItem, it’s not on the list, so it’s denied in every member account regardless of a perfect identity policy. The fix is to add the service to the SCP allow-list, not to touch the identity policy. In Terraform:

resource "aws_organizations_policy" "allowed_services" {
  name    = "allowed-services"
  type    = "SERVICE_CONTROL_POLICY"
  content = data.aws_iam_policy_document.allowed.json
}

The two SCP strategies denote where the denial comes from — an omission or an explicit Deny — and you debug them differently:

Gate 3 — Resource-based policies

Some services let you attach a policy to the resource: an S3 bucket policy, an SQS/SNS access policy, a Lambda resource policy, a KMS key policy, an IAM role trust policy, a Secrets Manager secret policy, an ECR repository policy, and more. A resource policy names a Principal (who) and grants directly. Same-account, a resource allow or an identity allow is enough. Cross-account, the resource allow is mandatory (plus the caller’s identity allow). The commonest bug: the resource policy’s Principal doesn’t include your ARN, or names the role ARN when the caller is an assumed-role session (or vice versa), or uses "*" narrowed by a Condition you fail.

Read a bucket policy and a role’s trust policy — the two you check most:

# The bucket's own resource policy — is your principal in it?
aws s3api get-bucket-policy --bucket my-app-prod-bucket \
  --query Policy --output text | jq .

# The role's trust policy (resource policy) — who is allowed to assume it?
aws iam get-role --role-name cross-account-reader \
  --query 'Role.AssumeRolePolicyDocument' --output json | jq .

Gate 4 — Identity-based policies

The policy attached to the calling user or role. This is where you usually wrote the Allow, and where three concrete mistakes cause 90% of self-inflicted denials: (a) missing Action — you allowed s3:GetObject but the call is s3:ListBucket (listing is a different action on a different resource, the bucket not the object); (b) wrong Resource ARN — bucket vs object (arn:aws:s3:::b vs arn:aws:s3:::b/*), wrong account/region/partition, or a wildcard that doesn’t cover the path; © Condition fails — an MFA, SourceIp or tag condition evaluates false. The simulator is built exactly for this gate.

# Does alice's identity policy allow GetObject on this exact object?
aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::123456789012:user/alice \
  --action-names s3:GetObject \
  --resource-arns arn:aws:s3:::my-app-prod-bucket/reports/q3.csv \
  --query 'EvaluationResults[].{Action:EvalActionName,Decision:EvalDecision}' \
  --output table

A result of implicitDeny means “no statement allowed it” — add the Action/Resource. explicitDeny means “a Deny matched” — hunt the Deny (often a boundary you can pass with --permissions-boundary-policy-input-list, or an SCP the simulator can’t see — see the tool caveats).

Gate 5 — Permission boundaries

A permission boundary is a managed policy attached to an IAM user or role that sets the maximum permissions that entity can have. The effective permissions are the intersection: an action is allowed only if both the identity policy and the boundary allow it. Boundaries never grant; they cap. They exist to let you safely delegate IAM — a junior admin can create roles, but a boundary condition forces every new role to carry the same boundary, so nobody can escalate beyond it. The failure mode: the identity policy has the Allow, the boundary omits the action, and you get an implicit deny that looks impossible (“but AdministratorAccess is attached!”). See AWS Organizations & IAM Foundations for how boundaries and SCPs layer together.

# Simulate WITH the boundary applied — reveals the intersection
aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::123456789012:role/app-role \
  --action-names dynamodb:PutItem \
  --resource-arns arn:aws:dynamodb:ap-south-1:123456789012:table/orders \
  --permissions-boundary-policy-input-list file://boundary.json

Gate 6 — Session policies

When you assume a role (or federate), you can pass an inline --policy or managed --policy-arns that further restricts the session. The session’s effective permissions are the intersection of the role’s identity policy and the session policy. This is invisible unless you inspect the assume-role call, so “the role works when I test it, but the running session is denied” points straight here. The service that assumed the role (a container, a Lambda, a federation broker) may be attaching a tighter session policy than you realise.

# The session policy caps this session below the role's own permissions
aws sts assume-role \
  --role-arn arn:aws:iam::123456789012:role/app-role \
  --role-session-name debug \
  --policy '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":"s3:GetObject","Resource":"*"}]}'
# This session can ONLY GetObject, even if app-role allows far more.

Gate 7 — VPC endpoint policies, and the KMS key gate

Two more gates that live off to the side. A VPC endpoint policy (on a gateway endpoint for S3/DynamoDB, or an interface endpoint) is a resource-style policy that caps what can be done through that endpoint. The default is allow-all; the moment someone attaches a restrictive one, calls that traverse the endpoint can be denied even with perfect identity and resource policies. And the KMS key policy — covered as a gate in its own right below because it is the single most surprising denier — must allow your principal kms:Decrypt/GenerateDataKey for any encrypted-resource read/write.

Because the KMS key policy is the single most surprising denier, know its anatomy. A console-created customer-managed key ships with these statements; the absence of the first is the classic trap that makes identity policies silently useless on that key:

Decoding the Access Denied message

The message is the fastest diagnostic you own, because the newer AWS wording names the gate. Learn to parse it. A typical modern message:

An error occurred (AccessDenied) when calling the GetObject operation:
User: arn:aws:iam::123456789012:user/alice is not authorized to perform:
s3:GetObject on resource: arn:aws:s3:::my-app-prod-bucket/reports/q3.csv
because no identity-based policy allows the s3:GetObject action

Every field is a clue: the principal ARN (are you even who you think you are?), the exact action, the exact resource (bucket vs object!), and — the gold — the reason clause at the end. Map the reason clause to the gate:

The encoded authorization failure message

Some services (notably EC2, and others) return an encoded authorization failure message — an opaque, base64-ish blob — instead of a plain reason, so as not to leak policy detail to an unauthorised caller. If you are authorised to decode it (sts:DecodeAuthorizationMessage), it is the richest signal available: it shows whether the decision was allowed, whether an explicit deny fired, and which statements matched.

# EC2 example: the RunInstances error contains <Encoded>...</Encoded>
aws sts decode-authorization-message \
  --encoded-message "<the long encoded blob>" \
  --query DecodedMessage --output text | jq .

The decoded JSON tells you exactly which gate and statement decided the request:

Reading context.conditions against your policy’s Condition is how you catch a aws:SourceIp that didn’t match, or an aws:PrincipalTag/Project that was absent because the session wasn’t tagged.

The diagnostic tools

Four tools, each with a different view and a different blind spot. Use the right one for the gate you suspect.

CloudTrail: find the denied call

Every denied API call is logged. In production, CloudTrail is where you learn who was denied doing what — often revealing that the principal was not who you assumed (a container using the task role, not your user). Query it in the console (Event history, filter by User name or Event source), via lookup-events, or with Athena/CloudTrail Lake for volume.

# Recent denied calls via CloudTrail Lake (or use Athena on the S3 trail)
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=AssumeRole \
  --query 'Events[?contains(CloudTrailEvent, `AccessDenied`)].CloudTrailEvent' \
  --output text | jq '{who:.userIdentity.arn, err:.errorCode, msg:.errorMessage}'

The errorCode families you’ll see, and what each means:

An Athena query over the CloudTrail table to pull every denial in a window — the fastest way to see a pattern (one principal, one action, one resource):

SELECT eventtime, useridentity.arn AS who, eventsource, eventname,
       errorcode, errormessage
FROM cloudtrail_logs
WHERE errorcode IN ('AccessDenied','AccessDeniedException','Client.UnauthorizedOperation')
  AND eventtime > '2026-07-13T00:00:00Z'
ORDER BY eventtime DESC
LIMIT 100;

The fields inside each CloudTrail event are the diagnosis. Read them in this order — the first three routinely change the whole investigation because they reveal the request ran as a different principal than you assumed:

When it’s not IAM: aws --debug and the client side

Before you blame a policy, rule out the client. Wrong region, wrong profile, or an expired session all look like denials but aren’t policy problems. aws --debug prints the signed request and the credentials used:

Access Analyzer and the simulator caveats

IAM Access Analyzer answers different questions: external access findings (which resources are shared outside your account/org), unused access findings (permissions and roles nobody uses), policy validation (100+ checks on a policy document), and custom policy checks like check-no-new-access (does this change grant anything new?) and check-access-not-granted (does this policy grant a sensitive action?). It won’t tell you “why was this one call denied,” but it will tell you your resource policy is too open (the inverse problem) or that the action you’re fighting over is unused and should perhaps be removed. The most important caveat to internalise, so you don’t trust a green simulator over a red production call:

Architecture at a glance

The diagram traces a single request left to right through all seven gates. On the left the principal makes the call and CloudTrail records it; the request enters denied by default and must survive each gate. First the Org guardrails — an explicit Deny anywhere wins instantly (badge 1), then the SCP/RCP cap must allow it (badge 2). Next the resource side — the resource policy must name the principal for cross-account (badge 3), and if the target is encrypted the KMS key policy is an extra gate for kms:Decrypt (badge 4). Then identity + limits — the identity policy must allow the exact Action on the exact Resource with any Condition satisfied (badge 5), intersected with the permission boundary and any session policy (badge 6). Only a request that collected an Allow and passed every cap with no Deny reaches Allow; anything else exits as AccessDenied, which you then decode back to the gate that stopped it.

Real-world scenario

Nimbus Analytics runs a 40-account AWS Organization with a data-perimeter guardrail: a top-level SCP denies every action unless aws:PrincipalOrgID equals the org, plus an RCP that forces the same on all resource policies. A new team ships a nightly job in the data-prod account: a Fargate task assumes role/etl-runner, reads raw files from an S3 bucket in the same account, and writes curated Parquet to a bucket in the data-lake account, both encrypted with customer-managed KMS keys. In staging it works. In production, the task fails with AccessDenied on the cross-account PutObject, and — intermittently — on the same-account read too.

The on-call engineer’s first instinct is the identity policy on etl-runner, which plainly allows s3:PutObject on the target ARN. Simulator says Allow. That green result is the trap: the simulator doesn’t evaluate SCPs, RCPs, the cross-account resource policy, or the KMS gate. She pulls CloudTrail and reads the actual errors. Two distinct errorMessages appear. The cross-account write says “because no resource-based policy allows the s3:PutObject action” — the data-lake bucket policy names the role ARN arn:aws:iam::…:role/etl-runner, but the caller is an assumed-role session and, more to the point, cross-account requires the resource policy to allow, which it did — except the RCP on data-lake denied it because the bucket policy’s Principal block didn’t carry an aws:PrincipalOrgID condition, so the resource guardrail stripped the grant. Reason clause: nailed to gate 2/3.

The intermittent same-account read is stranger. It fails only on files written that day. She decodes the pattern: the new raw files are encrypted with a second KMS key the platform rotated in, and that key’s key policy grants kms:Decrypt only to a named admin role — not to etl-runner. The data-service permission (s3:GetObject) was fine; the KMS gate (badge 4) denied it, and the S3 error surfaced as a 403 whose errorMessage named kms:Decrypt on the new key ARN. Old files, encrypted with the old (correctly-delegated) key, still read — hence “intermittent.”

The fix is three precise edits, none of them “add more IAM to the role”: (1) add aws:PrincipalOrgID to the data-lake bucket policy’s Principal grant so the RCP stops stripping it; (2) add etl-runner to the new KMS key policy with kms:Decrypt/GenerateDataKey (or, better, restore the key’s Enable IAM User Permissions delegation so identity policies work); (3) change the bucket-policy Principal to the role ARN form that matches assumed sessions and scope it with aws:PrincipalOrgID. Total change: eleven lines across a bucket policy and a key policy. Time lost before the method kicked in: two hours of staring at a green simulator. The lesson Nimbus wrote into its runbook: read the CloudTrail errorMessage first; it names the gate. The simulator is for before you deploy, not for why production is denied.

Advantages and disadvantages

The layered, deny-by-default model is what makes AWS safe at scale — and what makes debugging it a skill. The trade-offs, then when each matters.

The layering matters most in regulated, multi-account estates where a central team must guarantee limits no member-account admin can exceed — there, SCPs and boundaries are non-negotiable and the debugging cost is the price of provable containment. In a single-account startup, most gates are dormant (no SCPs, no boundaries, no cross-account), so denials collapse to “the identity policy is wrong,” and the simulator alone usually suffices.

Hands-on lab

You will manufacture and then diagnose three real denials — an identity implicit deny, a boundary intersection deny, and a KMS-gate deny — using only free-tier-safe resources. Everything here is free except a few cents of S3/KMS if you leave it running; teardown is at the end. Use a sandbox account, not production.

Step 0 — Confirm who you are. Half of all “impossible” denials are “you’re not the principal you think.” Always start here.

aws sts get-caller-identity
# Expected: { "UserId": "...", "Account": "123456789012",
#             "Arn": "arn:aws:iam::123456789012:user/you" }

Step 1 — Create a test user with a deliberately narrow policy.

aws iam create-user --user-name denytest
aws iam put-user-policy --user-name denytest --policy-name narrow \
  --policy-document '{"Version":"2012-10-17","Statement":[
    {"Effect":"Allow","Action":"s3:ListAllMyBuckets","Resource":"*"}]}'
aws iam create-access-key --user-name denytest   # note the keys for a scratch profile

Step 2 — Reproduce an identity implicit deny. Configure a scratch profile with those keys, then try an action you did not grant:

aws s3api list-objects-v2 --bucket some-bucket --profile denytest
# Expected: An error occurred (AccessDenied) ...
#   because no identity-based policy allows the s3:ListBucket action

Note the reason clause names identity and the action is s3:ListBucket (not ListObjects — the IAM action differs from the API name). Confirm with the simulator:

aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::123456789012:user/denytest \
  --action-names s3:ListBucket --resource-arns arn:aws:s3:::some-bucket \
  --query 'EvaluationResults[0].EvalDecision' --output text
# Expected: implicitDeny

Fix it (add the action + the bucket ARN, not the object ARN):

aws iam put-user-policy --user-name denytest --policy-name narrow \
  --policy-document '{"Version":"2012-10-17","Statement":[
    {"Effect":"Allow","Action":["s3:ListAllMyBuckets","s3:ListBucket"],
     "Resource":["*","arn:aws:s3:::some-bucket"]}]}'

Step 3 — Reproduce a boundary intersection deny. Attach a boundary that allows only EC2 describe calls, then grant the user S3 in the identity policy and watch the intersection deny S3:

aws iam put-user-policy --user-name denytest --policy-name broad \
  --policy-document '{"Version":"2012-10-17","Statement":[
    {"Effect":"Allow","Action":"s3:*","Resource":"*"}]}'
# Boundary that does NOT include S3:
aws iam create-policy --policy-name ec2-only-boundary \
  --policy-document '{"Version":"2012-10-17","Statement":[
    {"Effect":"Allow","Action":"ec2:Describe*","Resource":"*"}]}'
aws iam put-user-permissions-boundary --user-name denytest \
  --permissions-boundary arn:aws:iam::123456789012:policy/ec2-only-boundary
aws s3 ls --profile denytest
# Expected: AccessDenied — the identity policy allows s3:* but the
#   boundary (ec2 only) intersects it to nothing for S3.

Confirm the intersection with the simulator by supplying the boundary:

aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::123456789012:user/denytest \
  --action-names s3:ListAllMyBuckets --resource-arns "*" \
  --permissions-boundary-policy-input-list \
    '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":"ec2:Describe*","Resource":"*"}]}' \
  --query 'EvaluationResults[0].EvalDecision' --output text
# Expected: implicitDeny  (allowed by identity, not by the boundary)

Step 4 — Reproduce the KMS gate. Create a CMK whose key policy delegates to IAM, put an encrypted object, and see it read; then remove the delegation and watch the same S3 permission 403 on the KMS gate.

KEY_ID=$(aws kms create-key --description labkey --query KeyMetadata.KeyId --output text)
aws s3 mb s3://kv-iam-lab-$RANDOM
# ... put an object with --sse aws:kms --sse-kms-key-id $KEY_ID, then GetObject works
# Now replace the key policy with one that names ONLY your admin, not denytest:
aws kms put-key-policy --key-id $KEY_ID --policy-name default --policy '{
  "Version":"2012-10-17","Statement":[
   {"Sid":"AdminOnly","Effect":"Allow",
    "Principal":{"AWS":"arn:aws:iam::123456789012:user/you"},
    "Action":"kms:*","Resource":"*"}]}'
# denytest GetObject on the encrypted object now returns 403 naming kms:Decrypt

The lesson in one line: the S3 permission never changed — the key started denying you.

Step 5 — Teardown (avoid lingering charges).

aws iam delete-access-key --user-name denytest --access-key-id AKIA...
aws iam delete-user-policy --user-name denytest --policy-name narrow
aws iam delete-user-policy --user-name denytest --policy-name broad
aws iam put-user-permissions-boundary --user-name denytest --permissions-boundary "" 2>/dev/null || \
  aws iam delete-user-permissions-boundary --user-name denytest
aws iam delete-user --user-name denytest
aws iam delete-policy --policy-arn arn:aws:iam::123456789012:policy/ec2-only-boundary
aws kms schedule-key-deletion --key-id $KEY_ID --pending-window-in-days 7  # ⚠️ CMK: min 7-day window
aws s3 rb s3://kv-iam-lab-XXXX --force

⚠️ A customer-managed KMS key costs ~$1/month while it exists and can only be scheduled for deletion (7–30 day window). Schedule it now so it stops billing.

Common mistakes & troubleshooting

This is the playbook. Find your symptom, read across to the exact command that confirms the gate, then the fix. Work it top-down: the earlier gates (Deny, SCP) short-circuit everything, so rule them out first.

First, the 30-second triage — the pattern you notice maps almost 1:1 to the gate:

Now the full playbook:

The nastiest three, in prose

The explicit Deny you didn’t write (row 1). This is the number-one time-sink. An engineer sees their Allow, cannot fathom the denial, and starts broadening the identity policy — s3:*, then * on * — and it still fails. Nothing in an identity policy can beat a Deny that lives in an SCP, a permission boundary, an RCP or a resource policy. The instant you suspect this, stop editing Allows and read the errorMessage reason clause or decode the message; it will contain “with an explicit deny in a <policy type>,” which is the entire answer. Then go to that policy type and scope the Deny. The lesson: when broadening a policy makes no difference, you are fighting a Deny, and Allows are the wrong tool.

The KMS gate hiding behind an S3/EBS/Secrets 403 (rows 7, 18). Encrypted resources have a second, invisible authorization: the KMS key policy. A perfectly correct s3:GetObject grant returns 403 because the object is encrypted with a customer-managed key whose policy never named your principal. The tell is in the errorMessage — it references kms:Decrypt and a key ARN, not an S3 ARN. The default AWS-managed key delegates to IAM (so identity policies work), but a customer-managed key with a hand-written key policy may grant no one but an admin role. Cross-account secrets are the double-whammy: you need the secret’s resource policy and the secret’s KMS key to both allow you. Fix at the key, not the data service.

Cross-account is two doors, and people open one (rows 5, 6). Same-account habits break across accounts. In your own account, an identity Allow is enough. Across accounts, the caller’s identity policy in account A must allow the action and the resource policy (bucket policy, trust policy, key policy) in account B must allow the caller — both, independently, or it’s an implicit deny with “because no resource-based policy allows.” The assume-role case is the archetype: sts:AssumeRole on the identity side, and the role’s trust policy naming your principal on the resource side. If you’re using an external ID (the confused-deputy defence), the sts:ExternalId in the trust policy Condition must match exactly what the caller passes — a mismatch reads as a plain AssumeRole denial. See IAM Cross-Account Roles & AssumeRole for the full pattern.

The trust policy is where most assume-role denials actually live. Enumerate its failure modes so you check the right half of the two doors:

Condition-key mismatches — the quiet deniers

When the Action and Resource are right but you’re still denied, a Condition evaluated false. These are the ones that bite in production, and the operator matters as much as the key.

And the operator pitfalls themselves, which turn a “correct” value into a false match:

ARN and identity traps

Half of “no identity-based policy allows” is a subtly wrong Resource ARN. Enumerate the fields before you rewrite the policy.

Two more sources of “correct-looking but denied” policies: the wrong partition in an ARN, and the inverted policy elements (NotAction/NotResource/NotPrincipal) that quietly grant or deny far more than intended.

Best practices

• Start every diagnosis with aws sts get-caller-identity. More “impossible” denials than you’d believe are “you’re a different principal than you assumed” (a task role, a default profile, an expired session).
• Read the errorMessage reason clause before touching any policy. It names the gate — implicit vs explicit, and which policy type. Editing before reading it wastes the most time.
• When broadening a policy changes nothing, stop — you’re fighting a Deny. Switch from adding Allows to hunting the Deny (SCP, boundary, RCP, resource, session).
• Trust CloudTrail over the simulator for production denials. The simulator can’t see SCPs/RCPs or the live resource policy; CloudTrail records what actually happened.
• Grant least privilege, but grant the exact Action and Resource. Prefer specific ARNs; use Access Analyzer policy generation to build a policy from real CloudTrail activity.
• Treat cross-account as two policies from the start. Write the identity Allow and the resource/trust policy together, scoped by aws:PrincipalOrgID.
• Keep KMS keys delegating to IAM (Enable IAM User Permissions) unless you have a reason not to — hand-written key policies that name only an admin are a top hidden denier.
• Validate policies in CI with Access Analyzer (check-no-new-access, check-access-not-granted, policy validation) so over-broad or broken policies fail the PR, not production.
• Prefer permission boundaries over hope for delegation. Force every delegated role to carry the boundary so nobody can escalate past the ceiling.
• Alarm on AccessDenied spikes. An EventBridge rule / CloudWatch metric filter on errorCode = AccessDenied catches broken deploys and probing alike.
• Never fix a denial with "*". It usually doesn’t even work (the denier is a cap or a Deny), and it leaves a standing security finding.
• Document the seven gates in your runbook with the confirm command for each — the whole team debugs faster.

Security notes

Debugging denials is a security-sensitive activity: the temptation under pressure is to over-grant, and each of the gates exists to stop exactly that. Some principles specific to this topic:

Cost & sizing

IAM itself is free — policies, users, roles, evaluation, the simulator and Access Analyzer’s core features cost nothing. The bill, such as it is, comes from the evidence you keep and the encryption you gate with. Rough figures (region-dependent; INR at ~₹86/USD):

The practical guidance: keep the free management-events trail on in every account (it’s how you debug), enable data events only where you need object-level denials (they’re the cost driver), and remember every customer-managed KMS key is ~$1/month whether used or not — the lab teardown schedules yours for deletion for exactly this reason. Access Analyzer unused access is worth its ~$0.20/principal/month in a large estate because it directly shrinks the attack surface that causes these denials in the first place.

Interview & exam questions

Q1. State AWS’s policy-evaluation order and the one rule that overrides all others. Deny by default; then: any explicit Deny wins immediately; SCPs/RCPs must allow; a resource-based or identity-based Allow is required; permission boundary, session policy and VPC endpoint policy each must allow. The overriding rule: an explicit Deny anywhere beats every Allow. (SCS-C02, SAP-C02)

Q2. A principal has AdministratorAccess but is denied. Name three gates that can still deny it. A permission boundary (effective = policy ∩ boundary), an SCP/RCP in Organizations, an explicit Deny in a resource policy or session policy, or a KMS key policy for an encrypted resource. AdministratorAccess is an identity Allow and cannot beat a cap or a Deny. (SCS-C02)

Q3. Same-account, does an S3 GetObject need both an identity policy allow and a bucket policy allow? No — same-account, either suffices. Cross-account it needs both the caller’s identity allow and the bucket’s resource-policy allow. (SAA-C03, DVA-C02)

Q4. You get 403 on an encrypted S3 object though s3:GetObject is granted. What’s the likely gate and how do you confirm? The KMS key policy — the object is encrypted with a CMK that doesn’t grant your principal kms:Decrypt. Confirm from the errorMessage, which names kms:Decrypt and a key ARN; fix in the key policy. (SCS-C02, DVA-C02)

Q5. How do you tell an implicit deny from an explicit deny from the message alone? “no identity-based policy allows the X action” = implicit (nothing granted it). “with an explicit deny in a <policy type>” = explicit Deny, and it names which policy type. (SCS-C02)

Q6. What is the encoded authorization failure message and when do you see it? An opaque blob some services (notably EC2) return instead of a plain reason, to avoid leaking policy detail. Decode with aws sts decode-authorization-message (needs sts:DecodeAuthorizationMessage) to see allowed, explicitDeny, matchedStatements and failures. (SCS-C02)

Q7. Why can the IAM Policy Simulator say Allow while production says Deny? The simulator does not evaluate SCPs/RCPs, doesn’t auto-fetch the live cross-account resource policy, and approximates some conditions. Cross-check with CloudTrail’s errorMessage and organizations describe-effective-policy. (SCS-C02, SAP-C02)

Q8. An action works in the management account but fails in a member account. What’s the cause? An SCP — the management account is exempt from SCPs, member accounts are not. It’s either an allow-list gap (action not on the SCP’s Allow list) or an explicit Deny. Confirm with describe-effective-policy. (SAP-C02, SCS-C02)

Q9. AssumeRole is denied with “not authorized to perform: sts:AssumeRole.” Where do you look? Two places: the caller’s identity policy (needs sts:AssumeRole on the role) and the role’s trust policy (must name the caller as Principal). If an external ID is required, sts:ExternalId must match. (DVA-C02, SCS-C02)

Q10. What does a permission boundary do, and how does it differ from an SCP? A boundary is a max-permission filter on an IAM entity; effective permissions are the intersection of the entity’s policies and the boundary. An SCP is a max-permission filter on an account/OU in Organizations. Both only cap, never grant; the boundary is per-entity, the SCP is per-account. (SCS-C02, SAP-C02)

Q11. A cross-account role works for some callers, denied for others. What condition is the usual culprit? aws:PrincipalOrgID (or aws:SourceArn/aws:SourceAccount) in the trust or resource policy — callers outside the allowed org fail. Decode the message and compare context.conditions. (SCS-C02)

Q12. Which tool finds over-broad access rather than why-denied, and name two of its checks. IAM Access Analyzer — external-access findings, unused-access findings, policy validation, and custom checks check-no-new-access and check-access-not-granted. It’s the inverse of denial-debugging: it finds too much access. (SCS-C02)

Quick check

1. In what order does AWS evaluate: identity policy, explicit Deny, SCP, permission boundary?
2. Cross-account S3 GetObject: how many policies must allow it, and which?
3. You see “with an explicit deny in a permissions boundary.” Do you edit the identity policy or the boundary?
4. Which tool can’t see SCPs, and what do you use instead for a production denial?
5. An encrypted-object 403 names kms:Decrypt and a key ARN. Which gate, and where’s the fix?

Answers

1. Explicit Deny first (wins outright) → SCP (must allow) → identity policy (the Allow) → permission boundary (must allow). Session and VPC endpoint policies follow the boundary. Deny short-circuits everything.
2. Two: the caller’s identity policy (in the caller’s account) and the bucket policy (resource policy, in the bucket’s account) must both allow it.
3. The boundary. The message names the gate — the effective set is identity ∩ boundary, so add the action to the boundary.
4. The IAM Policy Simulator can’t see SCPs/RCPs; use CloudTrail (errorMessage) plus organizations describe-effective-policy for production.
5. The KMS key policy gate. Fix it in the key policy — grant the principal kms:Decrypt/GenerateDataKey, or keep the key’s IAM delegation — not in the S3/identity policy.

Glossary

• Implicit deny — The default outcome when no policy allows an action; the base state every request starts in.
• Explicit Deny — A Statement with Effect: Deny; overrides every Allow and ends evaluation immediately.
• Identity-based policy — A policy attached to an IAM user, group, or role; grants permissions to that principal.
• Resource-based policy — A policy attached to a resource (S3 bucket, KMS key, SQS queue, IAM role trust policy) naming a Principal.
• SCP (Service Control Policy) — An AWS Organizations guardrail on accounts/OUs that caps the maximum permissions of member-account principals; never grants.
• RCP (Resource Control Policy) — An Organizations guardrail that caps what resource-based policies can grant across the org.
• Permission boundary — A managed policy that sets the maximum permissions of an IAM entity; effective permissions are the intersection with its identity policies.
• Session policy — An inline or managed policy passed at AssumeRole/federation that further restricts the temporary session.
• Trust policy — The resource-based policy on an IAM role that defines which principals may assume it.
• KMS key policy — The resource-based policy on a KMS key; authoritative for who may use the key (kms:Decrypt, GenerateDataKey).
• VPC endpoint policy — A resource policy on an interface/gateway VPC endpoint that caps what can be done through it.
• Encoded authorization failure message — An opaque blob some services return in AccessDenied errors; decode with sts decode-authorization-message.
• aws:PrincipalOrgID — A condition key matching the caller’s Organizations org id; the backbone of data-perimeter guardrails.
• Effective permissions — The net of every gate: (identity ∪ resource) ∩ SCP ∩ RCP ∩ boundary ∩ session ∩ endpoint, minus any explicit Deny.
• Confused deputy — An attack where a service is tricked into using its permissions on an attacker’s behalf; mitigated with aws:SourceArn/ExternalId.

Next steps

• Ground the model in the fundamentals: AWS Organizations & IAM Foundations.
• Build the muscle memory for writing the policies you’re debugging: IAM Users, Groups, Roles & Policies (hands-on).
• Master the gate people forget: AWS Organizations SCPs & Multi-Account Guardrails.
• Go deep on the two-doors cross-account pattern, external IDs and session policies: IAM Cross-Account Roles & AssumeRole (hands-on).