GCP 'Permission Denied'? The IAM Troubleshooting Decision Tree That Always Finds It

You shipped nothing. You changed nothing. And yet a deploy that worked yesterday now dies with PERMISSION_DENIED, the console tells a user “You do not have permission to view this page”, or a Cloud Run service returns a stark 403 to a caller that has always been allowed. Welcome to the single most common — and most misdiagnosed — failure in Google Cloud IAM, the engine that decides who can do what on which resource. The maddening part is that the error almost never tells you the whole truth on its face. “Permission denied” is the symptom; the cause is one of roughly a dozen distinct things living in completely different places — a missing role, a binding on the wrong resource in the hierarchy, an IAM Condition that evaluated false, an org policy that forbade the action before IAM ever ran, a deny policy that vetoed an otherwise-valid allow, a VPC Service Controls perimeter that blocked the call from the wrong network, a service account that can’t be impersonated, or simply a disabled API.

This article is the decision tree that always finds it. We treat PERMISSION_DENIED not as one bug but as a symptom class with a fan-out of root causes, each confirmed by a specific gcloud command or Console path. You will learn to read the error like an X-ray — pulling the principal (who), the permission (the verb, like storage.objects.get), and the resource (the noun) straight out of it — then walk Google Cloud’s evaluation order in the exact sequence the platform uses: org-policy gate, VPC-SC perimeter, deny policies, then allow policies inherited down the hierarchy with conditions applied along the way. Localise the failure to one stage and the fix is obvious. Guess, and you burn an hour granting roles/owner to a service account that a deny policy was blocking the whole time. By the end you will know — in under two minutes, using gcloud policy-troubleshoot iam, Policy Analyzer, the Console Troubleshooter, and audit logs — whether you face a missing role, a wrong-level grant, a false condition, an org-policy or deny override, a perimeter block, a broken impersonation chain, or a disabled API. Keep the prose for the mental model; keep the tables open at 02:00.

What problem this solves

Google Cloud hides a remarkable amount of machinery behind a one-line error so it can stay secure. A detailed “you’re missing bigquery.tables.getData because the binding on dataset X expired” message handed to an under-privileged caller would itself be a leak — it tells an attacker exactly what to ask for. So the information you need is real and captured (in audit logs, the policy, the org-policy and VPC-SC config), but deliberately not in the red box — and if you don’t know which stage to inspect, you thrash.

What breaks without this knowledge is predictable and expensive. An engineer sees PERMISSION_DENIED and reaches for the biggest hammer — roles/editor or roles/owner. Sometimes it “works,” teaching the worst lesson: now you have an over-privileged principal and the real cause (a deny policy or perimeter) is untouched and will bite again. Other times it doesn’t work — because a deny or org policy sits above IAM allow — and the engineer concludes “IAM is broken” when it is refusing an action a higher gate already forbade. Either way: hours lost, an audit finding created, root cause untouched. It bites hardest on service accounts, multi-project / shared-VPC setups, hierarchy confusion, and anything inside a VPC Service Controls perimeter. The fix is almost never “grant more” — it is “find the stage saying no and make that stage say yes, at the smallest scope.”

To frame the field, every cause class, the question it forces, and the one place to look first:

Learning objectives

By the end of this article you can:

• Read a GCP PERMISSION_DENIED / 403 error and extract the three things that matter — the principal, the exact permission (service.resource.verb), and the resource — from the message and the audit log.
• Walk Google Cloud’s authorization evaluation order in sequence — org policy → VPC Service Controls → deny policy → allow policy (with conditions, inherited down the hierarchy) — and localise any 403 to the single stage that refused.
• Distinguish an access denial (IAM: you lack a permission) from a configuration denial (org policy: the platform forbids the action regardless of permission) — the most common source of wasted hours.
• Find the exact missing permission and the smallest predefined role that contains it, instead of reaching for basic roles like roles/editor or roles/owner.
• Diagnose service-account failures: missing roles on the runtime SA, broken impersonation chains (roles/iam.serviceAccountTokenCreator), and Workload Identity Federation misconfiguration for keyless CI/CD.
• Confirm root cause fast with the real tools — gcloud policy-troubleshoot iam, Policy Analyzer, the Console Troubleshooter, deny-policy and org-policy listings, and Cloud Audit Logs.
• Apply each fix with both gcloud and Terraform, at the smallest scope that resolves it, and verify the grant actually took effect.

Prerequisites & where this fits

You should already understand the IAM basics: a policy binding is a triple of (member, role, optional condition) on a resource; a role is a named bundle of fine-grained permissions (each shaped service.resource.verb); and resources live in a hierarchy — Organization → Folder → Project → resource — with allow policies inheriting downward. You should be comfortable running gcloud in Cloud Shell and switching the active project with gcloud config set project. If those triples and that hierarchy are fuzzy, read GCP IAM Explained: Members, Roles & Bindings, Simply first.

This sits in the Observability & Troubleshooting track, alongside the broader GCP Troubleshooting Methodology: IAM, VPC, Compute, GKE. It assumes the depth of the GCP IAM Deep Dive: Roles, Policies, Conditions & the Recommender and the hierarchy mechanics from GCP Resource Hierarchy Explained: Org, Folders, Projects. When the cause is a higher gate, lean on GCP Resource Hierarchy & Org-Policy Guardrails, GCP VPC Service Controls: Perimeters & Exfiltration Prevention, and GCP IAM Deny Policies, Conditions & Impersonation Chains.

A quick map of who confirms what during an incident, so you escalate fast:

Core concepts

Six mental models make every later diagnosis obvious — the playbook is an application of them.

1. The error names the verb, not the fix. Every GCP permission is shaped service.resource.verb — storage.objects.get, iam.serviceAccounts.actAs. A PERMISSION_DENIED almost always names the exact permission the principal lacked — the most valuable clue you have, because it tells you which predefined role to look for (the one that contains it) instead of carpet-bombing with roles/editor. Don’t ask “what role should I give?”; ask “what permission was denied, and what’s the smallest role that holds it?”

2. Authorization is a pipeline, and IAM is only one stage. A request does not just “check IAM.” Google Cloud evaluates several gates in order — org policy, then VPC Service Controls, then deny policy, then the allow policy (with conditions, inherited down the hierarchy) — and any can say no (full sequence in the pipeline table below). Not knowing this order, you’ll inspect IAM bindings for a problem that lives a gate higher, where no amount of role-granting fixes it. “Deny and the gates above it win” is the single biggest unlock in GCP troubleshooting.

3. Allow inherits down; deny and org policy flow down too. A grant at the organization applies to every folder, project, and resource beneath it — so “I don’t see the binding on the project” doesn’t mean the principal lacks access; it might be granted three levels up. The corollary that trips everyone: when access is too broad, the cause is often an inherited grant high in the tree; when access is missing, the fix often belongs at a lower scope than where you first looked.

4. The identity making the call is rarely the one you assume. “Permission denied” frequently means the wrong principal made the request. A workload on Compute Engine or GKE runs as a service account — if you didn’t attach one, the default SA, with different permissions than the human who deployed it. Step one of any 403: confirm which principal actually made this call from authenticationInfo.principalEmail in the audit log, not from assumption.

5. Conditions turn one binding into “sometimes.” A binding can carry a Condition — a CEL expression over request attributes (time, resource name, IP, tag) — granting the role only when true. So a binding can name the right member and role and still not apply. When get-iam-policy shows the binding but access is denied, read the condition field first.

6. Org policy denies configurations; IAM denies actions. This distinction saves more hours than any other. IAM answers “may this principal perform this action?” — access. Org policy answers “is this configuration allowed here at all?” — governance, enforced regardless of your roles, so roles/owner cannot defeat it. When an owner gets PERMISSION_DENIED creating an SA key, suspect org policy — the text hints it, saying the action violates constraint X rather than “you don’t have permission Y.”

The vocabulary in one table

Before the deep sections, pin down every moving part — the mental model side by side:

How to read the error (the X-ray skill)

Every fast diagnosis starts with extracting three facts from the error — who (principal), what verb (permission), on what (resource). The same denial across surfaces, and what to pull from each:

Two reading notes that save the most time. First, gcloud’s appended “or it may not exist” is not noise: GCP sometimes returns 404 NOT_FOUND when you lack permission to even know a resource exists — a “not found” can be a permission problem in disguise. Second, the audit log is the source of truth for the principal: the Console shows you, but the failing call may have run as a service account or a forgotten gcloud config — always confirm authenticationInfo.principalEmail.

The diagnostic toolkit: what each tool answers

You reach for a small, fixed set of tools — knowing which one answers which question is half the speed:

The authorization pipeline and the decision tree

This is the spine of the whole method. A request is checked by a sequence of gates, and the first one that says no produces your error — so diagnose top to bottom, because checking IAM bindings is pointless if an org policy or perimeter above already refused. The full pipeline — what each stage decides, its denial fingerprint, and how to confirm it:

The crucial mental model: stages 0–4 can refuse before IAM (stage 5) is even consulted, and stages 1–4 cannot be overcome by granting more roles. This is why an owner can be denied — stage 5 would pass, but an org policy (1), perimeter (2), or deny policy (4) refuses first. So: work the pipeline top-down, and grant roles only once you’ve ruled out everything above stage 5.

Now the decision tree — the centrepiece. Walk it top to bottom: the first question that answers YES is your stage — stop, confirm, fix (if every branch fails to the bottom, it’s a plain allow-policy miss).

Reading the resource hierarchy correctly

Half of all “I granted the role and it still doesn’t work” cases are hierarchy mistakes: the grant is real, but on the wrong resource. Allow policies inherit downward, so a resource’s effective permissions are the union of its bindings plus every ancestor’s. The classic errors: granting on the wrong project, granting on the project when the resource is governed at the folder, or forgetting resource-level policies (per-bucket, per-dataset IAM) exist. Reading and unioning four levels of policy in your head is where mistakes creep in — prefer policy-troubleshoot iam, which names the deciding binding, its role, level, and any condition or deny. The decision rule for where to grant:

Finding the exact missing permission and the smallest role

Once you’ve ruled out the higher gates and confirmed a genuine allow-policy miss (stage 5), the discipline is: find the exact permission denied, then grant the smallest predefined role that contains it — never a basic role as a reflex.

The error usually hands you the permission; when a Console denial hides it, the Troubleshooter (“Why don’t I have access?”) names it. To find which roles contain a permission, filter IAM → Roles by it or run gcloud iam roles describe roles/NAME --format='value(includedPermissions)'. Confirm a role holds the permission before granting it, then grant at the smallest scope — the bucket, not project-wide objectAdmin (the lab below walks this end to end). On role type: basic roles (owner/editor/viewer) are thousands of permissions project-wide — almost never right in prod, an instant audit/CIS finding; predefined roles are the curated default; custom roles are for when no predefined role is tight enough. A trap worth its own table — these pairs look interchangeable, but the left is the over-grant people reach for and the right is what was actually required:

Service accounts and impersonation: the number-one source

More 403s trace to service accounts than to anything else, because the non-human identity that runs the workload is an afterthought. Three failure shapes, routinely confused:

Shape 1 — the runtime SA is under-privileged. A Cloud Run service, GKE workload, Compute VM, or Cloud Function runs as a service account; if you didn’t specify one, the default SA, with broad legacy permissions or very few. The workload’s permissions matter at call time, not the deploying human’s. Fix: attach a dedicated SA with minimal roles (GCP IAM Service Accounts & Least Privilege).

Shape 2 — the impersonation chain is broken. gcloud --impersonate-service-account=, Terraform’s impersonate_service_account, and many CI flows have one principal mint a short-lived token as a service account, which requires the caller to hold roles/iam.serviceAccountTokenCreator on the target SA. Miss it and you get PERMISSION_DENIED on getAccessToken or actAs. The classic confusion is granting it on the wrong SA or project-wide. Confirm with gcloud iam service-accounts get-iam-policy TARGET_SA; grant on that SA — in Terraform, a google_service_account_iam_member whose member can even be a WIF principalSet://…:

resource "google_service_account_iam_member" "ci_can_impersonate" {
  service_account_id = google_service_account.deploy_sa.name
  role               = "roles/iam.serviceAccountTokenCreator"
  member             = "principalSet://iam.googleapis.com/${var.pool}/attribute.repository/my-org/my-repo"
}

Shape 3 — Workload Identity Federation isn’t wired correctly. For keyless CI/CD, WIF maps an external identity to a GCP SA with no downloaded key. If the attribute mapping or condition doesn’t match the incoming token’s claims, the federated principal never resolves to anything that can actAs the SA — a denial that looks like a missing role but is a mapping mismatch. Depth in GCP Workload Identity Federation: Keyless CI/CD.

The actAs-adjacent roles are constantly mixed up — the table to keep:

The memorable tell of all these: works locally, fails in the cloud — locally the call runs as your credentials, in the cloud as the SA, so compare gcloud auth list against the runtime SA.

When IAM is innocent: org policy, deny, and VPC-SC

These three stages refuse before IAM and cannot be fixed by granting roles — recognising them stops the over-granting.

Org policy enforces configuration rules. The error says not allowed or violates constraint X, and denies owners just the same. The most commonly-hit constraints:

gcloud org-policies list --project=PROJECT_ID   # effective policies on the project
gcloud org-policies describe iam.disableServiceAccountKeyCreation \
  --project=PROJECT_ID --effective              # inspect the suspected constraint

Deny policies are explicit denies attached to the org, folder, or project; a deny beats any allow and is evaluated first — so even roles/owner is blocked if a deny rule matches their permission and they aren’t in its exception list. List them with gcloud iam policies list-deny --attachment-point="cloudresourcemanager.googleapis.com/projects/PROJECT_ID" and read the deniedPrincipals, deniedPermissions, and exceptionPrincipals.

VPC Service Controls wraps APIs in a network perimeter to stop data exfiltration; a call from outside the perimeter to a protected service is blocked regardless of IAM. The tell is the literal VPC Service Controls or RESOURCES_NOT_IN_SAME_SERVICE_PERIMETER plus a unique violation id; find the detail with gcloud logging read 'protoPayload.status.details.violations.type="VPC_SERVICE_CONTROLS"' --limit=5. The violation reason tells you which perimeter rule to reach for — they are not interchangeable:

For all four stages the wrong move is identical — granting more roles does nothing — and the fix-owner is usually security.

Conditions: the binding that grants “sometimes”

A binding can carry an IAM Condition — a CEL expression that must be true for the grant to apply. It shows up in get-iam-policy looking correct, names the right member and role, and still denies access because the request didn’t satisfy the condition — one of the sneakiest 403s precisely because the binding looks present. Read it with gcloud projects get-iam-policy PROJECT_ID --format=json | jq '.bindings[] | select(.condition != null)'. The attributes a condition can test, and the denial each produces:

# A time-bound grant in Terraform — note: when request.time passes the expiry, access is denied
resource "google_project_iam_member" "temp_break_glass" {
  project = "my-project-id"
  role    = "roles/compute.admin"
  member  = "user:oncall@example.com"
  condition {
    title       = "expires-2026-07-01"
    description  = "Break-glass access, auto-expires"
    expression   = "request.time < timestamp(\"2026-07-01T00:00:00Z\")"
  }
}

Architecture at a glance

The diagram traces an API call the way Google Cloud actually authorizes it, then maps each failure class to the gate where it bites. Read it left to right. A request leaves a caller — a human via gcloud/Console, or a workload running as a service account — carrying a token, and arrives at the policy engine, which is not one check but a sequence: the org-policy and VPC-SC gates decide whether this configuration and network path are even permitted; then deny policies (an explicit deny wins outright); then the allow-policy evaluation walks the resource hierarchy — org, folder, project, resource — unioning every inherited binding and applying any conditions. Only if every gate passes does the call reach the target resource (a bucket, a Cloud Run service, a BigQuery dataset).

The numbered badges mark where each denial class strikes: blocks at the org-policy or perimeter gate (badges 1–2) happen before IAM is consulted, so granting roles there is futile; a deny-policy veto (badge 3) overrides any allow; a missing or wrong-level grant (badge 4) is the classic IAM miss; and a broken impersonation chain (badge 5) fails when the caller can’t even mint the token to become the SA. The whole method is in the picture: localise the 403 to the leftmost gate that refuses, confirm with that gate’s command, fix it there.

Real-world scenario

Nimbus Logistics runs a parcel-tracking platform on Google Cloud: a Cloud Run API in project nimbus-app-prod, reading and writing a Cloud Storage bucket (gs://nimbus-events-prod) that lives in a separate, security-owned project nimbus-data-prod. The data project sits inside a VPC Service Controls perimeter to satisfy a customer contract about data exfiltration. The platform team is five engineers; the security team owns the org policies and the perimeter. Monthly GCP spend is about ₹220,000.

The incident began after a routine change: security rolled the Cloud Run service onto a new dedicated runtime SA, tracker-run@nimbus-app-prod.iam.gserviceaccount.com, replacing the old default Compute SA. Within minutes the API returned 500s: PERMISSION_DENIED: Permission 'storage.objects.create' denied on resource (or it may not exist). The on-call engineer’s first instinct was textbook-wrong — grant roles/storage.admin at the project level. It didn’t help, so he escalated to storage.admin at the org level — an alarming over-grant — and still the writes failed. An hour gone, and the over-grant had tripped a security alert of its own.

The breakthrough came from walking the pipeline top-down. The audit-log query showed the denial carried a VPC Service Controls violation with a unique id — not a plain IAM miss. The old default SA had worked because, by historical accident, it was referenced in an ingress rule on the data project’s perimeter; the new tracker-run@ SA was not, so every call from nimbus-app-prod into the protected bucket was being blocked at stage 2, before IAM was ever consulted. No storage.admin grant at any level could have worked, because the perimeter refuses the request regardless of roles.

The real fix was small and belonged to security: add the new SA to the perimeter’s ingress policy for the Storage API, and grant it exactly roles/storage.objectCreator on the bucket. The org-level over-grant was reverted the same afternoon and writes resumed immediately. The retro put one line on the wall: “If an owner-level grant doesn’t fix it, the problem isn’t IAM — read the audit log before you grant anything.”

The incident as a timeline, because the order of moves is the lesson:

Advantages and disadvantages

GCP’s layered, deny-first, hierarchy-inheriting model is what causes the confusion here — and what makes a 403 diagnosable once you know the pipeline. Weigh it honestly:

Every one of these disadvantages is manageable — but only once you know the gates exist and diagnose them in order. It bites hardest on multi-project / shared-VPC architectures, anything inside a perimeter, and teams on default service accounts.

Hands-on lab

Reproduce a service-account PERMISSION_DENIED, find the exact missing permission with the Troubleshooter, and fix it with the smallest role — free-tier-friendly (a single project, no paid resources). Run in Cloud Shell.

Step 1 — Set up. Confirm the active project (operating on the wrong one is the most common silent failure), then create a roleless service account and a bucket.

export PROJECT_ID="$(gcloud config get-value project)"   # verify this is right!
gcloud iam service-accounts create lab-uploader --display-name="Lab uploader"
export SA="lab-uploader@${PROJECT_ID}.iam.gserviceaccount.com"
export BUCKET="gs://${PROJECT_ID}-iam-lab"
gcloud storage buckets create "$BUCKET" --location=asia-south1

Step 2 — Reproduce the denial. Grant yourself Token Creator to impersonate the SA, then upload as the SA — which has no storage role, so it fails.

gcloud iam service-accounts add-iam-policy-binding "$SA" \
  --member="user:$(gcloud config get-value account)" \
  --role="roles/iam.serviceAccountTokenCreator"
echo "hello" > /tmp/iam-lab.txt
gcloud storage cp /tmp/iam-lab.txt "$BUCKET/iam-lab.txt" --impersonate-service-account="$SA"
# Expected: ERROR ... PERMISSION_DENIED: ... 'storage.objects.create' denied ...

The error names the exact permission — storage.objects.create — which is your whole diagnosis.

Step 3 — Confirm with the Troubleshooter (the “why”).

gcloud policy-troubleshoot iam \
  "//storage.googleapis.com/projects/_/buckets/${PROJECT_ID}-iam-lab" \
  --principal-email="$SA" --permission="storage.objects.create"
# Expected: access = NOT_GRANTED, no binding found that contains it

Step 4 — Grant the smallest role and verify. roles/storage.objectCreator holds storage.objects.create (confirm with gcloud iam roles describe), so grant that, not storage.admin — then the upload succeeds.

gcloud storage buckets add-iam-policy-binding "$BUCKET" \
  --member="serviceAccount:$SA" --role="roles/storage.objectCreator"
gcloud storage cp /tmp/iam-lab.txt "$BUCKET/iam-lab.txt" --impersonate-service-account="$SA"
# Expected: Copying file ... done.

Step 5 — Teardown.

gcloud storage rm -r "$BUCKET"
gcloud iam service-accounts delete "$SA" --quiet && rm -f /tmp/iam-lab.txt

You reproduced the most common 403 (under-privileged SA), read the permission straight from the error, confirmed it with the Troubleshooter, and fixed it with the smallest role — the whole method, end to end.

Common mistakes & troubleshooting

This is the playbook — the centerpiece. Each row is a real failure mode: symptom, root cause, the exact command or Console path that confirms it, and the fix. Scan for your symptom, confirm before you act, fix at the smallest scope. The rows span stage 0 to stage 5.

Three moves apply to any row above: see who actually called (gcloud logging read 'protoPayload.authorizationInfo.granted=false' --limit=5); learn why a permission is or isn’t granted (gcloud policy-troubleshoot iam …); and check for a higher gate (gcloud org-policies list, gcloud iam policies list-deny …).

Error & reason-code reference

The denial shows up with a small set of status strings and reason codes; map the code to the stage and the fix. The non-obvious ones are the 403 sub-reasons in the JSON details.

When the surface error is too terse, the audit-log entry is the truth, field by field: status.code = 7 confirms PERMISSION_DENIED; authenticationInfo.principalEmail is the actual principal; authorizationInfo[].permission + granted:false is the denied permission; resourceName is the resource; and a status.details[].violations[].type = VPC_SERVICE_CONTROLS (with violationReason) means a perimeter block at stage 2, not an IAM miss.

Best practices

Crisp, production-grade rules from the pipeline:

• Diagnose top-down, fix bottom-up. Walk the pipeline org-policy → VPC-SC → deny → allow when finding the cause; grant at the smallest scope when fixing it.
• Read the audit log before you grant anything. authenticationInfo.principalEmail and authorizationInfo[].permission are the truth; the red box on screen is a summary.
• Grant the smallest predefined role that holds the denied permission. Never reach for roles/owner/editor/viewer to “unblock” — that creates the next audit finding.
• If an owner is denied, it isn’t IAM. Owners have every IAM permission, so an owner-denial means org policy, deny policy, or VPC-SC is refusing above IAM.
• Give every workload a dedicated SA with minimal roles, and use impersonation / Workload Identity Federation instead of keys. Default SAs are a smell; keyless also sidesteps iam.disableServiceAccountKeyCreation.
• Grant at the right level deliberately. Resource or folder for least privilege and least drift; the organization only for genuinely org-wide roles.
• Make conditional grants legible. Give conditions clear titles and expiries; a silent request.time lapse looks identical to a missing binding.
• Manage IAM as code (Terraform), reviewed, and run the Recommender. A bad binding caught in review never becomes a 02:00 incident; the Recommender surfaces over-grants to trim.
• Document your perimeters and deny policies. The app team must know a perimeter or deny exists, or they’ll chase IAM for hours.

Security notes

Troubleshooting 403s is, paradoxically, where most over-privilege is created — under incident pressure someone grants owner “just to unblock” and never reverts it. Hold the line when it’s hardest:

• Least privilege is the whole game. The denied-permission string tells you the minimum role to grant; granting more is the security regression, not the fix. Prefer resource- over project- over folder/org-scoped bindings.
• Guardrails are not obstacles — comply, don’t weaken. When a deny/org policy blocks you, the right response is usually to comply (signed URLs, WIF, an allowed region), not to weaken it. Loosening publicAccessPrevention or a domain restriction to “fix” a 403 can open an exfiltration path; likewise, adding a project/identity to a perimeter expands what can reach protected data — do it narrowly (specific identity + service), with security. See GCP VPC Service Controls: Perimeters & Exfiltration Prevention.
• Avoid service-account keys. Each downloaded key is a long-lived credential that can leak — use impersonation and Workload Identity Federation, and store anything you must keep in Secret Manager (GCP Cloud KMS, CMEK & Envelope Encryption).
• Audit every emergency grant, and separate duties. Pair break-glass grants with a time-bound condition so over-access auto-expires; the app team owns workload IAM while security owns org policy, deny policies, and perimeters — mixing the two is how a 403 fix disables a guardrail org-wide.

Cost & sizing

IAM itself is free — no charge for policies, bindings, roles, conditions, deny policies, org policies, or perimeters. What this topic costs you is indirect:

The single biggest “cost” of GCP IAM is engineer time spent diagnosing the wrong stage — exactly what the pipeline-ordered method eliminates. The one line item that can run up a bill is Data Access audit logs (needed to see denials in detail; Admin Activity logs, recording policy changes, are on by default and free) — enable them where you need visibility and set retention deliberately. Pair with GCP Cloud Monitoring & Logging: The Operations Suite Deep Dive to alert on granted=false spikes before users report them.

Interview & exam questions

Model answers are 2–4 sentences. These map to the Associate Cloud Engineer (ACE) and Professional Cloud Architect (PCA) exams, which test IAM troubleshooting heavily.

1. A user with roles/owner gets PERMISSION_DENIED creating a service-account key. Why, and how do you fix it? Owners hold every IAM permission, so this isn’t an allow-policy problem — it’s an org policy (iam.disableServiceAccountKeyCreation) refusing the configuration regardless of role. Use impersonation or Workload Identity Federation instead of keys, or (with security sign-off) scope an exception.

2. You granted a service account roles/storage.objectViewer on the project, but it still can’t read a specific bucket. What’s the first thing to check? Whether a higher gate is refusing before IAM — most commonly a VPC Service Controls perimeter (a VPC_SERVICE_CONTROLS violation in the audit log), which no IAM grant fixes. Otherwise verify the binding actually applies (right project, no false condition).

3. What is the difference between roles/iam.serviceAccountUser and roles/iam.serviceAccountTokenCreator? serviceAccountUser lets a principal deploy a workload that runs as the SA (actAs); serviceAccountTokenCreator lets it mint tokens / impersonate the SA. Deploying Cloud Run as an SA needs the former; gcloud --impersonate-service-account needs the latter, on the target SA.

4. In what order does Google Cloud evaluate authorization, and why does the order matter? API enablement → org policy → VPC Service Controls → deny policy → allow policy with conditions (inherited down the hierarchy). The first gate to refuse produces the error, and gates above the allow policy can’t be overcome by granting roles — so you diagnose top-down or grant roles that change nothing.

5. A binding clearly grants the right member the right role, but access is still denied. Name two likely reasons. Either an IAM Condition on the binding evaluated false (wrong time/IP/resource), or an explicit deny policy on the resource or an ancestor is overriding the allow. Read the binding’s condition.expression and run policy-troubleshoot iam, which surfaces a matched deny.

6. How do you find the exact permission a failed Console action requires when the error doesn’t show it? Use the IAM Policy Troubleshooter (Console, or gcloud policy-troubleshoot iam) against the resource and the signed-in principal — it reports the deciding permission and binding. The Console’s “Why don’t I have access?” prompt does the same for the current page.

7. A workload works on your laptop but returns 403 in Cloud Run. What changed? The identity: locally it ran as your credentials (ADC), but in Cloud Run as the service’s runtime service account, with different permissions. Grant the runtime SA the minimal roles the code needs.

8. What does “explicit deny beats allow” mean in practice for troubleshooting? Even a correct allow binding — up to roles/owner — is overridden if a deny policy matches the principal and permission and they aren’t in its exception list. So when an allow looks right but access fails, list deny policies on the resource and its ancestors first.

9. Why might a 404 NOT_FOUND actually be a permission problem? GCP sometimes hides a resource’s existence from principals who can’t see it, to prevent enumeration — so “not found” can mean “you can’t see it,” not “it’s gone.” Troubleshoot a read permission on the resource; if that’s the miss, grant a viewer role.

10. How does Workload Identity Federation cause 403s, and how do you diagnose it? If the pool/provider’s attribute mapping or condition doesn’t match the external token’s claims, the federated identity never resolves to a principal that can actAs the SA. Compare the token’s claims to the provider config, and ensure the mapped principal has roles/iam.workloadIdentityUser on the SA.

11. An afternoon-only access pattern stops working overnight with no change deployed. What’s the likely cause? A time-bound IAM Condition (a request.time expression) whose window has passed, or a temporary/break-glass grant that lapsed. Read the binding’s condition and extend the window or re-grant with a fresh expiry, ideally as code.

Quick check

1. A principal with roles/owner is denied an action. Is the cause more likely IAM allow-policy, or something else? Why?
2. You see RESOURCES_NOT_IN_SAME_SERVICE_PERIMETER. Which pipeline stage refused, and will granting an IAM role fix it?
3. Name the role you grant on the target service account so a caller can impersonate it.
4. A binding shows the correct member and role in get-iam-policy, but access is still denied. What single field do you read next?
5. Which command answers “does this principal have this permission on this resource, and via which binding?”

Answers

1. Something else (org policy, deny policy, or VPC-SC), not IAM allow. An owner already holds every IAM permission, so the allow stage would pass — the refusal must come from a gate above IAM, which roles can’t defeat.
2. VPC Service Controls (stage 2). No IAM grant will fix it; you must add the project/identity to the perimeter or create an ingress/egress rule.
3. roles/iam.serviceAccountTokenCreator, granted on the target SA (for keyless external identities via WIF it’s roles/iam.workloadIdentityUser).
4. The binding’s condition (its condition.expression) — a conditional binding that evaluated false looks exactly like a missing grant.
5. gcloud policy-troubleshoot iam RESOURCE --principal-email=… --permission=… (the Policy Troubleshooter).

Glossary

• Principal (member): the identity making a request (user, group, service account, federated) — found in authenticationInfo.principalEmail.
• Permission: a single fine-grained action shaped service.resource.verb (e.g. storage.objects.get); the error usually names the one you lack.
• Role: a named bundle of permissions — basic (owner/editor/viewer), predefined (per-service), or custom.
• Binding: a triple of (member, role, optional condition) in a resource’s allow policy; the unit of access.
• Allow policy: a resource’s bindings; effective access is the union of it plus every ancestor’s (inheritance).
• Deny policy: explicit deny rules that override any allow; evaluated before the allow policy.
• IAM Condition: a CEL expression on a binding (time, resource name, IP, tag) that must be true for the grant to apply.
• Org policy (constraint): a governance rule that permits/forbids configurations regardless of IAM roles.
• VPC Service Controls: a service perimeter around APIs that blocks calls crossing the boundary, before IAM is consulted.
• Service account (SA): a non-human identity workloads run as; the runtime identity whose permissions matter at call time.
• Impersonation: one principal minting short-lived credentials as an SA, needing serviceAccountTokenCreator on the target.
• Workload Identity Federation (WIF): mapping an external identity (e.g. a CI provider) to a GCP SA without a key.
• Policy Troubleshooter: the tool (gcloud policy-troubleshoot iam / Console) that explains whether and why a principal has a permission.

Next steps

• Master the foundations these diagnoses rest on with GCP IAM Explained: Members, Roles & Bindings, Simply and the GCP IAM Deep Dive: Roles, Policies, Conditions & the Recommender.
• Get the hierarchy right at design time with GCP Resource Hierarchy Explained: Org, Folders, Projects and the guardrails in GCP Resource Hierarchy & Org-Policy Guardrails.
• Go deeper on the gates above IAM: GCP IAM Deny Policies, Conditions & Impersonation Chains and GCP VPC Service Controls: Perimeters & Exfiltration Prevention.
• Kill service-account 403s for good with GCP IAM Service Accounts & Least Privilege and keyless CI via GCP Workload Identity Federation: Keyless CI/CD.
• Broaden the diagnostic muscle beyond IAM with the GCP Troubleshooting Methodology: IAM, VPC, Compute, GKE.