AWS KMS Hands-On: Keys, Key Policies & Envelope Encryption

“Encryption at rest” looks like a checkbox until the day it 403s you. You have s3:GetObject, you have an IAM policy an auditor would sign off on, and the download still fails — because the object is SSE-KMS and the key never named your principal. Or a batch job that ran for a year suddenly can’t decrypt anything after a teammate “cleaned up an unused key,” and there is no undo. Both are the same lesson: the moment you use a customer-managed key for anything — S3, EBS, RDS, Secrets Manager, or your own application data — AWS KMS becomes a second, independent authorization system with its own root of trust, its own policy language, and its own failure modes. This article gives you that system end to end, plus the hands-on muscle memory to wield it.

AWS Key Management Service (KMS) is a managed service that creates and controls cryptographic keys and performs cryptographic operations with them. The crucial design fact — the one that surprises people — is that for a symmetric key, the key material never leaves KMS in plaintext. You do not get the key bytes and encrypt locally with them; you either hand KMS ≤4 KB of data and get ciphertext back (Encrypt), or — the pattern that actually scales — you ask KMS for a data key you use locally and then throw away (GenerateDataKey). That second pattern is envelope encryption, and understanding it is the difference between an architecture that encrypts a 500 MB file cheaply and one that either can’t (the 4 KB wall) or hammers KMS into throttling.

You will build the whole model: the three key types (symmetric, asymmetric, HMAC) and when each fits; the three ownership tiers (AWS-owned, AWS-managed aws/service, and customer-managed CMKs) and why a CMK is worth the dollar a month; authorization — the key policy as the root of trust (and why an IAM allow does nothing without it), how IAM policies and grants layer on, and the sharp-edged conditions kms:ViaService and encryption context; envelope encryption step by step with openssl; rotation, aliases, multi-Region keys, and safe deletion with its 7–30 day window; CloudTrail as the audit trail of every key use; and the cost model. Then a hands-on lab you can run on a sandbox account and a troubleshooting playbook you keep open at 02:00 when AccessDeniedException is all you have. If you have already secured S3 with SSE-KMS, this is the layer underneath it — see Securing Amazon S3: Bucket Policies, Block Public Access & the 403 Playbook.

To frame the whole field before the deep dive, here is every KMS moving part this article covers, what it decides, and where it lives.

What problem this solves

Two failures, opposite in direction, same root cause: not knowing that the key is a separate authorization boundary. The first is self-inflicted denial — you can’t decrypt data you believe you own. Your IAM policy allows kms:Decrypt, the Policy Simulator says Allow, and you still get AccessDeniedException, because the key policy on that customer-managed key never named your principal and never delegated to IAM. Teams burn hours widening the IAM policy to kms:* — which changes nothing, because the denier is on the key, in a different console. The second is catastrophic loss or exposure — a key gets scheduled for deletion or disabled and every ciphertext under it becomes unreadable (KMS deliberately makes the key material unrecoverable), or a key policy is written so broadly (Principal: "*", no conditions) that any principal the account trusts can decrypt your most sensitive data.

What breaks without this knowledge is measured in incidents and in irreversibility. A deleted KMS key is the rare AWS action with no undo once the 7–30 day window elapses: the encrypted RDS snapshot, the S3 objects, the EBS volumes, the Secrets Manager secrets all become permanently opaque. On the access side, the cost is time and over-permissioning — the SSE-KMS 403 that hides behind an S3 error, the cross-account read that needs three doors open (bucket policy, IAM, and key policy), the service integration (EBS, Lambda, Aurora) that silently fails to launch because a grant couldn’t be created. Every one of these is precisely locatable once you hold the model.

Who hits this: everyone who turns on encryption with a customer-managed key, but hardest in cross-account designs (a shared key read by other accounts), regulated workloads (where a CMK’s audit trail and rotation are compliance requirements), high-throughput systems (where direct Encrypt/Decrypt per record throttles and envelope encryption plus data-key caching is mandatory), and anywhere a service integration quietly depends on a grant you didn’t know existed. The fix is almost never “make the key policy *” — it is “find the layer saying no (key policy, IAM delegation, grant, ViaService, or encryption context) and make that layer correct.”

Learning objectives

By the end of this article you can:

• Distinguish the three key types — symmetric (SYMMETRIC_DEFAULT), asymmetric (RSA/ECC for encrypt or sign), and HMAC — and choose the right key spec and key usage for a workload.
• Explain the three ownership tiers — AWS-owned, AWS-managed (aws/service), and customer-managed (CMK) — and justify when a CMK’s policy control, rotation and audit are worth ~$1/month.
• State the KMS authorization rule: the key policy is the root of trust, an IAM policy grants access only if the key policy delegates to IAM, and grants add temporary, scoped permission on top — then combine all three correctly for same-account and cross-account use.
• Use kms:ViaService and encryption context correctly, and predict the exact exception each causes when it doesn’t match.
• Implement envelope encryption with GenerateDataKey and openssl — encrypt a large file locally, discard the plaintext data key, store the encrypted data key — and say precisely why it beats direct Encrypt (the 4 KB limit, KMS traffic, cost).
• Configure automatic and on-demand rotation, aliases, and multi-Region keys, and explain what rotates and what stays the same.
• Disable vs schedule deletion of a key safely, and recover from the “pending deletion” state before the window closes.
• Read CloudTrail to see who used a key, and work any AccessDeniedException, IncorrectKeyException, InvalidCiphertextException or ThrottlingException to root cause with a playbook.

Prerequisites & where this fits

You should be comfortable reading an ARN and a JSON policy document, know what an IAM principal (user, role, or assumed-role session), an identity-based policy and a resource-based policy are, and be able to run the AWS CLI with a configured profile. You do not need cryptography theory — you need to know that a symmetric key encrypts and decrypts with the same secret, an asymmetric key pair splits into a private key (kept secret) and a public key (shareable), and an HMAC proves integrity and authenticity with a shared secret. If IAM evaluation itself is shaky, read Debugging IAM ‘Access Denied’: Policy Evaluation Logic, SCPs, Boundaries & a Playbook first — KMS authorization sits directly on top of that model, with the key policy as an extra, mandatory layer.

KMS is the encryption substrate under most of AWS. Almost every “enable encryption” checkbox in the console is really “pick a KMS key.” Knowing KMS makes all of those legible at once — the S3 SSE-KMS gate, the EBS volume key, the RDS snapshot key, the Secrets Manager and Parameter Store secret keys. Here is where KMS shows up and what permission the integration needs, so you recognise the pattern everywhere.

This maps directly onto certification objectives: KMS fundamentals, key policies and envelope encryption are core to SAA-C03 and DVA-C02, and the full data-protection picture — grants, ViaService, multi-Region keys, custom key stores and the audit trail — is central to SCS-C02 (Security Specialty).

Core concepts

Six mental models make every later diagnosis obvious.

A symmetric CMK’s key material never leaves KMS in plaintext — ever. You cannot export it. Every operation is an API call: Encrypt and Decrypt (KMS does the crypto on ≤4 KB), or GenerateDataKey (KMS mints a fresh data key, hands you a plaintext copy and an encrypted copy, and forgets it). This single fact is why envelope encryption exists and why KMS is auditable: every use is a logged request, not a copied secret.

Envelope encryption is “encrypt the data with a data key, encrypt the data key with the CMK.” For anything larger than 4 KB — or anything you do at volume — you ask the CMK for a data key, encrypt your payload locally with the plaintext copy, discard the plaintext copy, and store the encrypted data key next to the ciphertext. To read it back, you send the encrypted data key to Decrypt, recover the plaintext key, decrypt locally, and discard again. KMS only ever touches a 32-byte key, never your gigabytes.

The key policy is the root of trust; IAM is not enough on its own. Unlike most AWS resources, a KMS key requires a resource policy, and that policy is authoritative. An IAM policy granting kms:Decrypt works only if the key policy delegates to IAM — the default key policy does this with an “Enable IAM User Permissions” statement naming the account root. Remove that statement and no IAM policy in the account can touch the key; the key policy must name principals directly. This is the number-one KMS gotcha and the number-one KMS 403.

Grants are temporary, additive, programmatic permissions — the way services borrow your key. A grant delegates a specific set of operations (and optional encryption-context constraints) to a grantee principal, without editing the key policy. AWS services use grants constantly: when you launch an encrypted EBS volume, EC2 creates a grant so it can decrypt the volume on your behalf, then retires it. Grants only add access; they never deny.

Encryption context is non-secret additional authenticated data (AAD) bound to the ciphertext. It is a set of key–value pairs you pass on Encrypt/GenerateDataKey; KMS cryptographically binds it to the ciphertext and requires the identical context on decrypt. Get it wrong or omit it and you get InvalidCiphertextException. It is logged in CloudTrail (so never put secrets in it) and is the safest way to scope a grant or key-policy condition to, say, one tenant or one object.

A key is a stable reference (key ID/ARN); its material can change underneath it. Rotation swaps the backing cryptographic material but keeps the key ID, ARN, alias, and policy identical — nothing you reference changes, and old ciphertext still decrypts because KMS keeps the prior material. An alias is a second layer of indirection: a friendly, repointable name so your code says alias/app-data and you can swap the underlying key without a redeploy.

The vocabulary in one table

Pin down every moving part before the deep sections; the glossary repeats these for lookup.

Key types — symmetric, asymmetric, and HMAC

KMS keys come in three flavours, fixed at creation by two immutable properties: the key spec (the algorithm/size) and the key usage (what operation it is allowed to perform). You cannot change either after the key is created — a symmetric encrypt key can never sign, and an RSA sign key can never GenerateDataKey. Choose deliberately.

The key usage value is the second immutable choice; it determines which API calls the key accepts.

Symmetric keys — the default, and what you almost always want

A symmetric key (SYMMETRIC_DEFAULT) is a single 256-bit AES-GCM secret that both encrypts and decrypts. It never leaves KMS unencrypted, so all use is via API. It supports the operations that matter for data protection — Encrypt/Decrypt for ≤4 KB and, crucially, GenerateDataKey for envelope encryption of anything larger. Every AWS service integration (S3, EBS, RDS, Secrets Manager) uses symmetric keys. If you are not sure which type you need, you need this one.

Asymmetric keys — when you must share a public key

An asymmetric key is an RSA or ECC key pair: KMS holds the private key (which never leaves) and lets anyone download the public key with GetPublicKey. Use them only when a symmetric key genuinely can’t work: you need to let an external party encrypt data that only you can decrypt (they use the public key offline), or you need to sign data that others verify without calling AWS (they verify with the public key). The trade-offs are real — asymmetric operations are slower, cost more per request, there is no GenerateDataKey (so no native envelope encryption for large data), and the direct-encrypt payload limit is even smaller than symmetric (an RSA_2048 OAEP-SHA-256 key encrypts only ~190 bytes). Signing use cases include software-supply-chain signatures, JWT signing, and document signing.

HMAC keys — integrity and authenticity, not confidentiality

An HMAC key doesn’t encrypt anything. It computes a Message Authentication Code (GenerateMac) that proves a message wasn’t altered and came from someone holding the key, and verifies it (VerifyMac). Use it to sign API tokens, validate webhook payloads, or protect message integrity where both sides trust KMS. HMAC in KMS gives you a FIPS-validated MAC without the key material ever touching your code.

Key ownership — AWS-owned vs AWS-managed vs customer-managed

Every KMS key falls into one of three ownership tiers. The tier decides whether you can see the key, control its policy, rotate it, audit it, and how you pay. This is the single most important operational distinction after “symmetric vs asymmetric.”

AWS-owned keys are a shared pool AWS uses for default encryption (e.g., the default DynamoDB table key). You never see them, can’t audit them, and pay nothing — acceptable only when you have no compliance or access-control requirement.

AWS-managed keys appear in your account with an aws/<service> alias — aws/s3, aws/ebs, aws/rds, aws/secretsmanager, aws/ssm. AWS writes and owns their key policy (locked to the service via kms:ViaService), rotates them automatically once a year, and doesn’t charge the monthly fee (you pay only per request). You get CloudTrail visibility but no policy control — you can’t grant another account access, can’t restrict which principals use them, and can’t do cross-account. That last limitation is exactly why CMKs exist.

Customer-managed keys (CMKs) are the keys you create. You own the key policy (the entire point), choose the rotation behaviour, control cross-account sharing, and get full CloudTrail attribution — at ~$1/month per key plus per-request charges. The table below is the “why bother with a CMK” argument in one place.

The rule of thumb: use AWS-managed keys for low-stakes default encryption where you just want isolation and audit for free; reach for a CMK the moment you need policy control, cross-account sharing, a specific rotation posture, key deletion control, or an auditor asking “who can decrypt this, and prove it.”

Authorization — key policy, IAM, and grants

This is the depth anchor. A KMS request is authorized by evaluating up to three sources together: the key policy (always), any IAM policies on the caller (only if the key policy allows IAM to matter), and any grants on the key. Miss how they combine and you get the classic “my IAM policy is perfect and it still 403s.”

The key policy is the root of trust

Every KMS key has exactly one key policy — a resource policy you cannot remove (you can only replace it). It is authoritative: KMS starts every authorization decision here. The elements mirror an IAM policy, with KMS-specific traps.

The default key policy — what you get if you create a key without specifying one — contains the single most misunderstood statement in KMS: the root delegation.

{
  "Sid": "Enable IAM User Permissions",
  "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::111122223333:root" },
  "Action": "kms:*",
  "Resource": "*"
}

This does not mean “only the root user can use the key.” …:root here means “this account,” and the statement says: allow IAM policies in account 111122223333 to govern access to this key. With it present, an IAM policy that grants kms:Decrypt on the key ARN works normally. Remove it and no IAM policy can touch the key — you must name every principal directly in the key policy. That is a legitimate hardening move (it stops an over-broad IAM policy from reaching the key) but it is also how people accidentally lock themselves out. The Principal forms you’ll write:

How the three sources combine

Here is the rule that ends the confusion. For a same-account caller, access is allowed if the key policy grants it directly, or the key policy delegates to IAM and an IAM policy grants it, or a grant covers it — and no explicit Deny fires. For a cross-account caller, you need both sides: the key policy in the owner account must allow the external account/principal, and an IAM policy in the caller’s account must allow the call on that key ARN. The truth table:

Read it as: the key policy is the gate that must always open; IAM only helps when the key policy invited it (the root statement); grants are an independent additive path; and cross-account is always two-sided. This is why the S3 SSE-KMS cross-account read needs three things — the bucket policy, the caller’s IAM, and the key policy naming the external principal.

The CLI to read and set a key policy, and the same in Terraform:

# Read the current key policy (the 'default' policy is named "default")
aws kms get-key-policy --key-id alias/app-data --policy-name default \
  --query Policy --output text | jq .

# Replace it (e.g. delegate to IAM AND grant an app role Decrypt directly)
aws kms put-key-policy --key-id alias/app-data --policy-name default \
  --policy file://key-policy.json

resource "aws_kms_key" "app" {
  description             = "app-data envelope key"
  key_usage               = "ENCRYPT_DECRYPT"
  customer_master_key_spec = "SYMMETRIC_DEFAULT"
  enable_key_rotation     = true
  rotation_period_in_days = 365
  deletion_window_in_days = 30

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      { Sid = "EnableIAM", Effect = "Allow",
        Principal = { AWS = "arn:aws:iam::111122223333:root" },
        Action = "kms:*", Resource = "*" },
      { Sid = "AppRoleUse", Effect = "Allow",
        Principal = { AWS = aws_iam_role.app.arn },
        Action   = ["kms:Encrypt","kms:Decrypt","kms:GenerateDataKey*","kms:DescribeKey"],
        Resource = "*" }
    ]
  })
}

resource "aws_kms_alias" "app" {
  name          = "alias/app-data"
  target_key_id = aws_kms_key.app.key_id
}

Grants — temporary, scoped, service-friendly delegation

A grant is a way to delegate a specific set of KMS operations to a principal without editing the key policy — ideal for temporary or programmatic access, and the mechanism AWS services use to borrow your key. When you launch an encrypted EBS volume with a CMK, EC2 Auto Scaling (or the EBS service) calls CreateGrant so it can call Decrypt/GenerateDataKeyWithoutPlaintext on your behalf, then retires the grant when done. Grants are additive only (they never deny), take effect quickly (use the returned grant token to avoid eventual-consistency lag), and can be constrained by encryption context.

The three authorization mechanisms, side by side, so you pick the right tool:

# Grant a Lambda role permission to Decrypt, scoped to one tenant's context
aws kms create-grant --key-id alias/app-data \
  --grantee-principal arn:aws:iam::111122223333:role/report-fn \
  --operations Decrypt \
  --constraints EncryptionContextEquals={tenant=acme}
# → { "GrantId": "abcd12...", "GrantToken": "AQpAM..." }

resource "aws_kms_grant" "report_fn" {
  name              = "report-fn-decrypt-acme"
  key_id            = aws_kms_key.app.key_id
  grantee_principal = aws_iam_role.report_fn.arn
  operations        = ["Decrypt"]
  constraints { encryption_context_equals = { tenant = "acme" } }
}

kms:ViaService — restrict use to a service path

kms:ViaService is a condition key that limits key use to requests that arrive through a specific AWS service in a specific Region — for example s3.ap-south-1.amazonaws.com or ec2.ap-south-1.amazonaws.com. AWS-managed keys use it to lock themselves to their service; you use it on a CMK to say “this key may be used only when S3 (or EBS) is calling on a principal’s behalf — never by a person running aws kms decrypt directly.” Powerful, and a classic self-inflicted denial: add ViaService for S3 and your own direct aws kms decrypt from the CLI is denied, because that path isn’t S3.

{ "Sid": "OnlyViaS3InRegion", "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::111122223333:role/app" },
  "Action": ["kms:Decrypt","kms:GenerateDataKey"],
  "Resource": "*",
  "Condition": {
    "StringEquals": {
      "kms:ViaService": "s3.ap-south-1.amazonaws.com",
      "kms:CallerAccount": "111122223333" } } }

Encryption context — additional authenticated data that must match

Encryption context is a set of non-secret key–value pairs you pass on Encrypt, GenerateDataKey, and ReEncrypt. KMS folds it into the encryption as additional authenticated data (AAD) and requires the byte-identical context on Decrypt — same keys, same values (case-sensitive); pair order does not matter. It gives you three things: a tamper-evident binding (a ciphertext for tenant acme cannot be decrypted while claiming tenant beta), a grant/policy scoping hook (kms:EncryptionContext:tenant), and an audit breadcrumb (it appears in CloudTrail — which is exactly why it must never contain secrets).

Envelope encryption — GenerateDataKey vs direct Encrypt

Direct Encrypt sends your plaintext to KMS and gets ciphertext back — simple, but it caps at 4,096 bytes and turns every encryption into a KMS API call. Envelope encryption inverts that: you get a data key from the CMK, encrypt locally, and only the tiny data key ever travels to KMS. This is not an optional optimisation — it is how you encrypt anything bigger than 4 KB and how you keep KMS from throttling you at scale.

GenerateDataKey is the heart of it. One call returns three things:

The lifecycle, step by step — the mental model the diagram draws:

At scale, even one GenerateDataKey per object is too many — so you cache and reuse a data key across many objects with the AWS Encryption SDK and its data-key caching, trading a bounded reuse window for a massive drop in KMS traffic.

The one-screen version in shell — this is exactly what the lab automates:

# 1) Ask the CMK for a data key
DK=$(aws kms generate-data-key --key-id alias/app-data --key-spec AES_256 \
      --encryption-context purpose=lab,tenant=acme --output json)
echo "$DK" | jq -r .Plaintext     | base64 -d > /tmp/dk.bin      # plaintext data key (guard it)
echo "$DK" | jq -r .CiphertextBlob | base64 -d > /tmp/dk.enc      # encrypted data key (store it)

# 2) Encrypt a big file locally with the plaintext key, then 3) discard it
openssl enc -aes-256-cbc -pbkdf2 -in bigfile.dat -out bigfile.enc -pass file:/tmp/dk.bin
shred -u /tmp/dk.bin 2>/dev/null || rm -f /tmp/dk.bin              # DISCARD the plaintext key

Rotation, aliases, multi-Region keys, and deletion

These four lifecycle features share one theme: the key ID is a stable handle, and a lot can change underneath it without breaking anything that references it.

Key rotation — what rotates, what stays

Rotation replaces the key’s backing cryptographic material while keeping the key ID, ARN, alias, and policy identical. Because the identifiers don’t change, you re-encrypt nothing and update no references; KMS retains the older material so existing ciphertext still decrypts, and new encryptions use the newest material. KMS offers automatic rotation (a configurable period, default ~365 days, between 90 and 2560 days) and on-demand rotation (RotateKeyOnDemand, up to a limited number of times) for immediate rotation after, say, a suspected exposure.

Exactly what changes and what doesn’t — the source of most “did rotation break my app?” confusion (answer: no):

aws kms enable-key-rotation --key-id alias/app-data --rotation-period-in-days 365
aws kms rotate-key-on-demand --key-id alias/app-data          # rotate right now
aws kms get-key-rotation-status --key-id alias/app-data

Aliases — a repointable name

An alias is a friendly name (alias/app-data) that points to exactly one key ID and can be repointed to a different key later. Aliases are per-Region and per-account, the aws/ prefix is reserved for AWS-managed keys, and using an alias in your code decouples it from the physical key — you can migrate to a new key (or a multi-Region replica) by updating the alias, no redeploy. Note that aliases are Regional: alias/app-data in ap-south-1 is a different object from the same name in us-east-1.

Multi-Region keys — same key material in many Regions

A multi-Region key (MRK) is a set of interoperable keys — one primary and one or more replicas in other Regions — that share the same key material and the same key ID (with an mrk- prefix). Data encrypted by the primary in ap-south-1 can be decrypted by the replica in us-east-1 without any cross-Region call, which is what makes them right for disaster recovery, active-active global apps, DynamoDB global tables, and signing where you need the same key everywhere. Each replica is otherwise independent: its own key policy, grants, aliases, enable/disable state, and rotation.

# Create a primary MRK, then replicate it to another Region
PRIMARY=$(aws kms create-key --multi-region --description app-mrk \
  --query KeyMetadata.KeyId --output text --region ap-south-1)
aws kms replicate-key --key-id "$PRIMARY" --replica-region us-east-1 --region ap-south-1

resource "aws_kms_key" "primary" {
  description = "app-mrk"
  multi_region = true
  enable_key_rotation = true
}
resource "aws_kms_replica_key" "use1" {
  provider        = aws.us_east_1
  primary_key_arn = aws_kms_key.primary.arn
  description     = "app-mrk replica"
}

Disable vs delete — and the waiting period

There is no instant “delete a KMS key” button, by design — a deleted key makes every ciphertext under it permanently unrecoverable. So KMS forces a waiting period of 7 to 30 days (default 30) via ScheduleKeyDeletion; during it the key is in PendingDeletion and unusable, and you can still CancelKeyDeletion to recover. When you merely suspect a key is unused, disable it first: DisableKey makes it instantly unusable but instantly reversible, so you can watch CloudTrail for failed operations before committing to deletion.

The safe deprovisioning ritual: disable → wait and watch CloudTrail for AccessDenied/KMSInvalidStateException from real callers → if silent, schedule deletion with a 30-day window → cancel immediately if anything screams. The full set of key states you’ll see in describe-key:

CloudTrail — the audit trail of every key use

Because every KMS operation is an API call, CloudTrail records who used which key, when, from where, and with what encryption context — the reason a CMK satisfies “prove who can decrypt this” in an audit. KMS events are management events (logged by default in an org/account trail), so you get Decrypt, GenerateDataKey, CreateGrant, ScheduleKeyDeletion and the rest without enabling data events. For the deeper audit/compliance picture see AWS CloudTrail and Config: Audit and Compliance at Scale.

# Who called Decrypt on this key in the last events?
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=Decrypt \
  --query 'Events[].{who:Username,when:EventTime}' --output table

A CloudWatch metric filter that alarms on ScheduleKeyDeletion and PutKeyPolicy is one of the highest-value, lowest-cost KMS controls you can add — it catches an accidental (or malicious) key deletion or a policy that suddenly says Principal: "*" within minutes.

Architecture at a glance

The diagram traces envelope encryption left to right — the exact flow the lab performs. On the left, an application holds plaintext larger than the 4 KB direct-Encrypt limit (badge context: this is why we don’t just call Encrypt). It calls GenerateDataKey against the customer-managed key; that call is authorized by the key policy — the root of trust — and needs kms:GenerateDataKey granted there (badge 1), because an IAM allow alone does nothing unless the key policy delegates to IAM (badge 2). KMS returns a plaintext data key and its encrypted copy (the CiphertextBlob). The app encrypts the payload locally with the plaintext key and then discards it (badge 3) — the security-critical move — keeping only the encrypted data key (badge 4). The result is ciphertext with no size limit (badge 5), stored alongside the encrypted data key. To read it back, the app sends the encrypted data key to Decrypt with the same encryption context (badge 6) — mismatch it and you get InvalidCiphertextException — recovers the plaintext key, decrypts locally, and discards again. KMS never touches the payload, only the 32-byte data key, which is why this scales to gigabytes and dodges throttling.

Real-world scenario

Nordwind Logistics runs a 12-account AWS Organization. Their shipment-imaging pipeline stores ~2 TB/day of scanned customs documents in S3, encrypted with a customer-managed KMS key, and a separate analytics account reads a curated subset. Everything worked in staging. In the first week of production, three KMS failures land — each a textbook lesson.

Failure one — the pipeline throttles at the morning peak. The original design encrypted every scanned page by calling kms:Encrypt directly. At 200 pages/second across a fleet, KMS started returning ThrottlingException and the ingestion queue backed up. Two problems compounded: Encrypt is capped at 4 KB (some pages’ OCR JSON blobs exceeded it, throwing validation errors), and one KMS call per page blew through the Region’s cryptographic-operations request rate. The fix was to switch to envelope encryption with data-key caching: the workers now call GenerateDataKey once per batch via the AWS Encryption SDK, reuse the cached data key for a bounded window, and encrypt pages locally. KMS request volume dropped by two orders of magnitude, the throttling vanished, and the 4 KB wall became irrelevant because KMS only ever wrapped a 32-byte data key. Their runbook line: never call KMS per record — call it per data key, and cache the data key.

Failure two — the analytics account 403s cross-account. The analytics role had a clean IAM policy allowing kms:Decrypt on the key ARN, and the S3 bucket policy named the account. It still failed with AccessDeniedException. The trap was the third door: the key policy in the pipeline account had been hardened to remove the broad root-delegation statement, and it named only the pipeline’s own roles — never the external analytics principal. S3 cross-account SSE-KMS reads need all three: the bucket policy, the caller’s IAM, and the key policy naming the external role with kms:Decrypt. They added the analytics role to the key policy (scoped with an encryption-context condition on the dataset), and the read went green. The lesson they wrote down: an S3 403 that names a key ARN is a KMS-key-policy problem, and cross-account KMS is always two accounts’ policies plus the key policy.

Failure three — the “unused” key that wasn’t. A platform engineer, tidying up, saw a CMK with low console activity and ran schedule-key-deletion with the minimum 7-day window. Four days later a quarterly archival job — the key’s only heavy user — failed across the board with KMSInvalidStateException: key is pending deletion. Because the window hadn’t closed, they ran CancelKeyDeletion and recovered with zero data loss, but it was closer than anyone liked. The postmortem produced two standing controls: a CloudWatch alarm on ScheduleKeyDeletion and DisableKey for every production key, and a policy that keys are disabled and watched for 30 days before anyone schedules deletion. Had the window elapsed, ~9 months of encrypted archives would have been permanently unreadable — the one AWS mistake with no undo.

Advantages and disadvantages

KMS’s managed, policy-driven, audited model is what makes encryption usable at enterprise scale — and its irreversibility and layered authorization are what make it a skill. The trade-offs, then when each matters.

The layering matters most in multi-account, regulated estates, where a central security team must prove who can decrypt each dataset, rotate on a schedule, and share specific keys cross-account under tight conditions — a job AWS-managed keys can’t do and only CMKs with hand-written policies can. In a single-account app, most of this collapses to “make one CMK, keep the default key policy, turn on rotation, use an alias, and envelope-encrypt anything over 4 KB” — a fifteen-minute setup that still buys you audit, rotation and the deletion safety net.

Hands-on lab

You will create a customer-managed symmetric CMK with an alias and a scoped key policy, envelope-encrypt a file with GenerateDataKey + openssl, decrypt it with a matching encryption context, prove that a mismatched context fails, enable rotation, and tear down safely. Everything is free-tier-friendly except the CMK’s ~$1/month (prorated) and a few requests; teardown schedules the key for deletion at the end. Use a sandbox account. Region is ap-south-1 throughout — change it to yours. Here is what each step proves before you run it.

Step 0 — Confirm who you are. Half of all “impossible” KMS denials are “you’re not the principal you think.”

aws sts get-caller-identity
# Expected: { "Account": "111122223333", "Arn": "arn:aws:iam::111122223333:user/you" }

Step 1 — Create a customer-managed symmetric key and an alias.

KEY_ARN=$(aws kms create-key --description "kv-kms-lab" \
  --key-usage ENCRYPT_DECRYPT --key-spec SYMMETRIC_DEFAULT \
  --query KeyMetadata.Arn --output text --region ap-south-1)
echo "$KEY_ARN"
# Expected: arn:aws:kms:ap-south-1:111122223333:key/1234abcd-...

aws kms create-alias --alias-name alias/kv-kms-lab --target-key-id "$KEY_ARN" --region ap-south-1
aws kms describe-key --key-id alias/kv-kms-lab --region ap-south-1 \
  --query 'KeyMetadata.{State:KeyState,Spec:KeySpec,Usage:KeyUsage,MR:MultiRegion}'
# Expected: { "State":"Enabled","Spec":"SYMMETRIC_DEFAULT","Usage":"ENCRYPT_DECRYPT","MR":false }

Step 2 — Attach a scoped key policy. Keep the IAM delegation (so you can use the key) and add an explicit statement for an app role. Write it to a file and put it.

ME=$(aws sts get-caller-identity --query Arn --output text)
ACCT=$(aws sts get-caller-identity --query Account --output text)
cat > /tmp/key-policy.json <<JSON
{ "Version": "2012-10-17", "Statement": [
  { "Sid":"EnableIAM","Effect":"Allow",
    "Principal":{"AWS":"arn:aws:iam::${ACCT}:root"},
    "Action":"kms:*","Resource":"*" },
  { "Sid":"AdminAndUse","Effect":"Allow",
    "Principal":{"AWS":"${ME}"},
    "Action":["kms:Encrypt","kms:Decrypt","kms:GenerateDataKey*","kms:DescribeKey","kms:ScheduleKeyDeletion","kms:EnableKeyRotation"],
    "Resource":"*" }
]}
JSON
aws kms put-key-policy --key-id alias/kv-kms-lab --policy-name default \
  --policy file:///tmp/key-policy.json --region ap-south-1

Step 3 — Envelope-encrypt a file. Make a payload bigger than the 4 KB direct-Encrypt limit, get a data key, encrypt locally, and discard the plaintext key.

# A 1 MB payload — far over the 4 KB Encrypt limit, so envelope encryption is required
head -c 1048576 /dev/urandom > /tmp/payload.bin

# 1) Ask the CMK for a 256-bit data key WITH an encryption context
DK=$(aws kms generate-data-key --key-id alias/kv-kms-lab --key-spec AES_256 \
  --encryption-context purpose=lab,tenant=acme --region ap-south-1 --output json)

echo "$DK" | jq -r .Plaintext     | base64 --decode > /tmp/dk.plain   # plaintext data key
echo "$DK" | jq -r .CiphertextBlob | base64 --decode > /tmp/dk.enc     # encrypted data key (store this)

# 2) Encrypt the payload locally with the plaintext data key
openssl enc -aes-256-cbc -pbkdf2 -salt -in /tmp/payload.bin -out /tmp/payload.enc \
  -pass file:/tmp/dk.plain
# 3) DISCARD the plaintext data key — never store it beside the ciphertext
rm -f /tmp/dk.plain
ls -l /tmp/payload.enc /tmp/dk.enc
# Expected: payload.enc (~1 MB) and dk.enc (the wrapped key). Together they are the "envelope".

Step 4 — Decrypt: recover the data key, then the file. Send the encrypted data key back to KMS with the same encryption context.

aws kms decrypt --ciphertext-blob fileb:///tmp/dk.enc \
  --encryption-context purpose=lab,tenant=acme --region ap-south-1 \
  --query Plaintext --output text | base64 --decode > /tmp/dk.plain2

openssl enc -d -aes-256-cbc -pbkdf2 -in /tmp/payload.enc -out /tmp/payload.dec \
  -pass file:/tmp/dk.plain2
rm -f /tmp/dk.plain2

cmp /tmp/payload.bin /tmp/payload.dec && echo "ROUND-TRIP OK"
# Expected: ROUND-TRIP OK  (the decrypted file is byte-identical)

Step 5 — Prove a mismatched encryption context fails. Change one value and KMS refuses.

aws kms decrypt --ciphertext-blob fileb:///tmp/dk.enc \
  --encryption-context purpose=lab,tenant=WRONG --region ap-south-1
# Expected: An error occurred (InvalidCiphertextException) when calling the Decrypt operation

Step 6 — Enable rotation and confirm the key ID is unchanged.

aws kms enable-key-rotation --key-id alias/kv-kms-lab \
  --rotation-period-in-days 365 --region ap-south-1
aws kms get-key-rotation-status --key-id alias/kv-kms-lab --region ap-south-1
# Expected: { "KeyRotationEnabled": true, "RotationPeriodInDays": 365 }
# The key ARN/alias never change — rotation is transparent.

Step 7 — Teardown (⚠️ deletion has a waiting period). Delete the alias, then schedule key deletion with the minimum 7-day window (a CMK bills ~$1/month while it exists, including during the pending window).

aws kms delete-alias --alias-name alias/kv-kms-lab --region ap-south-1
aws kms schedule-key-deletion --key-id "$KEY_ARN" \
  --pending-window-in-days 7 --region ap-south-1
# Expected: { "KeyState":"PendingDeletion","DeletionDate": "...+7 days" }
# To abort while it's pending: aws kms cancel-key-deletion --key-id "$KEY_ARN"
rm -f /tmp/payload.* /tmp/dk.enc /tmp/key-policy.json

⚠️ A customer-managed key cannot be deleted immediately — only scheduled (7–30 days) — and it keeps billing until the window closes. Schedule it now so it stops billing, and remember: once deletion completes, anything encrypted under this key is unrecoverable.

The Terraform equivalent of the whole key setup, for when you graduate from click-ops:

resource "aws_kms_key" "lab" {
  description              = "kv-kms-lab"
  key_usage                = "ENCRYPT_DECRYPT"
  customer_master_key_spec = "SYMMETRIC_DEFAULT"
  enable_key_rotation      = true
  rotation_period_in_days  = 365
  deletion_window_in_days  = 7      # 7-30; teardown waiting period
  policy = data.aws_iam_policy_document.lab.json
}
resource "aws_kms_alias" "lab" {
  name          = "alias/kv-kms-lab"
  target_key_id = aws_kms_key.lab.key_id
}

Common mistakes & troubleshooting — the KMS AccessDenied playbook

This is the playbook. KMS gives you terse exceptions; the skill is mapping the exception plus the operation to the exact layer. Work it top-down: authorization first (key policy, then IAM delegation, then grant), then request-shape issues (encryption context, ViaService, key state, size, throttling). Always read the full error message — it often names the key ARN and the missing action.

First, the 30-second triage — the exception you see maps almost 1:1 to the layer.

Now the full playbook.

KMS exception reference

Not every failure is authorization. Distinguish the auth failures from request-shape and state failures.

Encryption context & ViaService — the quiet deniers

When the principal is clearly authorized but you’re still denied or getting InvalidCiphertext, a condition or the context evaluated wrong.

The nastiest three, in prose

The SSE-KMS 403 that hides inside an S3 error. You call s3:GetObject, S3 returns 403 AccessDenied, and you spend an hour on the bucket policy — but the object is SSE-KMS and the real denier is the key policy not granting you kms:Decrypt. Tell them apart by reading the message: an S3-authorization denial references S3; the KMS one names kms:Decrypt and a key ARN. Fix it on the key, not the bucket. Cross-account makes it a three-door problem — bucket policy, caller IAM, and key policy — as Securing Amazon S3 covers from the S3 side.

The encryption-context mismatch that looks like corruption. InvalidCiphertextException reads like “your data is damaged,” and people go hunting for a truncated blob. Nine times out of ten the blob is fine and the encryption context differs — a value changed, a pair got dropped, or the case is off. KMS binds the context as authenticated data, so an off-by-one context is indistinguishable from tampering, on purpose. Recover the exact context from the GenerateDataKey/Encrypt event in CloudTrail (it’s logged in the clear) and replay it.

The “unused” key that gets scheduled for deletion. Console activity graphs lag and don’t show every caller; a key that looks idle may back a quarterly job or a rarely-read archive. ScheduleKeyDeletion starts a 7–30 day countdown, and if it completes, every ciphertext under that key is gone forever — the one AWS action with no undo. Never delete straight away: disable the key, alarm on ScheduleKeyDeletion/DisableKey, watch CloudTrail for real callers hitting KMSInvalidStateException, and only schedule deletion (long window) once it’s been provably silent.

Best practices

• Default to one symmetric SYMMETRIC_DEFAULT CMK per data domain. Reach for asymmetric only when you must share a public key or verify off-AWS, and HMAC only for MAC use cases. Don’t over-fragment into hundreds of keys you can’t audit.
• Keep the root IAM-delegation statement unless you have a reason to remove it — then you manage day-to-day access with IAM and reserve the key policy for the deliberate exceptions (cross-account, service principals, hardening).
• Treat cross-account KMS as multi-door from the start: the owner’s key policy names the external principal and the caller’s IAM allows the action and (for S3) the bucket policy allows it. Write all of them together.
• Always envelope-encrypt anything over 4 KB or anything at volume. Use the AWS Encryption SDK with data-key caching so KMS traffic (and cost) stays flat as throughput grows; discard the plaintext data key immediately after use.
• Use encryption context on every envelope operation — bind data to a tenant/purpose/object, scope grants and policies to it, and get a free audit breadcrumb. Never put secrets in it (it’s logged).
• Prefer grants for services and temporary access, key policy/IAM for durable access. Add kms:GrantIsForAWSResource so only AWS services can create grants where that’s the intent.
• Turn on key rotation (automatic) for every symmetric CMK; it’s free, transparent, and an easy compliance win. Rotate on-demand immediately after any suspected exposure.
• Use aliases everywhere in code and config (alias/app-data, never a raw key ID) so you can repoint to a new or replica key without a redeploy.
• Lock direct use down with kms:ViaService where a key should only ever be used through one service — but keep an admin path so you’re not locked out of your own key.
• Never delete a key on a hunch. Disable → alarm on DisableKey/ScheduleKeyDeletion → watch CloudTrail → schedule deletion with the maximum 30-day window. Alarm on PutKeyPolicy too — it catches a policy suddenly widening to Principal: "*".
• Right-size against throttling before it happens: know your Region’s KMS request-rate quota, prefer envelope + caching + S3 Bucket Keys, and request a quota increase ahead of a launch rather than during the incident.

Security notes

Encryption keys are the crown jewels; the controls exist so a single mistake doesn’t hand them over or lose them.

Cost & sizing

KMS is cheap until it isn’t — the monthly key fee is trivial, and the real lever is request volume, which envelope encryption and caching control. Everything free (AWS-owned/managed keys’ monthly fee, the first tranche of requests) versus what actually shows up on the bill:

The practical guidance: a handful of CMKs cost a few dollars a month — don’t agonise over the monthly fee, agonise over request volume. A workload that calls Decrypt per record will dominate its own bill and risk throttling; the same workload using envelope encryption with data-key caching (or S3 Bucket Keys for SSE-KMS) can cut KMS requests by one-to-two orders of magnitude, taking both cost and throttling off the table. Delete keys you truly don’t need (after the disable-and-watch ritual) so you’re not paying ~$1/month each for abandoned keys — but never trade a dollar of key fee for the risk of unrecoverable data.

Interview & exam questions

Q1. Why does a symmetric KMS key never appear in your code, and how do you encrypt a 1 GB file with it? The key material never leaves KMS in plaintext, so you can’t encrypt locally with it directly. You use envelope encryption: GenerateDataKey returns a plaintext data key and an encrypted copy; you encrypt the file locally with the plaintext key, discard it, and store the encrypted data key with the ciphertext. (SAA-C03, DVA-C02)

Q2. An IAM policy grants kms:Decrypt on a key, but the call gets AccessDeniedException. Why, and how do you confirm? The key policy is the root of trust and it didn’t delegate to IAM (no “Enable IAM User Permissions” root statement) or didn’t name the principal. Confirm with aws kms get-key-policy; fix by adding the principal (or the root delegation) to the key policy. (SCS-C02, DVA-C02)

Q3. What’s the difference between direct Encrypt and GenerateDataKey? Encrypt has KMS encrypt your plaintext directly, capped at 4 KB and one KMS call per operation. GenerateDataKey returns a data key you use locally (no size limit, far less KMS traffic) — the basis of envelope encryption. (SAA-C03, DVA-C02)

Q4. What is encryption context and what happens if it doesn’t match on decrypt? Non-secret key–value pairs bound to the ciphertext as additional authenticated data; KMS requires the identical context on decrypt. A mismatch (or omission) yields InvalidCiphertextException. It’s logged in CloudTrail, so never put secrets in it. (SCS-C02, DVA-C02)

Q5. Compare AWS-owned, AWS-managed, and customer-managed keys. AWS-owned: invisible, free, no control. AWS-managed (aws/service): visible, free monthly, AWS-set policy, forced yearly rotation, no cross-account. Customer-managed (CMK): full policy control, chosen rotation, cross-account, deletion control, ~$1/month. (SAA-C03, SCS-C02)

Q6. How do grants differ from key policies and IAM policies? Grants are temporary, additive, programmatic delegations (CreateGrant) of specific operations to a principal — used heavily by AWS services (e.g., EBS) — that can’t express Deny and are revoked via RetireGrant/RevokeGrant. Key policies and IAM policies are durable and can express Deny. (SCS-C02)

Q7. What does automatic key rotation change, and what stays the same? It generates new backing key material; the key ID, ARN, alias, and policy stay identical, old material is retained so existing ciphertext still decrypts, and no re-encryption or code change is needed. Not supported for asymmetric/HMAC/imported keys. (SAA-C03, SCS-C02)

Q8. You get 403 reading an SSE-KMS object though s3:GetObject is granted. Cause and fix? The KMS key policy doesn’t grant your principal kms:Decrypt. The errorMessage names kms:Decrypt and a key ARN; fix it in the key policy (cross-account also needs the caller’s IAM and the bucket policy). (SCS-C02, DVA-C02)

Q9. What is a multi-Region key and when do you use one? A primary plus replicas that share the same key material and key ID (mrk-…), so data encrypted in one Region decrypts in another with no cross-Region call — for DR, active-active global apps, and cross-Region data movement. Each replica has its own policy/grants/aliases. (SCS-C02, SAA-C03)

Q10. How do you safely decommission a key you think is unused? Don’t delete first. Disable it, alarm on DisableKey/ScheduleKeyDeletion, watch CloudTrail for callers hitting KMSInvalidStateException, then ScheduleKeyDeletion with a long window; CancelKeyDeletion recovers it while pending. Deletion is irreversible once the 7–30 day window closes. (SCS-C02)

Q11. What does kms:ViaService do, and what’s a common self-inflicted failure with it? It restricts key use to requests arriving through a specific AWS service in a Region (e.g., s3.ap-south-1.amazonaws.com). The classic failure: after locking a key to S3, your own direct aws kms decrypt is denied because that path isn’t S3 — keep an admin path. (SCS-C02)

Q12. How do you keep a high-throughput workload from throttling KMS? Use envelope encryption (one GenerateDataKey per data key, not per record) with data-key caching in the AWS Encryption SDK, enable S3 Bucket Keys for SSE-KMS, and request a KMS request-rate quota increase ahead of the peak. (DVA-C02, SCS-C02)

Quick check

1. You need to encrypt a 200 MB file with a symmetric CMK. Which API do you call and why not Encrypt?
2. Your IAM policy allows kms:Decrypt, but the call 403s. Which layer is most likely, and how do you confirm?
3. A decrypt returns InvalidCiphertextException though the blob is intact. What’s the cause and the fix?
4. What changes and what stays the same when a symmetric CMK auto-rotates?
5. Why is scheduling a key for deletion the most dangerous KMS operation, and what’s the safe alternative first?

Answers

1. GenerateDataKey (envelope encryption). Direct Encrypt is capped at 4 KB, and one KMS call per file doesn’t scale; with GenerateDataKey you encrypt the file locally with the plaintext data key, discard it, and store the encrypted data key.
2. The key policy. It’s the root of trust — an IAM allow works only if the key policy delegates to IAM (the “Enable IAM User Permissions” root statement) or names the principal directly. Confirm with aws kms get-key-policy --key-id <k> --policy-name default.
3. Encryption-context mismatch. The context on decrypt must byte-for-byte match the context used on encrypt (case-sensitive). Recover the exact context from the GenerateDataKey/Encrypt event in CloudTrail and pass it identically.
4. The backing key material changes (new version); the key ID, ARN, alias, and policy stay the same, old material is kept so existing ciphertext still decrypts, and you re-encrypt nothing.
5. Because completed deletion is irreversible and makes every ciphertext under the key permanently unrecoverable. Safe alternative: disable the key first, alarm on and watch CloudTrail for real callers, and only then schedule deletion with the maximum 30-day window (cancellable while pending).

Glossary

• AWS KMS — Managed service that creates and controls cryptographic keys and performs cryptographic operations, backed by FIPS-validated HSMs.
• CMK (customer-managed key) — A KMS key you create and fully control (policy, rotation, deletion); costs ~$1/month.
• AWS-managed key — A service key with an aws/<service> alias; AWS owns its policy and rotates it yearly; free monthly, pay per request.
• AWS-owned key — A key in a shared AWS pool, invisible and free, used for default encryption with no customer control.
• Key policy — The mandatory resource policy on a KMS key; the root of trust for all access decisions.
• Root delegation (“Enable IAM User Permissions”) — The default key-policy statement (Principal: account root) that lets IAM policies govern the key; without it, IAM alone can’t grant access.
• Grant — A temporary, additive, programmatic delegation of specific KMS operations to a principal, via CreateGrant; used heavily by AWS services.
• Data key — A key KMS generates (GenerateDataKey) for you to use locally; returned as a plaintext copy (use and discard) and an encrypted copy (store).
• Envelope encryption — Encrypting data with a data key and encrypting that data key with a CMK; removes the 4 KB limit and cuts KMS traffic.
• Encryption context — Non-secret key–value additional authenticated data bound to a ciphertext; must match exactly on decrypt.
• kms:ViaService — A condition key restricting key use to requests coming through a specific AWS service and Region.
• Key spec / key usage — The immutable algorithm (SYMMETRIC_DEFAULT, RSA_2048, ECC_NIST_P256, HMAC_256, …) and permitted operation (ENCRYPT_DECRYPT, SIGN_VERIFY, GENERATE_VERIFY_MAC) fixed at creation.
• Key rotation — Replacing a key’s backing material while keeping its key ID/ARN/alias/policy; automatic (configurable period) or on-demand.
• Alias — A friendly, repointable, per-Region name (alias/…) for a key; aws/ is reserved for AWS-managed keys.
• Multi-Region key (MRK) — A primary plus replicas sharing key material and key ID (mrk-…) so ciphertext moves across Regions without cross-Region calls.
• Key state — The key’s lifecycle status (Enabled, Disabled, PendingDeletion, Unavailable, …); only Enabled keys perform crypto.
• Scheduled deletion — ScheduleKeyDeletion with a mandatory 7–30 day waiting period; irreversible once complete, cancellable while pending.

Next steps

• Apply KMS to storage you’ve already secured — turn on SSE-KMS and Bucket Keys the right way in Securing Amazon S3: Bucket Policies, Block Public Access & the 403 Playbook, and encrypt block storage in Amazon EBS Hands-On: Volume Types, Snapshots, Resizing & Encryption.
• Store and rotate application secrets on top of a CMK with AWS Secrets Manager rotation hands-on and configuration with SSM Parameter Store for config.
• Master the authorization model beneath every KMS 403 in Debugging IAM ‘Access Denied’: Policy Evaluation Logic, SCPs, Boundaries & a Playbook, and the cross-account mechanics in Cross-Account Access with IAM Roles & sts:AssumeRole.
• Prove who used every key over time with AWS CloudTrail and Config: Audit and Compliance at Scale.