Google Cloud KMS & Secret Manager, In Depth: Keys, CMEK, Envelope Encryption & Secrets

Every byte you store in Google Cloud is already encrypted at rest. You did nothing to make that happen — Compute Engine disks, Cloud Storage objects, BigQuery tables and Cloud SQL volumes are all encrypted with keys Google generates, rotates and guards for you, and you never see them. So a fair first question is: if encryption is automatic and free, why does Google ship two whole products — Cloud KMS and Secret Manager — and why does every serious security review ask whether you use them?

The answer is control. “Encrypted by default” protects you against someone walking out of a data centre with a disk. It does not answer the questions auditors, regulators and your own incident-response team actually ask: Who can revoke access to the plaintext, and how quickly? Where does the key material physically live? Can you prove a key was destroyed? Where do your application’s passwords, API keys and certificates live, who can read them, and how often do they change? Default encryption gives Google the lever; Cloud KMS hands the lever to you. And the credentials your code needs at runtime — the database password, the third-party API token — do not belong in environment variables or a config.json in your repo; they belong in Secret Manager, versioned, access-controlled and rotatable.

This lesson is the exhaustive, beginner-accessible map of both services. By the end you will understand the full Cloud KMS hierarchy (key rings, keys, versions), every key purpose (symmetric, asymmetric sign, asymmetric decrypt, MAC), rotation, how Customer-Managed Encryption Keys (CMEK) wire into GCP services, the envelope encryption mechanics underneath all of it, the protection levels (SOFTWARE, HSM, EXTERNAL, EXTERNAL_VPC), IAM on keys, Cloud HSM and Autokey; and on the Secret Manager side, secrets and versions, access, rotation, the replication models, IAM and CMEK. It maps to the Associate Cloud Engineer (ACE) and Professional Cloud Security Engineer (PCSE) exams and gives a working architect everything needed to operate both services in production.

Learning objectives

By the end of this lesson you can:

• Explain Google’s default encryption at rest and the difference between Google-managed keys, CMEK and CSEK, and articulate why you would move from one to the next.
• Lay out the full Cloud KMS hierarchy — key ring → key → key version — and choose locations, purposes and algorithms deliberately.
• Configure key rotation and reason correctly about why old versions stay enabled, and the version lifecycle (ENABLED → DISABLED → DESTROY_SCHEDULED → DESTROYED).
• Describe envelope encryption with DEKs and KEKs and explain how it makes encrypting petabytes both fast and revocable.
• Wire CMEK into Cloud Storage, BigQuery, Cloud SQL, Compute Engine disks and Pub/Sub, including the service agent grant that everyone forgets.
• Choose a protection level (SOFTWARE / HSM / EXTERNAL / EXTERNAL_VPC) and explain Cloud HSM and External Key Manager, plus Autokey for automated CMEK provisioning.
• Apply least-privilege IAM to keys and secrets, and design separation of duties so no single admin can both use and destroy a key.
• Operate Secret Manager end to end — secrets, immutable versions, the latest alias, access, rotation schedules, automatic vs user-managed replication, IAM and CMEK on secrets.

Prerequisites & where this fits

You need a Google Cloud project with billing enabled (the $300 free trial is plenty for the lab), the gcloud CLI installed and authenticated, and a working grasp of IAM — principals, roles and policy bindings — because access to every key and every secret is governed by IAM. If IAM is hazy, read Google Cloud IAM Fundamentals: Roles, Service Accounts & the Allow Policy first. This is the Security lesson of the Google Cloud Zero-to-Hero course, sitting in the Intermediate tier after Pub/Sub and before the Operations Suite. Two companion lessons go deeper than this one: Cloud KMS in Depth: CMEK, Envelope Encryption, Cloud HSM and External Key Manager and Secret Manager Rotation Pipelines with Cloud Functions, IAM and CMEK — this lesson is the complete on-ramp to both.

Core concept: default encryption and the three key-management models

Start with the baseline, because it is the thing you are choosing to change. All customer data at rest in Google Cloud is encrypted by default, with no action and no extra cost. Under the hood Google uses envelope encryption (we will pull this apart in full below): your data is encrypted with a data encryption key (DEK), and that DEK is itself encrypted by a key encryption key (KEK) held in Google’s internal KMS. Encryption uses AES-256 (or AES-128 for some legacy data) in GCM or CBC/CTR mode depending on the service. You inherit strong cryptography for free.

What you are deciding when you reach for Cloud KMS is who controls the top key in that chain. There are three models, and naming them precisely is the foundation of every conversation that follows:

The practical progression is almost always default → CMEK. CMEK is the sweet spot: Google still does the heavy lifting (high availability, durability of the key, the actual crypto operations), but you own the key object, decide its rotation, control who may use it through IAM, get an audit log of every use, and — the headline capability — can make a petabyte of data instantly unreadable by disabling a single key version. CSEK is a niche tool with serious downsides (lose the key and your data is gone forever; Google cannot help) and most architectures never need it. This lesson is overwhelmingly about CMEK via Cloud KMS.

Core concept: the Cloud KMS hierarchy

Cloud KMS organises keys into a strict four-level hierarchy. Getting this vocabulary exact saves you from IAM, location and rotation mistakes later:

A few consequences matter enormously in practice:

• A key ring’s location is permanent and the key ring cannot be deleted. This surprises everyone. You will accumulate empty key rings forever — that is normal and they cost nothing. Choose the location to match the data you will protect: a CMEK key must live in a location compatible with the resource it encrypts. A regional resource (a Cloud SQL instance in europe-west2) needs a key in europe-west2; a multi-region Cloud Storage bucket in EU needs a key in the eu multi-region; the special global location exists for application-level use that is not tied to a region.
• A key’s purpose and protection level are fixed at creation. You cannot turn a symmetric key into an asymmetric one, nor move a SOFTWARE key to HSM. Plan it once.
• Keys are never deleted; versions are. You disable or schedule destruction of versions. The key object persists.

Locations: regional, multi-regional and global

KMS keys exist in a location, which determines where the cryptographic operations happen and where the key material is stored. The three flavours mirror the rest of GCP:

The cardinal rule: the key’s location must be compatible with the resource it protects. GCP will reject a CMEK assignment where the key location and the resource location do not match, and this is the single most common first-time CMEK error.

Core concept: keys, purposes and algorithms

A CryptoKey has a purpose chosen at creation, and the purpose constrains which algorithms and operations are available. There are four purposes:

Symmetric (encryption) is the workhorse — it is the only purpose that CMEK uses, and it is what you reach for unless you specifically need public-key cryptography. For symmetric keys the algorithm is the Google-managed GOOGLE_SYMMETRIC_ENCRYPTION (AES-256-GCM); you do not choose it. For asymmetric and MAC keys you do choose the algorithm and curve/size:

A subtle but exam-worthy point: asymmetric keys do not auto-rotate. Because the public half is distributed to verifiers/encryptors, Google cannot silently swap the version — you manage versions explicitly. Only symmetric keys support automatic rotation.

Core concept: key versions, rotation and the version lifecycle

The CryptoKeyVersion is where the real cryptographic material lives, and it has a strict lifecycle:

ENABLED  ->  DISABLED  ->  DESTROY_SCHEDULED  ->  DESTROYED
   ^___________|              (24h+ wait)

• ENABLED — usable for encrypt and decrypt.
• DISABLED — rejects all operations, but can be re-enabled. This is your emergency “make the data unreadable” switch — disabling the version that protects a dataset locks it instantly, and re-enabling restores access. Reversible, fast, and the reason CMEK is so powerful.
• DESTROY_SCHEDULED — you have requested destruction; KMS waits a mandatory period (default 24 hours, configurable per key from 1 to 120 days via --destroy-scheduled-duration) before the material is gone. During the wait you can restore the version.
• DESTROYED — the material is permanently deleted. Any data still encrypted with this version is unrecoverable. Google cannot help. This is irreversible.

The destruction delay is a deliberate safety net against fat-fingered or malicious destruction — it gives you a window to notice and restore.

Rotation

For symmetric keys you set a rotation period. On each rotation KMS generates a new version and makes it the primary — the version used for all new encryptions. Crucially, old versions stay ENABLED so that anything encrypted previously can still be decrypted. You are not re-encrypting existing data on rotation; you are changing the key used going forward.

PROJECT=sec-kms-lab
LOCATION=us-central1

# Create a key ring (location is permanent)
gcloud kms keyrings create app-keyring \
  --project="$PROJECT" --location="$LOCATION"

# Symmetric CMEK key with 90-day automatic rotation
gcloud kms keys create app-cmek \
  --project="$PROJECT" --location="$LOCATION" \
  --keyring=app-keyring \
  --purpose=encryption \
  --protection-level=software \
  --rotation-period=90d \
  --next-rotation-time="$(date -u -d '+90 days' +%Y-%m-%dT%H:%M:%SZ 2>/dev/null \
      || date -u -v+90d +%Y-%m-%dT%H:%M:%SZ)"

Rotation guidance worth committing to memory:

• Recommended default: 90 days for symmetric keys. It balances cryptographic hygiene against the operational overhead of managing many versions. Some compliance regimes mandate 30 or 365 days.
• Rotation does not re-encrypt existing data. To force new data under a new key you simply rotate; to re-encrypt old data (e.g. to retire a compromised version), you must rewrite the data through the service (rewrite the object, recreate the resource) so it picks up the new primary.
• Manual rotation is possible at any time with gcloud kms keys versions create ... --primary.
• Asymmetric and external keys must be rotated manually.

Core concept: envelope encryption (DEKs and KEKs)

This is the single most important mental model in the whole topic, and it underpins both default encryption and CMEK. You almost never use a KMS key to encrypt your data directly. Why? Because KMS has request-size limits (you cannot stream a 5 GB file through it), every call costs latency, and you do not want your key material leaving its boundary on every byte. Instead you use envelope encryption:

1. Generate a data encryption key (DEK) locally — a fresh AES-256 key.
2. Encrypt your data with the DEK (fast, local, any size).
3. Send the DEK to KMS and ask the key encryption key (KEK) — your CMEK key — to encrypt it. KMS returns the wrapped (encrypted) DEK.
4. Store the wrapped DEK alongside the ciphertext, and throw away the plaintext DEK.

To read the data: send the wrapped DEK to KMS, the KEK unwraps it, you decrypt locally with the DEK, then discard it. The genius of this design:

• Performance — bulk encryption is local AES; KMS only ever sees the tiny DEK.
• Revocability — disable or destroy the KEK and every wrapped DEK becomes un-unwrappable, so all the data is instantly unreadable. One small key controls a petabyte.
• Rotation cheapness — rotate the KEK and you only need to re-wrap DEKs (small), not re-encrypt data (huge).

   plaintext data ──(AES with DEK)──► ciphertext
        DEK ──(wrap with KEK in KMS)──► wrapped DEK
   stored together: [ ciphertext | wrapped DEK ]

This is exactly what GCP services do internally when you attach a CMEK: they generate per-resource DEKs and wrap them with your KMS key (the KEK). KMS even offers generateRandomBytes and an EncryptedKey flow to help you implement this pattern yourself in application code.

Core concept: protection levels — SOFTWARE, HSM, EXTERNAL, EXTERNAL_VPC

The protection level decides where the key material physically lives and the security boundary it sits behind. It is fixed at key creation. There are four:

Key facts an interviewer probes:

• Cloud HSM is fully managed — you do not rack hardware. It gives you FIPS 140-2 Level 3 assurance and, importantly, an attestation statement you can verify, proving the key was generated and lives in genuine HSM hardware. The API is identical to software KMS; only --protection-level=hsm differs.
• External Key Manager (EKM) lets the key material live outside Google entirely in a supported partner (Fortanix, Thales, Virtru, Equinix and others). Google calls out to the external manager to wrap/unwrap; if you cut off the external manager, Google loses access too — the ultimate “hold your own key” control. EXTERNAL connects over the public internet via a key URI; EXTERNAL_VPC connects privately through a VPC for latency and to avoid internet exposure.
• The vast majority of real CMEK deployments use SOFTWARE — reach for HSM/EKM only when a specific compliance requirement forces it, because they cost more and (for EKM) add an external dependency on your data’s availability.

Core concept: CMEK on GCP services — wiring the key in

CMEK is the headline use of Cloud KMS: you tell a GCP service to use your KMS key instead of Google’s default key. The pattern is the same across services and has one step everyone forgets — granting the service’s service agent permission to use your key.

Every GCP service that supports CMEK runs as a Google-managed service agent (a special service account like service-PROJECT_NUMBER@gcp-sa-storage.iam.gserviceaccount.com for Cloud Storage). That service agent must hold the roles/cloudkms.cryptoKeyEncrypterDecrypter role on your key, or the service literally cannot wrap/unwrap the DEKs and the CMEK assignment fails.

Here is the canonical Cloud Storage example end to end:

PROJECT=sec-kms-lab
PROJECT_NUMBER=$(gcloud projects describe "$PROJECT" --format='value(projectNumber)')
LOCATION=us-central1
KEY=projects/$PROJECT/locations/$LOCATION/keyRings/app-keyring/cryptoKeys/app-cmek

# 1. Grant the Cloud Storage service agent use of the key
gcloud kms keys add-iam-policy-binding app-cmek \
  --project="$PROJECT" --location="$LOCATION" --keyring=app-keyring \
  --member="serviceAccount:service-${PROJECT_NUMBER}@gs-project-accounts.iam.gserviceaccount.com" \
  --role="roles/cloudkms.cryptoKeyEncrypterDecrypter"

# 2. Create a bucket that uses the CMEK as its default encryption key
gcloud storage buckets create gs://${PROJECT}-cmek-bucket \
  --location="$LOCATION" \
  --default-encryption-key="$KEY"

CMEK support and the gotchas, service by service:

The two universal rules: location compatibility (key and resource in compatible locations) and the service-agent grant. If a CMEK operation fails, check those two first.

Core concept: IAM on keys and separation of duties

Access to KMS is pure IAM, and the predefined roles are deliberately fine-grained so you can separate managing a key from using it — the heart of good key hygiene:

The separation-of-duties design that interviewers love: grant the admin role to your key administrators (who can rotate and destroy but cannot read plaintext) and the encrypter/decrypter role to the applications and service agents (who can use the key but cannot destroy it). No single principal should hold both — so a compromised application cannot destroy keys, and a compromised admin cannot exfiltrate plaintext. Grant roles at the key level (most specific) rather than project-wide whenever possible, and remember that every KMS operation is written to Cloud Audit Logs, giving you a complete trail of who used which key version when.

Core concept: Autokey

Manually creating a key ring, a key, setting rotation and wiring the service-agent grant for every resource gets tedious and error-prone at scale. Cloud KMS Autokey automates it. With Autokey configured at the folder level, when a developer creates a CMEK-enabled resource (a bucket, a disk, a Cloud SQL instance), KMS automatically provisions a dedicated key with sensible defaults — correct location, rotation policy and the service-agent IAM binding — on demand. The benefits:

• Per-resource keys by default without manual toil — better blast-radius isolation than one shared key.
• Consistent, policy-compliant key configuration (rotation, location) enforced centrally.
• A clean separation between the security team (who set up Autokey and the key-management folder) and developers (who just request CMEK resources).

Autokey is the “paved road” for CMEK at organisation scale; without it, teams either skip CMEK (too much friction) or share one key too widely. It is administered by a key administrator and consumed via a --kms-key=autokey style request in supporting services.

From KMS to secrets: why Secret Manager exists

KMS protects keys. But your application also needs secrets — database passwords, third-party API keys, OAuth client secrets, TLS private keys, service-account credentials. These do not belong in environment variables baked into an image, in a .env committed to git, or in a config file on disk. They belong in Secret Manager: a fully managed, global service that stores secret payloads, versions them, controls access with IAM, encrypts them at rest (with Google keys or your CMEK), and can rotate them on a schedule.

The relationship to KMS is direct: Secret Manager uses envelope encryption with KMS underneath to protect every secret at rest, and you can bring your own CMEK key to control that encryption — the two services compose.

Core concept: the Secret Manager object model

Get the model exact before you automate against it:

PROJECT=sec-kms-lab

# Create the container (no payload yet), automatic replication
gcloud secrets create db-app-password \
  --replication-policy="automatic" \
  --project="$PROJECT"

# Add a version — versions are immutable; "add" always makes a new number
echo -n "initial-bootstrap-pw" | \
  gcloud secrets versions add db-app-password --data-file=- \
  --project="$PROJECT"

# Access the payload: pin to a number, or use the moving 'latest'
gcloud secrets versions access latest --secret=db-app-password --project="$PROJECT"
gcloud secrets versions access 1       --secret=db-app-password --project="$PROJECT"

The immutability of versions plus the latest alias is what makes zero-downtime rotation possible. Because latest follows the newest enabled version, a rotator that adds version N+1 instantly shifts latest while version N stays enabled and valid — consumers pinned to latest pick up the new value on their next read; in-flight consumers keep working. You only break something by disabling or destroying the old version too early. The discipline: add new, cut over consumers, then disable old (never destroy immediately).

Core concept: accessing secrets from workloads

A secret is useless if your code cannot read it cleanly. The access surface:

• Direct API / client libraries — call AccessSecretVersion from Python, Go, Java, Node, etc. The library uses Application Default Credentials, so on GCP the workload’s own service account authenticates with no embedded key.
• gcloud / CLI — gcloud secrets versions access for humans and scripts.
• Cloud Run & Cloud Functions — mount a secret as an environment variable or as a file in a volume, pinned to latest or a fixed version. The platform fetches it at deploy/start; you never write retrieval code.
• GKE — via the Secret Manager CSI driver (mounts secrets as files) or Workload Identity + the client library. Prefer this over native Kubernetes Secrets, which are only base64-encoded, not encrypted, by default.
• Compute Engine — the client library with the VM’s service account, or fetch at startup.

The golden rule across all of these: the workload’s identity (its service account) must hold roles/secretmanager.secretAccessor on the specific secret. Grant it at the secret level, not the project, so each workload reads only the secrets it needs.

Core concept: Secret Manager IAM

Like KMS, access is IAM and the roles separate managing from reading:

The least-privilege pattern mirrors KMS exactly: applications get secretAccessor on the precise secrets they consume; rotation logic gets secretVersionManager; humans who administer the catalogue get admin. No application should ever hold admin. And as with KMS, every access is audit-logged (enable data-access audit logs to capture reads) so you can prove who read which secret version.

Core concept: Secret Manager replication

When you create a secret you choose how its payload is replicated, and this is a permanent decision per secret:

# User-managed replication pinned to two European regions for residency
gcloud secrets create eu-only-secret \
  --replication-policy="user-managed" \
  --locations="europe-west1,europe-west2" \
  --project="$PROJECT"

A critical CMEK interaction: if you want CMEK on a secret, the key(s) must match the replication policy. An automatically-replicated secret needs a CMEK key per replica location (or you use user-managed replication so you can pair each region with a key in that region). This is why data-residency CMEK secrets almost always use user-managed replication.

Core concept: Secret Manager rotation and CMEK

Rotation in Secret Manager is notification-driven, not magic. You attach a rotation schedule (a period and/or a next-rotation time) and a Pub/Sub topic; when the secret is due, Secret Manager publishes a message to the topic. You supply the logic — typically a Cloud Function or Cloud Run job — that mints a fresh credential, calls add-version, and (after consumers cut over) disables the old version. Secret Manager does not generate the new secret value itself; it orchestrates the timing and notification.

# A secret that emits a Pub/Sub notification every 30 days for a rotator to act on
gcloud secrets create db-app-password-rotating \
  --replication-policy="automatic" \
  --topics="projects/${PROJECT}/topics/secret-rotation" \
  --rotation-period="2592000s" \
  --next-rotation-time="$(date -u -d '+30 days' +%Y-%m-%dT%H:%M:%SZ 2>/dev/null \
      || date -u -v+30d +%Y-%m-%dT%H:%M:%SZ)" \
  --project="$PROJECT"

For CMEK on secrets, you supply a KMS key so that your key wraps the secret’s DEK at rest (Secret Manager uses envelope encryption underneath). The Secret Manager service agent needs cryptoKeyEncrypterDecrypter on the key (the same service-agent pattern as every other CMEK service), and the key location must be compatible with the secret’s replication. The full rotation pipeline — Pub/Sub, a Cloud Functions rotator, CMEK and least-privilege IAM — is built end to end in the companion lesson; here you should retain the shape: schedule + Pub/Sub notification → your rotator adds a version → consumers on latest cut over → disable the old version, all optionally wrapped in your own CMEK.

The diagram above ties the two services together: on the left, the Cloud KMS hierarchy (key ring → key → versions) feeding CMEK into GCP services through envelope encryption (KEK wrapping per-resource DEKs); on the right, Secret Manager’s secret → versions model with the latest alias, IAM-gated access from workloads, and a rotation loop driven by Pub/Sub — with KMS sitting underneath Secret Manager to encrypt secrets at rest.

Hands-on lab: a CMEK-encrypted bucket and a versioned secret

This lab creates a KMS key, encrypts a Cloud Storage bucket with it (CMEK end to end including the service-agent grant), demonstrates instant revocation by disabling the key version, and creates a versioned secret with least-privilege access. It runs comfortably within the $300 free trial; KMS keys cost about $0.06 per active key version per month plus a tiny per-operation fee, and the bucket/secret usage here is negligible.

Step 1 — Set up variables and enable APIs

export PROJECT=$(gcloud config get-value project)
export PROJECT_NUMBER=$(gcloud projects describe "$PROJECT" --format='value(projectNumber)')
export LOCATION=us-central1

gcloud services enable cloudkms.googleapis.com secretmanager.googleapis.com \
  storage.googleapis.com --project="$PROJECT"

Step 2 — Create a key ring and a rotating CMEK key

gcloud kms keyrings create lab-keyring \
  --project="$PROJECT" --location="$LOCATION"

gcloud kms keys create lab-cmek \
  --project="$PROJECT" --location="$LOCATION" --keyring=lab-keyring \
  --purpose=encryption --protection-level=software \
  --rotation-period=90d \
  --next-rotation-time="$(date -u -d '+90 days' +%Y-%m-%dT%H:%M:%SZ 2>/dev/null \
      || date -u -v+90d +%Y-%m-%dT%H:%M:%SZ)"

Step 3 — Grant the Cloud Storage service agent use of the key, then create a CMEK bucket

SA="service-${PROJECT_NUMBER}@gs-project-accounts.iam.gserviceaccount.com"

gcloud kms keys add-iam-policy-binding lab-cmek \
  --project="$PROJECT" --location="$LOCATION" --keyring=lab-keyring \
  --member="serviceAccount:${SA}" \
  --role="roles/cloudkms.cryptoKeyEncrypterDecrypter"

KEY="projects/$PROJECT/locations/$LOCATION/keyRings/lab-keyring/cryptoKeys/lab-cmek"
gcloud storage buckets create "gs://${PROJECT}-lab-cmek" \
  --location="$LOCATION" --default-encryption-key="$KEY"

echo "hello cmek" > /tmp/lab.txt
gcloud storage cp /tmp/lab.txt "gs://${PROJECT}-lab-cmek/"

Step 4 — Prove revocation: disable the key version, watch access fail, then restore

# Disable the only (primary) version — this locks the data
gcloud kms keys versions disable 1 \
  --project="$PROJECT" --location="$LOCATION" \
  --keyring=lab-keyring --key=lab-cmek

# Reading the object now FAILS — the DEK can no longer be unwrapped
gcloud storage cat "gs://${PROJECT}-lab-cmek/lab.txt" || echo ">> Access denied: key disabled"

# Re-enable and access is restored
gcloud kms keys versions enable 1 \
  --project="$PROJECT" --location="$LOCATION" \
  --keyring=lab-keyring --key=lab-cmek
gcloud storage cat "gs://${PROJECT}-lab-cmek/lab.txt"   # -> hello cmek

Step 5 — Create a versioned secret with least-privilege access

gcloud secrets create lab-secret --replication-policy="automatic" --project="$PROJECT"
echo -n "s3cr3t-v1" | gcloud secrets versions add lab-secret --data-file=- --project="$PROJECT"
echo -n "s3cr3t-v2" | gcloud secrets versions add lab-secret --data-file=- --project="$PROJECT"

# 'latest' follows the newest enabled version
gcloud secrets versions access latest --secret=lab-secret --project="$PROJECT"   # -> s3cr3t-v2

# Grant a workload SA accessor on THIS secret only (replace with a real SA to test)
gcloud secrets add-iam-policy-binding lab-secret \
  --member="serviceAccount:${PROJECT_NUMBER}-compute@developer.gserviceaccount.com" \
  --role="roles/secretmanager.secretAccessor" --project="$PROJECT"

Validation

# Confirm the bucket's default key is your CMEK
gcloud storage buckets describe "gs://${PROJECT}-lab-cmek" \
  --format='value(default_kms_key)'

# Confirm key version state and rotation
gcloud kms keys describe lab-cmek --location="$LOCATION" --keyring=lab-keyring \
  --format='value(rotationPeriod,nextRotationTime)'

# List secret versions and states
gcloud secrets versions list lab-secret --project="$PROJECT"

You should see the CMEK resource name on the bucket, a 90-day rotation period on the key, and two ENABLED secret versions.

Cleanup

gcloud storage rm -r "gs://${PROJECT}-lab-cmek"
gcloud secrets delete lab-secret --project="$PROJECT" --quiet
gcloud kms keys versions destroy 1 \
  --project="$PROJECT" --location="$LOCATION" --keyring=lab-keyring --key=lab-cmek --quiet
# NOTE: key rings and keys CANNOT be deleted. Destroying the version stops billing for it.

Cost note

A key version costs roughly $0.06/month while active, plus ~$0.03 per 10,000 crypto operations; HSM and EKM keys cost more per version and per operation. Secret Manager bills about $0.06 per active secret version per month and $0.03 per 10,000 access operations, with a free allowance. Destroying the key version and deleting the secret stops those charges; the empty key ring and key remain forever at no cost.

Common mistakes & troubleshooting

Best practices

• Default to CMEK with SOFTWARE protection for data that has any compliance or control requirement; reserve HSM/EKM for explicit mandates.
• One purpose, planned once — purpose and protection level are immutable; design them before creating the key.
• Rotate symmetric keys (90 days is the sweet spot) and let old versions stay enabled so existing data still decrypts.
• Separate duties: key/secret admins manage; applications and service agents use. Never grant both to one principal.
• Grant at the most specific level — bind IAM on the individual key or secret, never project-wide, so blast radius is minimal.
• Per-resource keys where it matters (Autokey makes this the easy default) so disabling one key affects one resource, not your whole estate.
• Pin consumers to latest for zero-downtime secret rotation, and add → cut over → disable, never destroy immediately.
• Use user-managed replication for any secret with data-residency constraints, and pair it with region-matched CMEK keys.
• Enable data-access audit logs for KMS and Secret Manager so every key use and secret read is provable.

Security notes

The whole value proposition of these services is control under attack, so treat the controls themselves as the crown jewels. Lock down cloudkms.admin and secretmanager.admin to a tiny set of break-glass identities — anyone with admin on a key can schedule its destruction, and anyone with admin on a secret can read or delete it. Rely on the destroy-scheduled delay (raise it from the 24-hour default if your change-control cadence is slower) as a safety net, and alert on DestroyCryptoKeyVersion and on secret deletions in audit logs. Never store raw key material or secret payloads in code, images, build logs or environment files committed to source control — that single habit defeats everything KMS and Secret Manager give you. Prefer keyless access (Application Default Credentials, Workload Identity) over service-account keys, because a key file in a repo is itself the worst secret leak. For the highest assurance, use Cloud HSM for FIPS 140-2 Level 3 plus verifiable attestation, or EKM when policy demands the key material never reside in Google at all — but understand that EKM makes your data’s availability depend on the external manager.

Interview & exam questions

1. What does “encrypted at rest by default” not give you, and how does CMEK fix it? Default encryption protects against physical media theft but leaves Google holding the only key — you cannot revoke access to plaintext. CMEK gives you the key, so you can disable/destroy it and make the data unreadable on demand, with an audit trail and your own rotation.
2. Walk through envelope encryption. A local DEK encrypts the data (fast, any size); the DEK is wrapped by a KEK in KMS; the wrapped DEK is stored with the ciphertext; the plaintext DEK is discarded. To read, KMS unwraps the DEK and you decrypt locally. It gives performance, cheap rotation (re-wrap small DEKs, not re-encrypt data) and revocation (kill the KEK, all DEKs die).
3. Difference between disabling and destroying a key version? Disabling is reversible — it instantly blocks use (your emergency revocation switch) and can be re-enabled. Destroying schedules permanent deletion after a mandatory delay (default 24h, up to 120 days); once destroyed, any data still encrypted with it is unrecoverable.
4. Does rotating a key re-encrypt existing data? No. Rotation creates a new primary version used for new encryptions; old versions stay enabled to decrypt existing data. To re-encrypt old data you must rewrite/recreate it so it picks up the new primary.
5. Name the four protection levels and the headline difference. SOFTWARE (FIPS 140-2 L1, Google software), HSM (FIPS 140-2 L3 hardware with attestation), EXTERNAL (key material in a third-party manager over the internet), EXTERNAL_VPC (the same but over private VPC). They differ in where the material lives and the assurance/cost.
6. What is the one step people forget when enabling CMEK on a service? Granting the service’s service agent the cryptoKeyEncrypterDecrypter role on the key — without it the service cannot wrap/unwrap DEKs and CMEK fails.
7. Which KMS roles enforce separation of duties? cloudkms.admin can manage (rotate/destroy) but not use a key; cloudkms.cryptoKeyEncrypterDecrypter can use but not manage. Splitting them means no single principal can both read plaintext and destroy the key.
8. What does the latest alias do in Secret Manager and why does it enable zero-downtime rotation? It resolves to the highest-numbered enabled version. Adding version N+1 instantly shifts latest while N stays valid, so consumers cut over on their next read with no outage — provided you don’t disable N too early.
9. Automatic vs user-managed replication for a secret — when each? Automatic = Google picks regions, simplest, max availability. User-managed = you list specific regions, required for data-residency; also necessary to pair region-matched CMEK keys.
10. Does Secret Manager generate new secret values on rotation? No — it publishes a Pub/Sub notification on a schedule; you provide the rotator (e.g. a Cloud Function) that mints the new value and adds a version. Secret Manager orchestrates timing, not generation.
11. What is Cloud KMS Autokey for? It auto-provisions a per-resource CMEK key (correct location, rotation, service-agent IAM) when a developer creates a CMEK resource, removing manual toil and enforcing consistent, policy-compliant keys at folder scale.
12. Why store secrets in Secret Manager rather than Kubernetes Secrets or env vars? K8s Secrets are only base64-encoded by default (not encrypted); env vars/config files leak via images and source control. Secret Manager gives encryption at rest (optionally CMEK), versioning, IAM-gated access and audit logs.

Quick check

1. True/False: A Cloud KMS key ring can be deleted once it has no keys.
2. Which key purpose does CMEK use, and is it symmetric or asymmetric?
3. You disable the single key version protecting a BigQuery dataset. What happens to queries, and is it reversible?
4. Your CMEK assignment to a bucket fails with a permission error. What is the most likely missing grant?
5. A secret needs its payload to stay only in europe-west2. Which replication policy do you choose?

Answers

1. False. Key rings (and keys) cannot be deleted or renamed; their location is permanent. Empty ones are harmless and free.
2. The encryption purpose — symmetric (ENCRYPT_DECRYPT, AES-256-GCM). It is the only purpose CMEK uses.
3. Queries fail (the data is unreadable because the DEK can no longer be unwrapped). It is reversible — re-enable the version and access is restored.
4. The Cloud Storage service agent is missing roles/cloudkms.cryptoKeyEncrypterDecrypter on the key.
5. User-managed replication, with --locations=europe-west2 (and a region-matched CMEK key if encrypting with your own key).

Exercise

In your free-trial project, build a small “secure config” setup that an interviewer could probe:

1. Create a key ring and a symmetric CMEK key in your region with 30-day rotation, and verify the rotation period.
2. Create a Pub/Sub-notified, rotating secret (--topics, --rotation-period, --next-rotation-time) holding a fake API key, add two versions, and confirm latest returns the second.
3. Wrap the secret in your CMEK key (grant the Secret Manager service agent cryptoKeyEncrypterDecrypter first) and confirm creation succeeds — then disable the key version and observe that accessing the secret now fails.
4. Apply separation of duties: grant a “rotator” SA secretVersionManager and an “app” SA secretAccessor on the secret, and grant a “key-admin” SA cloudkms.admin (but not encrypterDecrypter) on the key. Confirm the app SA can read but cannot destroy, and the key-admin can rotate but cannot read plaintext.
5. Clean up: delete the secret, destroy the key version, and note that the key ring remains.

Write down, in one paragraph each, why disabling the key broke secret access (envelope encryption) and why the latest alias made adding a version a zero-downtime change.

Certification mapping

• Associate Cloud Engineer (ACE): Configuring access and security and Setting up a cloud solution environment — creating Cloud KMS keys and key rings with gcloud, enabling CMEK on services (and the service-agent grant), and managing Secret Manager secrets, versions and IAM. Expect hands-on-style questions that mirror this lab.
• Professional Cloud Security Engineer (PCSE): the deep end — default vs CMEK vs CSEK, the KMS hierarchy and protection levels (SOFTWARE/HSM/EXTERNAL/EXTERNAL_VPC), envelope encryption with DEKs/KEKs, rotation and the version lifecycle, separation-of-duties IAM on keys, Cloud HSM, EKM and Autokey, plus Secret Manager replication, rotation pipelines and CMEK. This lesson is the on-ramp; the Cloud KMS in Depth and Secret Manager Rotation Pipelines companions complete it.
• Cloud Digital Leader (CDL): the conceptual layer only — what encryption at rest is, why customers want control of their keys (CMEK), and that secrets belong in a managed secret store.

Glossary

• Default encryption at rest — Google’s automatic AES-256 encryption of all customer data, using Google-managed keys, at no cost.
• CMEK (Customer-Managed Encryption Keys) — using your Cloud KMS key for a service’s encryption, giving you control, rotation, revocation and audit.
• CSEK (Customer-Supplied Encryption Keys) — supplying raw key material on each request; key never persists in Google; rare and high-burden.
• Key ring (KeyRing) — a location-bound, non-deletable grouping of keys and an IAM boundary.
• Key (CryptoKey) — a named key with a fixed purpose and rotation policy; the IAM and reference target.
• Key version (CryptoKeyVersion) — the actual cryptographic material; the unit you enable/disable/destroy.
• Purpose — what a key does: encryption (symmetric/CMEK), asymmetric-signing, asymmetric-encryption, or mac.
• Protection level — where key material lives: SOFTWARE, HSM, EXTERNAL, EXTERNAL_VPC.
• Envelope encryption — encrypting data with a DEK, then wrapping the DEK with a KEK in KMS.
• DEK / KEK — data encryption key (encrypts data, local) / key encryption key (wraps the DEK, in KMS).
• Cloud HSM — managed FIPS 140-2 Level 3 hardware-backed keys with verifiable attestation.
• EKM (External Key Manager) — keys held by a third-party manager outside Google, reached over internet (EXTERNAL) or VPC (EXTERNAL_VPC).
• Autokey — automatic per-resource CMEK key provisioning (location, rotation, service-agent IAM) at folder scope.
• Service agent — a Google-managed service account a GCP service uses; must hold cryptoKeyEncrypterDecrypter for CMEK.
• Secret — a Secret Manager container (name, replication, IAM, optional rotation) holding no payload itself.
• Secret version — an immutable, numbered payload; state ENABLED/DISABLED/DESTROYED.
• latest alias — resolves to the highest-numbered enabled secret version; enables zero-downtime rotation.
• Replication policy — automatic (Google-chosen regions) or user-managed (regions you list, for residency).

Next steps

You can now reason about the full encryption story on Google Cloud — default vs CMEK vs CSEK — build and rotate Cloud KMS keys, wire CMEK into services with the service-agent grant, explain envelope encryption and the protection levels, and operate Secret Manager with versioned, IAM-gated, optionally CMEK-encrypted secrets. To go deeper on the key side, follow Cloud KMS in Depth: CMEK, Envelope Encryption, Cloud HSM and External Key Manager, and for production rotation automation read Secret Manager Rotation Pipelines with Cloud Functions, IAM and CMEK. Next in the course, make all of this observable in Google Cloud Operations Suite, In Depth: Cloud Monitoring, Logging, Trace & Error Reporting, where audit logs for key use and secret access become the evidence behind your alerts.