Secrets Manager Rotation at Scale: Custom Rotation Lambdas, RDS Credentials, and Cross-Account Sharing

Storing a secret in AWS Secrets Manager is the easy part. The thing that actually reduces blast radius is rotation — and rotation is where teams quietly fail, because a botched cycle either locks an application out of its own database or, worse, succeeds silently while applications keep using a credential that no longer works. The mechanics are precise: one staging-label state machine you must respect, a versioning model you cannot shortcut, and networking/KMS permissions that are unforgiving when wrong. Get one of those three wrong and the failure mode is the same flavour every time — a Lambda that times out with no useful error, a secret stuck in AWSPENDING, and a 02:00 pager.

This guide walks the four-step rotation model end to end, contrasts single-user and alternating-user RDS strategies, shows how to write a custom rotator for non-RDS credentials, and covers the cross-account sharing platform teams always end up needing. It is built as a reference you keep open mid-incident: read the prose once, then keep the tables — the step contract, the IAM grants, the VPC/KMS requirements, the failure playbook — open at 02:00. Every operation gets both the aws CLI and the IaC (Terraform) form, because the console wires permissions you will forget to declare in code.

By the end you will stop guessing. When rotation fails you will know whether it is a non-idempotent step, a missing Secrets Manager VPC endpoint, a master secret that cannot clone the user, an aws/secretsmanager-encrypted secret that can never go cross-account, or a consumer whose identity policy was never granted even though the resource policy was. Knowing which in ninety seconds is what separates a five-minute incident from a two-hour one.

What problem this solves

A long-lived database password is a standing liability. It leaks through a log line, a .env committed by accident, a laptop image, an offboarded contractor who still remembers it. The only durable mitigation is to make the credential short-lived — rotate it on a schedule so that any copy an attacker holds expires on its own. Secrets Manager automates that, but the automation has sharp edges, and the edges are exactly where production breaks.

What breaks without getting this right: an engineer enables single-user rotation on a hot-path credential and every cycle produces a spike of authentication errors as forty services slowly notice the password changed; a rotation Lambda is dropped into the database VPC but never given a Secrets Manager endpoint, so it times out reaching the API and the secret sticks half-rotated in AWSPENDING; a central platform team shares a secret cross-account using the AWS-managed key and the consumer account gets AccessDenied on kms:Decrypt forever, because that key’s policy cannot be edited; a finishSecret step is non-idempotent, Secrets Manager retries it, and the staging labels desync. Each of these is perfectly diagnosable and each costs a team an afternoon the first time.

Who hits this: anyone running RDS/Aurora, DocumentDB, Redshift, RDS Proxy, or any credentialed third-party service at scale. It bites hardest on multi-account organisations (cross-account KMS is non-obvious), VPC-isolated databases (the networking is the silent killer), high-traffic hot paths (single-user rotation windows show up as real error budget), and custom integrations where AWS publishes no managed rotation function and you must write all four steps yourself.

To frame the whole field before the deep dive, here is every failure class this article covers, the question it forces, and the one place to look first:

Learning objectives

By the end of this article you can:

• Explain the staging-label state machine (AWSCURRENT / AWSPENDING / AWSPREVIOUS) and why rotation is a relabel, never an in-place edit.
• Implement the four-step rotation Lambda (createSecret → setSecret → testSecret → finishSecret) with each step correctly idempotent against Secrets Manager’s retries.
• Choose between single-user and alternating-user RDS rotation and explain exactly why alternating-user eliminates the invalid-credential window.
• Stand up the VPC, security-group, and Secrets Manager endpoint wiring a rotation Lambda needs to reach a private database without timing out.
• Author the IAM execution role, Lambda resource policy, and KMS grants rotation requires — and avoid the aws/secretsmanager-key cross-account trap.
• Write a custom rotator for a third-party credential where the provider issues the secret instead of accepting one, collapsing createSecret/setSecret safely.
• Share a secret cross-account so both the resource policy and the consumer’s identity policy allow, and rotation keeps flowing to consumers automatically.
• Integrate consumers with the caching client so they follow AWSCURRENT without rotation-window failures, and monitor rotation health and staleness with EventBridge and AWS Config.

Prerequisites & where this fits

You should already understand the Secrets Manager basics — that a secret is a named, versioned, KMS-encrypted blob you read with GetSecretValue, billed per secret per month plus per 10,000 API calls. You should be comfortable with IAM identity-based vs resource-based policies, KMS key policies, and the difference between an AWS-managed key (aws/secretsmanager) and a customer-managed key (CMK). Basic VPC literacy (private subnets, security groups, interface endpoints) and the ability to run aws CLI v2 and read JSON output are assumed.

This sits at the intersection of the Security and Databases tracks. It builds directly on the AWS Secrets Manager & Parameter Store Deep Dive (storage, naming, versioning fundamentals) and the AWS KMS Encryption Deep Dive (key policies, grants, envelope encryption — the cross-account half of this article is a KMS problem). On the database side it pairs with the AWS RDS & Aurora Deep Dive and especially RDS Proxy: Connection Pooling, Failover, IAM Auth, because RDS Proxy is what turns rotation cutover into a non-event. The cross-account sharing pattern leans on IAM Cross-Account Roles, External ID, Confused Deputy and Multi-Region KMS Keys & Envelope Encryption. The rotation engine itself is a Lambda running inside the database VPC.

A quick map of who owns what during a rotation incident, so you call the right person fast:

Core concepts

Five mental models make every later diagnosis obvious.

Rotation is a relabel, never an in-place edit. A secret holds one or more versions, each carrying one or more staging labels. Rotation creates a new version, parks it under AWSPENDING, proves it works, then atomically moves AWSCURRENT onto it. Applications reading AWSCURRENT (the default) never see the pending value until cutover completes — that is the entire reason rotation can be zero-downtime. If you ever find yourself “updating” a secret value during rotation, you are off the rails.

Secrets Manager drives your Lambda four times. It invokes your function once per step (createSecret, setSecret, testSecret, finishSecret), passing the step name and the new version’s ID (ClientRequestToken). Your function is a dispatcher on that field. Secrets Manager retries on failure, so every step must be idempotent — running twice must be safe.

RDS rotation has two strategies, and the choice is operational, not cosmetic. Single-user changes the password on the same user every cycle and has a brief window where the old credential is invalid. Alternating-user keeps two users and swaps between them, so the promoted credential always belonged to a user that already existed with a known-good password — no invalid window. The alternating strategy needs a second, elevated master secret to clone the user.

The rotation Lambda must reach two endpoints, and a VPC Lambda loses easy egress. It must reach the database (security-group access on the DB port) and the Secrets Manager API. A Lambda attached to a VPC has no default internet egress, so it needs either a Secrets Manager interface VPC endpoint or a NAT path to call the API. Forgetting this is the single most common rotation failure, and it presents as a bare timeout.

Cross-account sharing requires three independent allows. The owning account’s secret resource policy must grant the consumer; the KMS key must permit the consumer to Decrypt (and it must be a CMK — the managed key cannot be shared); and the consumer’s own identity-based policy must allow GetSecretValue and kms:Decrypt. All three must be true or it silently fails, because each account authorises independently.

The vocabulary in one table

Before the deep sections, pin down every moving part. The glossary repeats these for lookup; this table is the mental model side by side:

1. The versioning and staging-label model

Everything in rotation hinges on staging labels. A secret holds one or more versions, each carrying labels. Three are reserved and load-bearing:

• AWSCURRENT — the version applications get by default when they call GetSecretValue without specifying a version.
• AWSPENDING — the in-flight version being created and tested during rotation. It does not exist between rotations.
• AWSPREVIOUS — automatically applied to the version that was AWSCURRENT once rotation finishes, so you can roll back one generation.

A single version label points to one version at a time. Rotation is fundamentally the act of creating a new version labelled AWSPENDING, proving it works, then atomically moving AWSCURRENT onto it. Applications that always read AWSCURRENT (the default) never see the pending value until cutover completes. That is the whole reason rotation can be zero-downtime: the new credential is fully provisioned and tested before anything points production at it.

Mental model: rotation never edits a secret in place. It writes a new version, parks it under AWSPENDING, and only relabels AWSCURRENT at the very end. If you ever find yourself “updating” a secret value during rotation, you are off the rails.

The three reserved labels, what moves them, and what each means for callers:

The version-stage invariants you must never violate — break one and rotation desyncs:

Inspect the live label-to-version mapping at any time — this is your first diagnostic command in any rotation incident:

aws secretsmanager describe-secret --secret-id prod/payments/db-app \
  --query 'VersionIdsToStages'
# Healthy: one version → ["AWSCURRENT"], one → ["AWSPREVIOUS"], NO "AWSPENDING".
# A lingering "AWSPENDING" means a step failed mid-cycle.

2. The four-step rotation Lambda

Secrets Manager drives rotation by invoking your Lambda four times, passing a Step field in the event each time. Your function is a dispatcher on that field. The contract is fixed:

{
  "SecretId": "arn:aws:secretsmanager:us-east-1:111122223333:secret:prod/app/db-AbCdEf",
  "ClientRequestToken": "uuid-of-the-new-version",
  "Step": "createSecret"
}

The ClientRequestToken is the version ID of the new (AWSPENDING) version. What each step must do, and the idempotency rule that keeps retries safe:

The four steps as a state machine — what is true before and after each, and who acts:

The most common bug is non-idempotent steps. Secrets Manager retries; if createSecret blindly generates a new password every invocation, you desync the stored secret from what setSecret applied. Always check for an existing AWSPENDING first.

def create_secret(service_client, arn, token):
    # Source of truth is AWSCURRENT; we derive the new value from it.
    current = service_client.get_secret_value(SecretId=arn, VersionStage="AWSCURRENT")

    try:
        service_client.get_secret_value(
            SecretId=arn, VersionId=token, VersionStage="AWSPENDING"
        )
        # Pending already staged on a retry — do not regenerate.
        return
    except service_client.exceptions.ResourceNotFoundException:
        pass

    secret = json.loads(current["SecretString"])
    secret["password"] = service_client.get_random_password(
        PasswordLength=32, ExcludePunctuation=True
    )["RandomPassword"]

    service_client.put_secret_value(
        SecretId=arn,
        ClientRequestToken=token,
        SecretString=json.dumps(secret),
        VersionStages=["AWSPENDING"],
    )

The finishSecret step is equally precise — it is a relabel, not a write. Find the version currently holding AWSCURRENT (returning early if it is already token, i.e. a retry), then move the label:

def finish_secret(service_client, arn, token, current_version):
    if current_version == token:
        return  # Already promoted (retry).
    service_client.update_secret_version_stage(
        SecretId=arn,
        VersionStage="AWSCURRENT",
        MoveToVersionId=token,
        RemoveFromVersionId=current_version,
    )

For most database engines you do not write this yourself — AWS publishes managed rotation functions as serverless application templates. The managed functions you should reach for before writing custom code:

The dispatcher skeleton that ties the four steps together — note the explicit guard that rotation is even enabled and that the version is pending:

def lambda_handler(event, context):
    arn, token, step = event["SecretId"], event["ClientRequestToken"], event["Step"]
    client = boto3.client("secretsmanager")

    md = client.describe_secret(SecretId=arn)
    if not md.get("RotationEnabled"):
        raise ValueError(f"Rotation not enabled for {arn}")
    versions = md["VersionIdsToStages"]
    if token not in versions:
        raise ValueError(f"Version {token} has no stage for {arn}")
    if "AWSCURRENT" in versions[token]:
        return  # Already current — nothing to do.

    current_version = next(v for v, s in versions.items() if "AWSCURRENT" in s)
    {
        "createSecret": lambda: create_secret(client, arn, token),
        "setSecret":    lambda: set_secret(client, arn, token),
        "testSecret":   lambda: test_secret(client, arn, token),
        "finishSecret": lambda: finish_secret(client, arn, token, current_version),
    }[step]()

3. RDS rotation: single-user vs alternating-user

For RDS, RDS Proxy, DocumentDB, and Redshift, you choose a rotation strategy, and the choice has real operational consequences.

Single-user rotation changes the password on the same database user every cycle. Simple, but hazardous: between setSecret (which runs ALTER USER ... PASSWORD) and the moment your application picks up the new AWSCURRENT, any connection attempt with the old password fails. If your app caches the secret and only refreshes on auth failure, you get a brief window of errors every rotation. Acceptable for low-traffic or fault-tolerant apps; not for a hot path.

Alternating-user rotation (the “multi-user” strategy) is the production-grade choice. It clones the existing user into a second account and alternates between the two each cycle. Cycle N uses app_user; cycle N+1 rotates app_user_clone, swaps AWSCURRENT to it, and leaves the previous user fully valid. Because the newly promoted credential belongs to a user that already existed with a known-good password, there is no window where the current credential is invalid.

A fuller side-by-side, because the trade-offs go beyond the failure window:

Alternating-user requires a second secret — the master/superuser secret — referenced via masterarn, because cloning a user and granting its privileges needs elevated rights the application user does not have. The rotation Lambda authenticates with the master secret to provision the clone, then rotates the application secret.

# Enable alternating-user rotation on an RDS secret using the AWS-managed
# Postgres rotation function, every 30 days.
aws secretsmanager rotate-secret \
  --secret-id prod/payments/db-app \
  --rotation-lambda-arn arn:aws:lambda:us-east-1:111122223333:function:SecretsManagerRDSPostgreSQLRotationMultiUser \
  --rotation-rules '{"ScheduleExpression": "rate(30 days)", "Duration": "2h"}'

The application’s secret must declare which strategy it uses. A multi-user Postgres secret looks like this — every field below is load-bearing for the managed function:

{
  "engine": "postgres",
  "host": "payments.cluster-abc123.us-east-1.rds.amazonaws.com",
  "username": "app_user",
  "password": "current-password",
  "dbname": "payments",
  "port": 5432,
  "masterarn": "arn:aws:secretsmanager:us-east-1:111122223333:secret:prod/payments/db-master-XyZ"
}

The required and optional keys in an RDS secret JSON, and what the managed function does with each:

The schedule controls in --rotation-rules, and how each behaves:

Put RDS Proxy in front of this. The proxy keeps a warm connection pool and integrates natively with Secrets Manager, so when AWSCURRENT flips it picks up the new credential without each application instance re-reading the secret. Combined with alternating-user rotation, cutover becomes a non-event — covered in depth in RDS Proxy: Connection Pooling, Failover, IAM Auth.

4. VPC, networking, and KMS permissions

This is where rotation breaks in practice, and the failure is always the same flavour: the Lambda times out with no useful error. The function has to reach two endpoints — the database and the Secrets Manager API — and both paths must be open.

If your database is in private subnets, the rotation Lambda must attach to the same VPC with security-group access to the DB port. But a VPC Lambda loses default internet egress and still needs to call Secrets Manager. The two ways to give it API access, side by side:

The two endpoints the rotation Lambda must reach, and what blocks each:

resource "aws_vpc_endpoint" "secretsmanager" {
  vpc_id              = var.vpc_id
  service_name        = "com.amazonaws.${var.region}.secretsmanager"
  vpc_endpoint_type   = "Interface"
  subnet_ids          = var.private_subnet_ids
  security_group_ids  = [aws_security_group.sm_endpoint.id]
  private_dns_enabled = true
}

# The rotation Lambda's SG must be allowed inbound on the DB port.
resource "aws_security_group_rule" "db_from_rotator" {
  type                     = "ingress"
  from_port                = 5432
  to_port                  = 5432
  protocol                 = "tcp"
  security_group_id        = aws_security_group.rds.id
  source_security_group_id = aws_security_group.rotation_lambda.id
}

For KMS: if the secret is encrypted with a customer-managed key (CMK) rather than the AWS-managed aws/secretsmanager key — and at scale it should be, for cross-account and auditability reasons — the rotation Lambda’s execution role needs kms:Decrypt and kms:GenerateDataKey on that key, and the key policy must permit Secrets Manager. The role also needs the standard rotation permissions plus secretsmanager:GetRandomPassword.

The exact IAM actions the rotation execution role needs, and why each is there:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:DescribeSecret",
        "secretsmanager:GetSecretValue",
        "secretsmanager:PutSecretValue",
        "secretsmanager:UpdateSecretVersionStage"
      ],
      "Resource": "arn:aws:secretsmanager:us-east-1:111122223333:secret:prod/*",
      "Condition": {
        "StringEquals": { "aws:ResourceTag/rotation": "managed" }
      }
    },
    {
      "Effect": "Allow",
      "Action": "secretsmanager:GetRandomPassword",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": ["kms:Decrypt", "kms:GenerateDataKey"],
      "Resource": "arn:aws:kms:us-east-1:111122223333:key/<cmk-id>"
    }
  ]
}

Secrets Manager invokes your function on your behalf, so the Lambda’s resource policy must grant lambda:InvokeFunction to secretsmanager.amazonaws.com. The AWS console wires this automatically; in IaC you add it explicitly:

resource "aws_lambda_permission" "allow_secretsmanager" {
  statement_id  = "AllowSecretsManagerInvoke"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.rotator.function_name
  principal     = "secretsmanager.amazonaws.com"
}

The three policies rotation touches — confusing them is a common time-sink, so keep them straight:

5. A custom rotator for non-RDS credentials

For third-party API keys, a MongoDB Atlas user, or an internal service token, there is no managed function — you implement the four steps against the provider’s API. The shape is identical; only setSecret and testSecret change. The key discipline: many providers do not let you set a key value — they generate and return it. There, createSecret and setSecret collapse — you call the provider’s “create credential” API once, store the result as AWSPENDING, and make setSecret a near no-op. Clean up the old credential only in finishSecret, never before — deleting the old key before the new one is promoted is how you cause an outage.

The two provider archetypes and how the four steps adapt to each:

def set_secret(service_client, arn, token):
    pending = json.loads(
        service_client.get_secret_value(
            SecretId=arn, VersionId=token, VersionStage="AWSPENDING"
        )["SecretString"]
    )
    # Idempotent: only create at the provider if this pending key is not live yet.
    if not provider_key_exists(pending["api_key_id"]):
        provider_create_key(pending["api_key_id"], pending["api_key_secret"])

def test_secret(service_client, arn, token):
    pending = json.loads(
        service_client.get_secret_value(
            SecretId=arn, VersionId=token, VersionStage="AWSPENDING"
        )["SecretString"]
    )
    resp = provider_authenticated_call(pending["api_key_secret"])
    if resp.status_code != 200:
        raise ValueError("Pending credential failed validation")

Custom-rotator pitfalls that cause real outages, and the discipline that avoids each:

Deploy and schedule a custom rotator the same way, pointing --rotation-lambda-arn at your function. Validate one step at a time before enabling the schedule — rotate-secret runs all four immediately, and you do not want to discover a broken finishSecret in production.

6. Cross-account sharing

A central account often owns secrets that spoke-account workloads consume — a shared database credential, a partner API key. Two things must both be true or it silently fails: a resource policy on the secret, and KMS permissions on the encrypting key. And on the consumer side, the principal needs its own identity policy. Three independent allows, evaluated in three different places.

The three grants cross-account sharing requires, where each lives, and the symptom when it is the one missing:

The resource policy grants the consumer account access to the secret:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "AWS": "arn:aws:iam::444455556666:root" },
      "Action": ["secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret"],
      "Resource": "*"
    }
  ]
}

The trap: you cannot use the AWS-managed aws/secretsmanager key for cross-account access — its key policy is not editable to permit another account. Encrypt cross-account secrets with a customer-managed CMK and grant the consumer account kms:Decrypt in the key policy:

{
  "Sid": "AllowConsumerDecrypt",
  "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::444455556666:root" },
  "Action": "kms:Decrypt",
  "Resource": "*",
  "Condition": {
    "StringEquals": {
      "kms:ViaService": "secretsmanager.us-east-1.amazonaws.com"
    }
  }
}

Why aws/secretsmanager cannot go cross-account, contrasted with a CMK — this single distinction is the most common cross-account failure:

On the consumer side, the IAM principal still needs its own identity-based policy allowing secretsmanager:GetSecretValue and kms:Decrypt against those ARNs — resource policy and identity policy must both allow, since each account authorizes independently. Confirm the full chain with the policy simulator before you trust it:

# From the CONSUMER account: does this role actually get the secret?
aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::444455556666:role/app-role \
  --action-names secretsmanager:GetSecretValue kms:Decrypt \
  --resource-arns arn:aws:secretsmanager:us-east-1:111122223333:secret:shared/partner-api-XyZ \
  --query 'EvaluationResults[].{action:EvalActionName, decision:EvalDecision}'

Rotation keeps running in the owning account; consumers always resolve AWSCURRENT and so follow rotation automatically. The deeper cross-account mechanics (external ID, confused-deputy, session policies) are in IAM Cross-Account Roles, External ID, Confused Deputy; the KMS half is in AWS KMS Encryption Deep Dive.

7. Application integration without rotation-window failures

The cache makes or breaks the consumer experience. Calling GetSecretValue on every request is slow and hits API throttling. The right pattern is the AWS caching client (Java, Python, Go, .NET, or the Lambda extension), which caches AWSCURRENT in memory and refreshes on an interval.

from aws_secretsmanager_caching import SecretCache, SecretCacheConfig
import boto3

client = boto3.client("secretsmanager")
cache = SecretCache(config=SecretCacheConfig(secret_refresh_interval=3600), client=client)

def get_connection():
    secret = json.loads(cache.get_secret_string("prod/payments/db-app"))
    return connect(secret)  # On auth failure, force-refresh and retry once.

The integration approaches ranked from worst to best, with the trade-off each makes:

Two rules keep you safe across rotations:

1. Refresh on auth failure, not only on a timer. If a DB connection is rejected, invalidate the cache entry, re-read AWSCURRENT, and retry once. With alternating-user rotation this almost never fires, but it is your safety net for single-user.
2. Never pin a VersionId. Read AWSCURRENT (the default). Pinning a version means you never follow rotation — the cardinal sin.

The caching-client knobs and how to reason about each:

The Lambda extension is cleanest for serverless consumers: it runs a local HTTP cache with built-in TTL, so your function makes a localhost call instead of hitting the Secrets Manager API.

8. Monitoring rotation health and staleness

Rotation that fails silently is worse than no rotation, because you believe you are covered. Wire three signals:

• RotationFailed — Secrets Manager emits this event on failure; route it through EventBridge to SNS.
• RotationSucceeded — track it to detect absence. A secret that has not rotated on schedule is the real risk, visible only by alarming on staleness.
• Last-rotated age — a scheduled check comparing LastRotatedDate against the schedule, alerting on overdue secrets.

The signals to wire, what each tells you, and how to surface it:

{
  "source": ["aws.secretsmanager"],
  "detail-type": ["AWS Service Event via CloudTrail"],
  "detail": {
    "eventName": ["RotationFailed"]
  }
}

For staleness, AWS Config is the durable approach. Its managed rules secretsmanager-rotation-enabled-check and secretsmanager-secret-periodic-rotation flag secrets without rotation enabled or not rotated within a maximum age — far better than a custom script you will forget to maintain. Pair this with the CloudTrail/CloudWatch foundations in CloudWatch & CloudTrail Observability Deep Dive.

Architecture at a glance

The diagram traces a single rotation across two accounts, left to right. On the far left, the consumer account (444455556666) runs the app — or, better, RDS Proxy — which only ever reads AWSCURRENT through the caching client, and a consumer IAM role that must carry its own GetSecretValue + kms:Decrypt grant. Next is the owner account (111122223333), which holds the secret (with its AWSCURRENT/AWSPENDING versions), the resource policy that grants the consumer, and the customer-managed CMK — explicitly not aws/secretsmanager, because the managed key cannot be shared. The third zone is the rotation tier living inside the database VPC: the four-step rotation Lambda and the Secrets Manager interface VPC endpoint that lets that VPC-bound Lambda reach the SM API at all. The right-most zone is the data tier in private subnets: the Aurora PostgreSQL cluster on 5432 with its two alternating users, and the elevated master secret whose superuser clones the application user.

Follow the flows: the consumer calls GetSecretValue and receives whatever AWSCURRENT points at; Secrets Manager invokes the rotation Lambda through its four steps; the Lambda reaches the database on 5432 to ALTER USER, then writes the new version back with PutSecretValue. The five numbered badges sit on exactly the hops that stall in practice — the Lambda losing its route to the DB (1), the missing SM endpoint (2), the master secret needed to clone (3), the un-shareable managed key (4), and the consumer’s missing identity grant (5) — and the legend narrates each as symptom, how to confirm, and the fix. Read the badges as the failure map laid directly over the architecture.

Real-world scenario

A payments platform team — call them NorthPay — ran 40+ microservices against an Aurora PostgreSQL cluster, all sharing one application credential rotated single-user every 7 days. Every rotation produced a 20-to-90 second spike of 500s as services hit auth failures and slowly refreshed their caches. The on-call playbook literally said “ignore the Thursday 02:00 error spike” — which is exactly how a real incident later got ignored for eleven minutes, because operators had been trained to wave off the Thursday alarm.

The constraints were hard: they could not coordinate a synchronized restart of 40 services, and the payments hot path’s error budget (a 99.95% SLO, ~21 minutes/month) was being burned by the rotation spike alone — four cycles a month at up to 90 seconds each was a meaningful fraction of the entire budget, spent on a self-inflicted event. The fix had two moves. First, they switched the secret to alternating-user rotation with a master secret, so the promoted credential always belonged to a user that already existed with a valid password — eliminating the invalid-credential window. Second, they put RDS Proxy in front of the cluster so cutover happened once, at the proxy, instead of independently in 40 connection pools.

aws rds create-db-proxy \
  --db-proxy-name payments-proxy \
  --engine-family POSTGRESQL \
  --auth '[{"AuthScheme":"SECRETS","SecretArn":"arn:aws:secretsmanager:us-east-1:111122223333:secret:prod/payments/db-app-XyZ","IAMAuth":"DISABLED"}]' \
  --role-arn arn:aws:iam::111122223333:role/payments-proxy-role \
  --vpc-subnet-ids subnet-aaa subnet-bbb

There was a second-order problem the team only found in staging: the rotation Lambda had originally been deployed outside the VPC (it had worked because the dev database was publicly reachable), and moving it into the production VPC immediately broke it with a bare timeout. The cause was the classic missing Secrets Manager interface endpoint — the VPC Lambda could now reach the database but had lost its path to the SM API. Adding com.amazonaws.us-east-1.secretsmanager with private DNS in the Lambda’s subnets fixed it in one change. They also moved the secret off aws/secretsmanager onto a CMK in the same motion, because a sibling analytics account needed read access and the managed key could never grant it.

The before/after, measured over a quarter:

After the change, rotation-window errors went to zero, and the “ignore the Thursday spike” line was deleted from the runbook — the real win, because a runbook that trains operators to ignore alerts is a latent incident.

Advantages and disadvantages

Automatic rotation is the right default for any credential that can rotate, but it is not free of cost or risk. The honest trade-off:

When each side dominates: for production database credentials on a hot path, the advantages are decisive — alternating-user + RDS Proxy turns the biggest risk (the cutover window) into a non-event, and the cost is rounding error against the cost of a leaked standing password. For a rarely-used, low-value, easily-revoked credential, the disadvantages can outweigh the benefit — a rotation Lambda you must maintain for a secret nobody attacks is overhead; a long manual-rotation interval may be enough. For third-party credentials with hard rate limits or fragile APIs, weigh the custom-rotator engineering against simply rotating manually on a calendar.

Hands-on lab

This lab stands up single-user rotation on an RDS PostgreSQL secret end to end, triggers a manual cycle, verifies the staging labels moved, and tears everything down. It assumes an existing RDS PostgreSQL instance reachable from where you run the rotation function; for a fully free-tier path, use a db.t3.micro (free tier eligible for 12 months) in a public subnet so you can skip the VPC endpoint for the lab (never do this in production).

The lab at a glance:

Step 1 — create the secret with the engine metadata the managed function needs:

aws secretsmanager create-secret \
  --name lab/rotation/pg-app \
  --secret-string '{
    "engine":"postgres",
    "host":"lab-db.abc123.us-east-1.rds.amazonaws.com",
    "username":"app_user",
    "password":"InitialPassw0rd!",
    "dbname":"appdb",
    "port":5432
  }'

Step 2 — deploy the managed Postgres single-user rotation function from the Serverless Application Repository (the console “Edit rotation” wizard does this for you; the CLI path uses aws serverlessrepo create-cloud-formation-change-set). For the lab, the console wizard is fastest: open the secret → Rotation → Edit rotation → Create a new Lambda function → choose the single-user template.

Step 3 — grant Secrets Manager permission to invoke the function (the console does this; shown here for completeness):

aws lambda add-permission \
  --function-name SecretsManagerRotationLabFn \
  --statement-id AllowSM \
  --action lambda:InvokeFunction \
  --principal secretsmanager.amazonaws.com

Step 4 — enable rotation pointing at the function, on a 30-day schedule:

aws secretsmanager rotate-secret \
  --secret-id lab/rotation/pg-app \
  --rotation-lambda-arn arn:aws:lambda:us-east-1:111122223333:function:SecretsManagerRotationLabFn \
  --rotation-rules '{"ScheduleExpression":"rate(30 days)"}'

Step 5 — trigger a manual cycle now (runs all four steps immediately):

aws secretsmanager rotate-secret --secret-id lab/rotation/pg-app

Step 6 — verify the labels moved. This is the moment of truth:

aws secretsmanager describe-secret --secret-id lab/rotation/pg-app \
  --query '{Rotated:LastRotatedDate, Versions:VersionIdsToStages}'
# Expect: LastRotatedDate just now; one version AWSCURRENT, one AWSPREVIOUS, NO AWSPENDING.

Step 7 — confirm the live credential authenticates:

CRED=$(aws secretsmanager get-secret-value --secret-id lab/rotation/pg-app \
  --query SecretString --output text)
PGPASSWORD=$(echo "$CRED" | jq -r .password) \
  psql -h $(echo "$CRED" | jq -r .host) -U app_user -d appdb -c '\conninfo'
# "You are connected to database appdb" → rotation produced a working credential.

Step 8 — tear down so you stop paying for the secret and the instance:

aws secretsmanager delete-secret --secret-id lab/rotation/pg-app \
  --force-delete-without-recovery
aws lambda delete-function --function-name SecretsManagerRotationLabFn
# Delete the RDS instance from the console or:
aws rds delete-db-instance --db-instance-identifier lab-db \
  --skip-final-snapshot --delete-automated-backups

What each teardown action stops costing you:

Common mistakes & troubleshooting

This is the section you keep open at 02:00. Each row is a real failure mode: the symptom, the root cause, the exact command or console path to confirm it, and the fix.

The decision table for the most common ambiguous signal — “rotation isn’t working”:

Best practices

• Use a managed rotation function for any supported database; write custom code only for targets AWS does not cover. The managed functions are battle-tested and idempotent.
• Choose alternating-user for any hot path, and reserve single-user for low-traffic, fault-tolerant credentials where a brief window is acceptable.
• Always front shared database credentials with RDS Proxy so cutover happens once, at the proxy, instead of independently in every connection pool.
• Put the rotation Lambda in the database VPC with a Secrets Manager interface endpoint and security-group access to the DB port — and verify both paths before enabling the schedule.
• Encrypt with a customer-managed CMK, never aws/secretsmanager, for anything that might go cross-account, needs key-level audit, or needs an editable key policy.
• Make every step idempotent: guard createSecret on an existing AWSPENDING, and finishSecret on an already-current version. Secrets Manager will retry.
• Never delete the old credential before finishSecret in custom rotators — the old one must stay valid until cutover completes.
• Consumers read AWSCURRENT only, through the caching client, refreshing on auth failure and never pinning a VersionId.
• Monitor for silence, not just failure: alarm on RotationFailed, and separately alarm on absence of RotationSucceeded / stale LastRotatedDate via AWS Config.
• Validate one step at a time in a non-prod environment before enabling the schedule; rotate-secret runs all four immediately.
• Tag secrets for scoped IAM (e.g. rotation=managed) so the rotator role can be least-privilege via a ResourceTag condition.
• Keep the rotation window (Duration) generous for slow or rate-limited targets so a single retry doesn’t blow the window.

Security notes

Rotation is a security control, so its own posture must be tight. The least-privilege and isolation rules that matter:

The encryption and identity story end to end: the secret is envelope-encrypted under a CMK; the rotator decrypts with kms:Decrypt scoped to that key; consumers decrypt only through secretsmanager.<region>.amazonaws.com via the kms:ViaService condition; and every read is a CloudTrail event you can alarm on. For the deeper KMS model see AWS KMS Encryption Deep Dive and Multi-Region KMS Keys & Envelope Encryption; for the IAM evaluation order behind the three-place authorization, IAM Fundamentals: Users, Roles, Policies, Evaluation and IAM Least Privilege & Permission Boundaries.

Cost & sizing

Secrets Manager pricing is simple but adds up at scale; rotation introduces Lambda and (usually) CMK and endpoint costs on top. The drivers:

Right-sizing guidance and rough monthly figures (INR at ~₹84/USD, indicative):

The free-tier and cost-control levers worth knowing:

The cost of not rotating — a leaked standing credential leading to data exfiltration — dwarfs all of the above. Treat rotation cost as insurance premium, not overhead.

Interview & exam questions

Q1. Walk me through the four steps of a rotation Lambda. createSecret generates the new value and stores it as AWSPENDING; setSecret applies it to the backing service; testSecret authenticates with the pending value; finishSecret moves the AWSCURRENT label onto the pending version. Each step must be idempotent because Secrets Manager retries.

Q2. Why is alternating-user rotation safer than single-user for a hot path? Single-user changes the password on one user, creating a window where the old cached credential is invalid until the app refreshes. Alternating-user swaps between two users, so the newly promoted credential belongs to a user that already existed with a known-good password — there is no invalid window.

Q3. A rotation Lambda for a private RDS instance times out with no useful error. What’s the most likely cause? It’s in the VPC and can reach the database but has no path to the Secrets Manager API — it’s missing a Secrets Manager interface VPC endpoint (or a NAT route). The classic tell is a timeout before any step logs.

Q4. Why can’t you use the aws/secretsmanager key for a cross-account secret? Its key policy is AWS-managed and not editable, so you cannot grant another account kms:Decrypt. Cross-account secrets must use a customer-managed CMK whose key policy you can edit to permit the consumer account.

Q5. A consumer account has a resource policy granting it the secret but still gets AccessDenied. Why? Authorization is evaluated independently in each account. The consumer’s own identity-based policy must also allow secretsmanager:GetSecretValue (and kms:Decrypt). Resource policy and identity policy must both allow.

Q6. What is masterarn and when is it required? It points to an elevated master/superuser secret. Alternating-user rotation needs it because cloning a user and granting its privileges requires rights the application user does not have; the Lambda authenticates with the master secret to provision the clone.

Q7. Why must rotation steps be idempotent, and where does it bite hardest? Secrets Manager retries on failure. If createSecret regenerates a password on every retry, the stored secret desyncs from what setSecret applied. Guard createSecret on an existing AWSPENDING and finishSecret on an already-current version.

Q8. How do you keep applications from failing during the rotation window? Read AWSCURRENT via the caching client (never pin a VersionId), refresh the cache on auth failure, prefer alternating-user rotation, and front shared DB credentials with RDS Proxy so cutover happens once at the proxy.

Q9. How do you detect a secret that silently stopped rotating? Alarm on the absence of RotationSucceeded, not just on RotationFailed. Use AWS Config rules secretsmanager-rotation-enabled-check and secretsmanager-secret-periodic-rotation, and check LastRotatedDate against the schedule.

Q10. In a custom rotator where the provider issues the key, when do you delete the old credential? Only in finishSecret, after AWSCURRENT has moved. Deleting the old key in createSecret or setSecret kills it before promotion and causes an outage.

Q11. What lingering AWSPENDING tells you, and how to recover. A step failed mid-cycle — usually setSecret/testSecret couldn’t reach the DB, or a permission was missing. Confirm with describe-secret’s VersionIdsToStages, fix the underlying cause (network/permission), then re-trigger rotate-secret.

Q12. Which IAM permissions does the rotation execution role need beyond the obvious secret actions? secretsmanager:GetRandomPassword (on *), kms:Decrypt and kms:GenerateDataKey on the CMK, plus the VPC ENI permissions (via the AWS managed Lambda-VPC policy) when it runs in a VPC.

These map to AWS Certified Security – Specialty (data protection, key management, incident response) and AWS Certified Solutions Architect – Professional (secure multi-account design). The cross-account KMS and resource-policy material is a recurring Security-Specialty theme.

Quick check

1. Which staging label do applications get by default from GetSecretValue, and which one only exists during a rotation?
2. You enable rotation on a private RDS secret and the Lambda times out before any step logs. What’s missing?
3. A spoke account has a secret resource policy granting it access but still gets AccessDenied on GetSecretValue. Name the two things that must also be true.
4. Why does alternating-user rotation eliminate the rotation-window auth errors that single-user can cause?
5. In a custom rotator for a provider that issues the key, at which step do you delete the old credential, and why not earlier?

Answers

1. Applications get AWSCURRENT by default; AWSPENDING exists only during a rotation cycle (and a lingering one means a step failed).
2. A Secrets Manager interface VPC endpoint (or a NAT route) in the Lambda’s subnets — the VPC Lambda can reach the DB but has no path to the SM API.
3. The encrypting key must be a customer-managed CMK granting the consumer kms:Decrypt (not aws/secretsmanager), and the consumer’s own identity-based policy must allow GetSecretValue + kms:Decrypt.
4. Because the promoted credential belongs to a second user that already existed with a known-good password, so there is never a moment where the current credential is invalid while the app catches up.
5. In finishSecret, after AWSCURRENT has moved — deleting it earlier kills the credential before promotion and causes an outage.

Glossary

• Staging label — a movable pointer (AWSCURRENT, AWSPENDING, AWSPREVIOUS, or custom) that points to exactly one version of a secret at a time.
• AWSCURRENT — the staging label for the live version returned by default from GetSecretValue; what consumers should always read.
• AWSPENDING — the transient label on the new version being created and tested during a rotation cycle; absent between rotations.
• AWSPREVIOUS — the label automatically applied to the version that was AWSCURRENT before the last rotation, enabling one-generation rollback.
• Rotation Lambda — the function Secrets Manager invokes four times per cycle to create, set, test, and finish a new credential version.
• Step — the field in the rotation event (createSecret/setSecret/testSecret/finishSecret) that your function dispatches on.
• Single-user rotation — strategy that changes the password on one database user each cycle; simple but has a brief invalid-credential window.
• Alternating-user rotation — strategy that swaps between two database users each cycle, eliminating the invalid-credential window; needs a master secret.
• masterarn — a secret field pointing to the elevated master/superuser secret used to clone the user in alternating-user rotation.
• Interface VPC endpoint — an ENI-backed private path (com.amazonaws.<region>.secretsmanager) letting a VPC-bound Lambda reach the Secrets Manager API without internet egress.
• Customer-managed key (CMK) — a KMS key whose key policy you control; the only key type shareable cross-account for secret encryption.
• aws/secretsmanager — the AWS-managed default encryption key; free but with an uneditable policy, so it cannot be shared cross-account.
• Resource policy — a policy attached to the secret itself, granting other accounts/principals access (the cross-account half of the grant).
• kms:ViaService — a KMS condition key restricting use of the key to requests coming through a named service (e.g. Secrets Manager).
• RDS Proxy — a managed connection pool that holds the secret and integrates with Secrets Manager, so rotation cutover happens once at the proxy.
• Caching client — the AWS SDK helper (or Lambda extension) that caches AWSCURRENT in memory and refreshes on a TTL, cutting API calls and following rotation.

Next steps

• Master the storage and naming fundamentals these mechanics build on in the AWS Secrets Manager & Parameter Store Deep Dive.
• Go deep on the encryption layer — key policies, grants, envelope encryption — in the AWS KMS Encryption Deep Dive and Multi-Region KMS Keys & Envelope Encryption.
• Make rotation cutover a non-event with RDS Proxy: Connection Pooling, Failover, IAM Auth, backed by the AWS RDS & Aurora Deep Dive.
• Nail the cross-account trust model in IAM Cross-Account Roles, External ID, Confused Deputy and the evaluation order in IAM Least Privilege & Permission Boundaries.
• Wire the alerting that catches silent rotation failure using the CloudWatch & CloudTrail Observability Deep Dive.