Change Data Capture with DynamoDB Streams: Lambda Triggers, EventBridge Pipes, and Exactly-Once Processing

Change data capture off DynamoDB looks deceptively simple — turn on a stream, point a Lambda at it, ship the deltas somewhere. The trouble starts the first time a single poison record wedges a shard, an OpenSearch index drifts out of sync, or someone discovers that “exactly-once” was never a property the stream promised. DynamoDB Streams give you an ordered, at-least-once feed of item-level mutations. Everything downstream — ordering across reprocessing, deduplication, partial-batch recovery — is yours to engineer. This is how to build that pipeline so it survives throttles, redrives, and the occasional malformed item, using Lambda event source mappings where you need code and EventBridge Pipes where you do not.

The reason this is hard is that the failure modes are invisible until load. A pipeline that processes ten records a minute in dev never reveals that you have no idempotency, no bisect, no DLQ — and then a flash sale drives one partition key hot, the iterator age on that shard climbs past forty minutes, a malformed item silently blocks every update behind it, and your “real-time” search index is now lying about stock. The platform did exactly what you configured; you simply never configured it for the bad day. So this article treats DynamoDB CDC the way a senior engineer learns it the hard way: not “how do I move a change” but “how do I move a change such that it survives replay, isolates poison, preserves per-key order, and never silently drops data.”

By the end you will know which of the four delivery surfaces (native Stream + Lambda ESM, native Stream + EventBridge Pipes, Kinesis Data Streams for DynamoDB, or a glue Lambda fan-out) fits a given freshness/retention/ordering requirement; exactly which event-source-mapping knobs (BatchSize, ParallelizationFactor, MaximumRecordAgeInSeconds, BisectBatchOnFunctionError, ReportBatchItemFailures) prevent a stalled shard; and how to make every downstream write idempotent so at-least-once delivery is a non-event. Because this is a reference you will return to mid-incident, the knobs, the error/limit numbers, the view types, and the symptom→cause→confirm→fix playbook are all laid out as scannable tables — read the prose once, then keep the tables open when the iterator age starts climbing.

What problem this solves

A relational database hands you a transaction log and a dozen mature CDC tools that tail it. DynamoDB hands you a Stream: an ordered, append-only feed of every INSERT, MODIFY, and REMOVE, retained for 24 hours, partitioned to match your table’s shards. That is a genuine gift — you get item-level deltas with per-partition-key ordering and no triggers to write — but it is a primitive, not a pipeline. The delivery contract is at-least-once, the retention is hard-capped at 24 hours, and the ordering guarantee is per partition key only, never table-wide. Everything that makes a production CDC pipeline trustworthy sits in the gap between that primitive and your sinks.

What breaks without this knowledge is depressingly consistent. A team points a single Lambda at the stream with default settings, fans out to OpenSearch + S3 + a warehouse from that one function, and ships it. Then: (1) a malformed item throws, the whole batch retries forever, and the shard stalls — every change behind the poison record is blocked for hours; (2) a redrive replays a batch and the search index double-applies a delete, or a stale replay overwrites a newer document; (3) one slow sink (OpenSearch under merge pressure) stalls the shared function, so the data lake and warehouse go stale too; (4) a record that needed reprocessing ages past the 24-hour window before anyone noticed, and it is gone for good. None of these are exotic. All of them are the default outcome of treating the stream as if it were exactly-once, infinitely retained, and globally ordered.

Who hits this: anyone projecting DynamoDB state into a search index, a cache, a data lake, an audit trail, or another service. It bites hardest on high-write, hot-key workloads (a single partition going hot starves the one-poller-per-shard model), multi-sink fan-out (coupling sinks behind one function), and first-time CDC builders who assume the stream gives them guarantees it explicitly does not. The fix is almost never “make the stream stronger” — the stream is what it is — it is “engineer idempotency, bisect, DLQ, and bus-level fan-out around it.” Here is the whole field at a glance: every guarantee the stream gives you, every one it withholds, and where you make up the difference.

Learning objectives

By the end of this article you can:

• Choose between the four CDC delivery surfaces — native Stream + Lambda ESM, native Stream + EventBridge Pipes, Kinesis Data Streams for DynamoDB, and a glue-Lambda fan-out — by retention, ordering, freshness, and code-vs-config requirements.
• Enable a stream with the correct StreamViewType for the consumer’s job, and explain what each of the four view types lets you compute (diffs, soft-delete capture, invalidation).
• Tune a Lambda event source mapping deliberately: BatchSize, MaximumBatchingWindowInSeconds, ParallelizationFactor, MaximumRetryAttempts, MaximumRecordAgeInSeconds, and the starting position — and name the trade-off each one makes.
• Stop a stalled shard by combining ReportBatchItemFailures (precise per-record checkpointing) with BisectBatchOnFunctionError (the safety net for unhandled exceptions), and read iterator age as the leading indicator.
• Build filter-enrich-route pipelines with EventBridge Pipes — pushing selectivity into the filter stage and reshaping with a target transformer — and know when to keep a Lambda ESM instead.
• Engineer exactly-once downstream semantics as an idempotent write: conditional-on-SequenceNumber in DynamoDB, external versioning in OpenSearch, deterministic pk#sk#seq keys elsewhere.
• Operate the pipeline: alarm on iterator age / ApproximateAgeOfOldestRecord, evict poison records to a DLQ while they are still readable, and run a deliberate, reviewed redrive from shardId + sequence range.

Prerequisites & where this fits

You should already understand DynamoDB’s data model — partition keys, sort keys, items, and that a table’s throughput is sharded by partition key. Familiarity with the basics of Amazon DynamoDB, In Depth: Tables, Keys, Capacity Modes, Indexes & Streams is assumed — this article picks up where the “Streams” section there ends and goes deep on consuming them. You should be comfortable with AWS Lambda at the level of AWS Lambda, In Depth: Runtimes, Triggers, Layers, Concurrency & Every Setting, reading IAM-scoped CLI/Terraform, and JSON event shapes.

This sits in the event-driven / serverless integration track. It is downstream of table design — if your partition keys create hot shards, no consumer tuning saves you, so DynamoDB Single-Table Design: Modeling Access Patterns, GSIs, and Hot Partition Avoidance is upstream of everything here. It pairs tightly with Designing Event-Driven Architectures with Amazon EventBridge: Buses, Rules, Schemas, and Archive/Replay (the bus you fan out to) and Resilient Messaging with SQS and SNS: Fan-Out, FIFO Ordering, DLQs, and Poison-Message Handling (the DLQ and poison-handling patterns reused wholesale). The “land it in a lake” sink is the front half of A Structured Logging Pipeline on AWS: JSON Logs, CloudWatch Metric Filters, and Firehose to OpenSearch.

A quick map of who owns what during an incident, so you escalate to the right layer fast:

Core concepts

Six mental models make every later decision obvious.

A Stream is an ordered, append-only log, sharded like the table. Records for a given partition key always land in the same shard lineage and are delivered in commit order. That per-partition ordering is the single most important property to internalize — there is no global ordering across the table, only within a partition key. Shard topology mirrors the table’s partition structure and changes as the table splits/merges partitions, so your consumer must tolerate shards appearing and closing.

Delivery is at-least-once; “exactly-once” is something you build downstream. Redrives, bisect retries, and poller restarts all mean a sink can see the same change more than once. There is no setting that makes the stream exactly-once. The good news: every record hands you a stable item key and a monotonic SequenceNumber — exactly the tools an idempotent write needs.

Retention is 24 hours, hard. A record not consumed within 24 hours is gone. This drives two rules: set MaximumRecordAgeInSeconds below 24h so poison records are evicted to a DLQ while still readable, and treat S3 (fed via Firehose) as your replayable system of record once the window rolls off — the stream cannot backfill history.

The poller is managed; you tune it, you don’t write it. A Lambda event source mapping (ESM) is a poller AWS runs for you. You never write the GetShardIterator/GetRecords loop — you set BatchSize, ParallelizationFactor, batching window, retries, and failure handling, and the platform polls, batches, invokes, and checkpoints.

A stalled shard is the master failure mode. By default, if your function throws on a batch, the ESM retries the entire batch, and because the shard checkpoint has not advanced, it keeps retrying until success, age-out, or retry-exhaustion. One un-processable record blocks everything behind it. You see it as a climbing iterator age. Two mechanisms fix it: partial-batch responses and bisect-on-error.

Fan-out belongs at the bus, not in one function. If a single mega-Lambda writes to OpenSearch + S3 + a warehouse, one slow sink stalls the shard for all of them. Land changes on an EventBridge bus (often via a Pipe) and let independent rules deliver to each sink with its own retry budget and DLQ.

The vocabulary in one table

Pin down every moving part before the deep sections; the glossary repeats these for lookup.

Delivery surfaces: native Streams, Pipes, Kinesis, or glue Lambda

Before any tuning, pick the right surface. Four options carry DynamoDB changes, and they differ on retention, ordering, dedup, freshness, and how much code you own. Choosing wrong here is the most expensive mistake in the article — you cannot tune your way out of a surface that doesn’t give you the retention or ordering you need.

The native Stream’s headline advantage is strict per-key ordering and no duplication at the source; its costs are the 24-hour cap and a practical limit on how many consumers can read it efficiently. Kinesis Data Streams for DynamoDB inverts that: configurable retention up to a year and clean multi-consumer fan-out, at the price of best-effort ordering and possible duplicates — so you must reorder and dedupe on a sequence number. The trade is real and worth stating plainly:

Enable the Kinesis surface when you need any of: retention beyond 24 hours, several independent teams consuming the same changes, or direct Firehose delivery to a lake without a Lambda in the path. Stay on the native Stream when strict per-key ordering matters most or your consumer count is small. A quick decision table:

Stream internals: shards, records, and view types

A DynamoDB Stream is an ordered, append-only log of item-level changes, retained for 24 hours, composed of shards whose topology mirrors the table’s partition structure. Enable it and choose a StreamViewType, which controls what each record carries:

aws dynamodb update-table \
  --table-name orders \
  --stream-specification StreamEnabled=true,StreamViewType=NEW_AND_OLD_IMAGES

resource "aws_dynamodb_table" "orders" {
  name             = "orders"
  hash_key         = "pk"
  billing_mode     = "PAY_PER_REQUEST"
  stream_enabled   = true
  stream_view_type = "NEW_AND_OLD_IMAGES"
  attribute {
    name = "pk"
    type = "S"
  }
}

The four view types are not cosmetic — they dictate what your consumers can compute. For CDC you almost always want NEW_AND_OLD_IMAGES: the diff is what lets you route on transitions (“status went from PENDING to SHIPPED”) and reconstruct deletes — on a REMOVE event there is no new image, only the old one.

Each stream record carries an eventName of INSERT, MODIFY, or REMOVE, plus the keys, the requested image(s), the SequenceNumber, and (on TTL deletes) a userIdentity block. Know the record anatomy cold — it is what your filters and idempotency keys read:

A subtle trap: a TTL deletion arrives as REMOVE with userIdentity.principalId = dynamodb.amazonaws.com and type = Service. If you mirror deletes downstream, filter TTL expiries explicitly or you replicate housekeeping you did not intend to.

The stream’s own limits are worth memorizing because several of them shape your tuning. These are the numbers that bite:

Consuming with Lambda: batch size, window, and parallelization

The native stream is consumed through a Lambda event source mapping — a managed poller AWS runs on your behalf. You tune it; you do not write the polling loop. Three knobs dominate throughput and latency:

aws lambda create-event-source-mapping \
  --function-name orders-cdc-processor \
  --event-source-arn "$STREAM_ARN" \
  --starting-position LATEST \
  --batch-size 100 \
  --maximum-batching-window-in-seconds 5 \
  --parallelization-factor 4 \
  --maximum-retry-attempts 3 \
  --bisect-batch-on-function-error \
  --maximum-record-age-in-seconds 21600 \
  --function-response-types ReportBatchItemFailures \
  --destination-config '{"OnFailure":{"Destination":"'"$DLQ_ARN"'"}}'

Every tunable on the mapping, what it does, its default, its bounds, and the trade-off — this is the table you keep open when you create or adjust an ESM:

ParallelizationFactor deserves a warning. Ordering is preserved per partition key even with a factor above 1: the poller hashes each record’s partition key to one of the concurrent invocations, so all records for a key still go to the same invocation in order. What you lose is ordering across different keys within the same shard — almost always fine, because cross-key ordering was never guaranteed table-wide. Use it freely as long as your processing logic only assumes per-key order. How the parallelization factor interacts with order and throughput:

--starting-position LATEST begins at the tip and ignores history; TRIM_HORIZON replays everything still in the 24-hour window. For a brand-new pipeline that must backfill, TRIM_HORIZON plus a one-time scan-and-load for older data is the usual combination — the stream alone cannot reach beyond 24 hours. The starting-position choices side by side:

Rather than reason about each knob in isolation, start from the workload and read off a recipe. A decision table mapping the requirement to the ESM configuration:

The concurrency limits that bound how far you can push parallelism — real numbers, because they cap your catch-up:

Ordering and partial-batch failures: bisect-on-error and checkpointing

Here is the failure mode that bites everyone. By default, if your function throws on a batch, the ESM retries the entire batch — and because the shard’s checkpoint has not advanced, it keeps retrying the same batch until it succeeds, expires past MaximumRecordAgeInSeconds, or exhausts MaximumRetryAttempts. A single un-processable record blocks every record behind it in that shard. That is a stalled shard, and you see it as a climbing iterator age.

Two mechanisms fix this; use both.

Partial batch responses let the function tell the poller exactly which records failed so the rest are checkpointed and never reprocessed. Set --function-response-types ReportBatchItemFailures and return the failed sequence numbers:

def handler(event, context):
    failures = []
    for record in event["Records"]:
        seq = record["dynamodb"]["SequenceNumber"]
        try:
            process(record)
        except Exception:
            failures.append({"itemIdentifier": seq})
    return {"batchItemFailures": failures}

The contract has a sharp edge worth memorizing: because ordering matters, the poller treats the earliest reported failure as the checkpoint. Everything at or after that sequence number is retried; everything before it is considered done. So return failures honestly — do not report a late record as failed while silently dropping an earlier one, or you will skip data. Returning an empty list ({"batchItemFailures": []}) means the whole batch succeeded. The exact response contract and how the poller interprets each shape:

Bisect-on-error (--bisect-batch-on-function-error) handles the case where the function throws instead of reporting failures. On error the poller splits the batch in two and retries each half, recursively narrowing down to the single offending record. That record then goes to the on-failure destination once retries are exhausted, and the rest of the batch proceeds. Without bisect, one bad record can burn your entire retry budget on a 100-record batch. The two mechanisms compared — ship both, but know which does what:

Rule of thumb: ReportBatchItemFailures is the primary mechanism — precise, avoids reprocessing good records. BisectBatchOnFunctionError is the safety net for unexpected exceptions and serialization bugs you did not anticipate. Ship both.

EventBridge Pipes: filtering, enrichment, and routing without glue Lambdas

A large fraction of CDC “processors” are glue: filter the events you care about, reshape them, send them to a target. EventBridge Pipes does exactly that as a managed integration — source, optional filter, optional enrichment, target — with no poller code to own. The source can be a DynamoDB Stream directly. The four stages and what each is for:

The filter stage runs before you pay for enrichment or invocation, so push as much selectivity into it as you can. Pipes filter patterns match the stream record shape, including the DynamoDB-typed attribute values:

{
  "eventName": ["MODIFY"],
  "dynamodb": {
    "NewImage": {
      "status": { "S": ["SHIPPED"] }
    }
  }
}

The Terraform wiring is compact, and the dynamodb_stream_parameters block carries the same batch/window/retry semantics as a Lambda ESM:

resource "aws_pipes_pipe" "orders_shipped" {
  name     = "orders-shipped-to-bus"
  role_arn = aws_iam_role.pipe.arn
  source   = aws_dynamodb_table.orders.stream_arn
  target   = aws_cloudwatch_event_bus.commerce.arn

  source_parameters {
    dynamodb_stream_parameters {
      starting_position                  = "LATEST"
      batch_size                         = 100
      maximum_batching_window_in_seconds = 5
      maximum_retry_attempts             = 4
      on_partial_batch_item_failure      = "AUTOMATIC_BISECT"
      dead_letter_config {
        arn = aws_sqs_queue.pipe_dlq.arn
      }
    }
    filter_criteria {
      filter {
        pattern = jsonencode({
          eventName = ["MODIFY"]
          dynamodb  = { NewImage = { status = { S = ["SHIPPED"] } } }
        })
      }
    }
  }
}

The enrichment stage can hydrate the event — look up a customer record, call a pricing service — and return a transformed payload, all without you running a poller. The target transformer then reshapes the final payload with an input template. The net effect: a filter-enrich-route pipeline that used to be 150 lines of Lambda becomes declarative infrastructure, and the only code you keep is genuine business logic in the enrichment. The decision between Pipes and a Lambda ESM, made explicit:

Choose Pipes when the work is route and reshape. Keep a Lambda ESM when the work is compute — non-trivial transformation, transactional writes to multiple sinks, or anything needing libraries and real error handling.

The Pipes target options, end to end, with what each is good for as a CDC destination:

Idempotency and exactly-once semantics in downstream sinks

DynamoDB Streams are at-least-once. Redrives, bisect retries, and poller restarts all mean a downstream sink can see the same change more than once. “Exactly-once” downstream is therefore not a delivery setting you toggle — it is an idempotent write you design. The good news is every stream record hands you the tools: a stable item key and a monotonic SequenceNumber.

The cleanest pattern is a conditional write keyed on sequence. Store the last applied sequence number per item and reject anything not strictly newer. This collapses duplicates and protects against out-of-order application during reprocessing:

def apply_idempotent(key, new_image, sequence):
    try:
        table.put_item(
            Item={**new_image, "_seq": sequence},
            ConditionExpression="attribute_not_exists(#s) OR #s < :seq",
            ExpressionAttributeNames={"#s": "_seq"},
            ExpressionAttributeValues={":seq": sequence},
        )
    except ClientError as e:
        if e.response["Error"]["Code"] != "ConditionalCheckFailedException":
            raise
        # Duplicate or stale replay -> safe no-op

DynamoDB SequenceNumber values are strictly increasing within a shard for a given key, which is exactly the scope this guard needs. For sinks that lack conditional writes, fall back to a deterministic idempotency key — partitionKey#sortKey#sequenceNumber — and let the sink’s natural upsert (OpenSearch _id, an S3 object key, a primary-key MERGE) absorb the duplicate. The principle is invariant: make replay a no-op, never an additive side effect. The idempotency strategy per sink type:

The failure modes idempotency defends against, so you can reason about why each guard exists:

Fan-out to OpenSearch, S3, and analytics targets

Real pipelines feed several sinks with different durability and freshness needs. Resist the urge to fan out from a single mega-Lambda — couple them and one slow target stalls the shard for all of them. Fan out at the bus instead: a Pipe lands changes on an EventBridge bus, and independent rules deliver to each consumer with its own retry budget and DLQ.

• OpenSearch — index the NewImage with _id set to the item key so writes are idempotent upserts; on REMOVE, issue a delete by that same _id. Honor per-key order by deriving a monotonic version from the sequence number and using external versioning so a stale replay cannot overwrite a newer doc.
• S3 (data lake) — buffer through Kinesis Data Firehose and write newline-delimited JSON partitioned by event date. Firehose batches and compresses, which keeps you off the small-object tax that kills Athena performance. This is your replayable system of record once the 24-hour stream window has rolled off.
• Analytics / warehouse — land in S3, then COPY/MERGE into Redshift or query in place with Athena. Keep the raw change log immutable and do transformation downstream; never let a warehouse load be the only place a change exists.

Each sink has a different freshness/durability/cost profile — match the sink to the requirement:

A single Lambda target can also do the OpenSearch fan-out directly when you want code-level control — emit per-document failures via ReportBatchItemFailures so a 409 on one doc does not block the batch:

from opensearchpy import helpers

def to_action(record):
    key = record["dynamodb"]["Keys"]["pk"]["S"]
    if record["eventName"] == "REMOVE":
        return {"_op_type": "delete", "_index": "orders", "_id": key}
    img = deserialize(record["dynamodb"]["NewImage"])
    version = int(record["dynamodb"]["SequenceNumber"])
    return {
        "_op_type": "index", "_index": "orders", "_id": key,
        "_version": version, "_version_type": "external", "_source": img,
    }

Mapping each eventName to the right sink action — the table that keeps your projector correct:

Handling poison records with on-failure destinations and DLQs

A poison record is one that will never succeed no matter how many times you retry — a schema your code cannot parse, a downstream constraint it will always violate. The entire point of MaximumRetryAttempts, MaximumRecordAgeInSeconds, and an on-failure destination is to evict that record so the shard keeps moving.

For a Lambda ESM, the on-failure destination (SQS or SNS) receives metadata about the failed batch, not the record bodies themselves — shard ID, the failing sequence-number range, and an error summary. That is deliberate: the records still live in the stream for 24 hours, so the destination’s job is to point you at them. Your redrive tooling reads the DynamoDBStreamArn, shard, and sequence-number window from the failure message and re-reads the offending records with GetShardIterator at AT_SEQUENCE_NUMBER. An EventBridge Pipes DLQ behaves the same way — it captures failure context after retries are exhausted.

{
  "requestContext": { "functionArn": "arn:aws:lambda:...:orders-cdc-processor" },
  "responseContext": { "statusCode": 200, "functionError": "Unhandled" },
  "DDBStreamBatchInfo": {
    "shardId": "shardId-00000001700000000000-abcdef01",
    "startSequenceNumber": "700000000000000000001",
    "endSequenceNumber": "700000000000000000099",
    "approximateArrivalOfFirstRecord": "2026-06-08T14:00:00Z",
    "batchSize": 99,
    "streamArn": "arn:aws:dynamodb:...:table/orders/stream/2026-06-08T00:00:00.000"
  }
}

What the failure message contains, and what you do with each field during a redrive:

Alarm on ApproximateAgeOfOldestRecord and on DLQ depth, and make the redrive a deliberate, reviewed action — not an automatic retry that re-poisons the pipeline. Set MaximumRecordAgeInSeconds comfortably below 24 hours (e.g. 6 hours) so records are evicted to the DLQ while they are still readable in the stream; if a record ages past the 24-hour retention before you redrive, it is gone for good. The on-failure surface differs slightly between ESM and Pipes:

The metrics you alarm on are the difference between catching a stall in minutes and discovering it from a customer complaint. The CloudWatch reference for a DynamoDB-Streams CDC pipeline — namespace, what it measures, and a starting alarm threshold:

The error and exception strings you will actually see, what each means on this path, and the first move:

Architecture at a glance

The diagram traces a change as it actually flows, then maps each failure class onto the exact hop where it bites. Read it left to right. A write commits to the orders table and lands on its Stream (NEW_AND_OLD_IMAGES, sharded, 24-hour retention) — badge 1 marks where records age out of the window if a stall lets them. The capture/poll zone holds the two managed pollers: the Lambda ESM (batch 100, parallelization 1–10) and EventBridge Pipes (filter, no code) — badge 2 sits on the ESM, where a hot shard stalls the single poller per shard. In process, the CDC processor writes idempotently (conditional on _seq) — badge 3 is the duplicate / out-of-order apply class — and the DLQ captures the shardId + sequence range when a poison record wedges the shard (badge 4). The fan-out sinks — OpenSearch (upsert by _id/_version), Firehose→S3 (NDJSON, gzip), encrypted with a KMS CMK — carry badge 5, the sink-drift / fan-out-coupling class. Finally CloudWatch watches IteratorAge, the leading indicator for every stall.

The whole method is in that left-to-right path: a change is captured once, processed exactly-once via an idempotent write, fanned out at the bus so each sink owns its retries, and observed so a climbing iterator age pages you before the 24-hour cliff. The five numbered badges are the five ways it goes wrong, and the legend narrates each as symptom · confirm · fix — so when the pager fires you localize the failure to a hop, read the cause, run the named confirm, and apply the fix.

Real-world scenario

A retail platform team — call them MeridianStock — ran inventory on a single DynamoDB table fronting 2,000 stores, with a Lambda ESM projecting every change into an OpenSearch index that powered the storefront’s “in stock near you” search. Average load was 600 writes/second; the team was five engineers; the monthly bill for the table + stream + Lambda + OpenSearch ran about ₹95,000. On a normal day it was fine.

During a flash sale, one SKU’s partition went hot. The ESM’s iterator age on that shard climbed past 40 minutes, and search results went stale right when traffic peaked — the exact failure that costs money. Worse, a malformed bundle item (a nested map their deserializer choked on) had been silently stuck in that shard for hours, blocking every inventory update behind it because the whole batch kept failing and retrying. Their original ESM had BatchSize=500, ParallelizationFactor=1, no partial-batch reporting, and no bisect — so a single bad record consumed the entire retry budget on a 500-record batch, over and over.

The constraint was twofold: they needed the hot shard to catch up without resharding a 2,000-store table mid-sale, and they could not let one un-parseable item hold the rest hostage. The breakthrough was reading the right metric. GetRecords.IteratorAgeMilliseconds was pinned near 2,400,000 (40 minutes) on exactly one shard while the others sat near zero — a textbook hot-shard-plus-poison signature, not a capacity problem. Scaling the table would have done nothing.

They made three changes, all configuration. They raised ParallelizationFactor to 8 so the hot shard drained across eight concurrent invocations (ordering held, because all records for a given SKU still hashed to one invocation). They switched the function to ReportBatchItemFailures and returned precise failed sequence numbers, so good records checkpointed immediately instead of being reprocessed behind the poison item. And they enabled BisectBatchOnFunctionError with MaximumRecordAgeInSeconds=21600, so the malformed bundle item was narrowed down and evicted to a DLQ within minutes instead of wedging the shard for hours.

aws lambda update-event-source-mapping \
  --uuid "$ESM_UUID" \
  --parallelization-factor 8 \
  --batch-size 100 \
  --maximum-record-age-in-seconds 21600 \
  --bisect-batch-on-function-error \
  --function-response-types ReportBatchItemFailures \
  --destination-config '{"OnFailure":{"Destination":"'"$DLQ_ARN"'"}}'

The outcome: iterator age on the hot shard fell back under 30 seconds within the next sale, the poison bundle item surfaced in the DLQ with its exact sequence range for an engineer to fix offline, and the storefront search stopped lying about stock during the one window where accuracy mattered most. The following sprint they also moved the data-lake sink off the projector Lambda and onto an EventBridge Pipe → bus, so a slow OpenSearch never again stalled the lake. No table redesign, no resharding — just the ESM knobs used the way they were designed. The incident as a timeline, because the order of moves is the lesson:

Advantages and disadvantages

DynamoDB Streams give you a genuinely powerful change feed, but the model’s strengths and its sharp edges are two sides of the same coin. Weigh them honestly:

The model is right whenever you need item-level CDC with strict per-key ordering and a short replay window — search projection, cache invalidation, audit trails, materialized views. It bites hardest on hot-key workloads, multi-sink fan-out done in one function, and teams that ship the defaults. Every disadvantage is manageable — but only if you know it exists, which is the entire point of building idempotency, bisect, DLQ, and bus-level fan-out in from the start.

Hands-on lab

Build a minimal CDC pipeline end to end — a table with a stream, a Lambda ESM with the safety knobs on, an idempotent projection, and a deliberate poison record — then watch it recover. Free-tier-friendly; teardown at the end. Run in CloudShell.

Step 1 — Variables and a table with a stream.

export AWS_PAGER=""
TABLE=cdc-orders-lab
DLQ=cdc-orders-dlq
FN=cdc-orders-processor

aws dynamodb create-table \
  --table-name "$TABLE" \
  --attribute-definitions AttributeName=pk,AttributeType=S \
  --key-schema AttributeName=pk,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST \
  --stream-specification StreamEnabled=true,StreamViewType=NEW_AND_OLD_IMAGES
aws dynamodb wait table-exists --table-name "$TABLE"
STREAM_ARN=$(aws dynamodb describe-table --table-name "$TABLE" \
  --query 'Table.LatestStreamArn' --output text)
echo "$STREAM_ARN"

Expected: a stream ARN ending in a timestamp. Note it — the ESM points at it.

Step 2 — Create the DLQ and capture its ARN.

DLQ_URL=$(aws sqs create-queue --queue-name "$DLQ" --query QueueUrl --output text)
DLQ_ARN=$(aws sqs get-queue-attributes --queue-url "$DLQ_URL" \
  --attribute-names QueueArn --query 'Attributes.QueueArn' --output text)
echo "$DLQ_ARN"

Step 3 — A processor that is idempotent and reports per-record failures. Save as app.py and zip it. It throws on a deliberately malformed item so you can watch bisect isolate it:

import json
def handler(event, context):
    failures = []
    for r in event["Records"]:
        seq = r["dynamodb"]["SequenceNumber"]
        try:
            img = r["dynamodb"].get("NewImage", {})
            if img.get("poison", {}).get("BOOL"):     # the malformed item
                raise ValueError("poison record")
            print("applied", r["dynamodb"]["Keys"]["pk"]["S"], seq)
        except Exception as e:
            print("FAILED", seq, str(e))
            failures.append({"itemIdentifier": seq})
    return {"batchItemFailures": failures}

Step 4 — Deploy the function (assumes an existing Lambda execution role with stream + SQS + logs permissions; see Security notes for the exact policy).

zip fn.zip app.py
aws lambda create-function --function-name "$FN" \
  --runtime python3.12 --handler app.handler \
  --role "$LAMBDA_ROLE_ARN" --zip-file fileb://fn.zip

Step 5 — Create the event source mapping with every safety knob on.

aws lambda create-event-source-mapping \
  --function-name "$FN" --event-source-arn "$STREAM_ARN" \
  --starting-position TRIM_HORIZON \
  --batch-size 10 --maximum-batching-window-in-seconds 5 \
  --parallelization-factor 2 \
  --maximum-retry-attempts 2 --maximum-record-age-in-seconds 3600 \
  --bisect-batch-on-function-error \
  --function-response-types ReportBatchItemFailures \
  --destination-config '{"OnFailure":{"Destination":"'"$DLQ_ARN"'"}}'

Expected: a mapping with State progressing to Enabled. Confirm: aws lambda list-event-source-mappings --function-name "$FN" --query 'EventSourceMappings[0].State'.

Step 6 — Write a few good items and one poison item, then watch.

aws dynamodb put-item --table-name "$TABLE" --item '{"pk":{"S":"ORDER#1"},"status":{"S":"PENDING"}}'
aws dynamodb put-item --table-name "$TABLE" --item '{"pk":{"S":"ORDER#2"},"poison":{"BOOL":true}}'
aws dynamodb put-item --table-name "$TABLE" --item '{"pk":{"S":"ORDER#3"},"status":{"S":"SHIPPED"}}'
sleep 30
aws logs tail "/aws/lambda/$FN" --since 5m

Expected in the logs: applied ORDER#1, a FAILED line for ORDER#2, and applied ORDER#3 — proof the good records checkpointed while the poison one was isolated and routed to the DLQ after its retries.

Step 7 — Confirm the poison landed in the DLQ.

aws sqs receive-message --queue-url "$DLQ_URL" --wait-time-seconds 5 \
  --query 'Messages[0].Body' --output text

Expected: a JSON body with DDBStreamBatchInfo carrying the shardId and the failing sequence range. The lab steps mapped to what each proves:

Cleanup.

ESM=$(aws lambda list-event-source-mappings --function-name "$FN" --query 'EventSourceMappings[0].UUID' --output text)
aws lambda delete-event-source-mapping --uuid "$ESM"
aws lambda delete-function --function-name "$FN"
aws sqs delete-queue --queue-url "$DLQ_URL"
aws dynamodb delete-table --table-name "$TABLE"

Cost note. On-demand DynamoDB, a tiny Lambda, and an SQS queue for an hour cost a few rupees; deleting the four resources stops everything. There is no charge for enabling a stream — you pay per GetRecords call the ESM makes, which is negligible at this scale.

Common mistakes & troubleshooting

This is the playbook — the part you bookmark. First as a scannable table you can read while the iterator age climbs, then the detail underneath for the entries that bite hardest.

The expanded form for the entries that cause the most lost hours:

1. One shard’s iterator age climbs to minutes; downstream goes stale. Root cause: A hot partition key funnels writes into one shard, and the default one-poller-per-shard model falls behind. Confirm: aws cloudwatch get-metric-statistics --namespace AWS/DynamoDB --metric-name ... — or simpler, the Lambda IteratorAge metric high on the one shard. The signature is one shard high while the rest are near zero. Fix: Raise ParallelizationFactor (up to 10) so the shard drains across concurrent invocations; per-key order still holds because each key hashes to one invocation. Enable ReportBatchItemFailures so good records checkpoint immediately. The durable fix is key design — see single-table modeling — but tuning buys you the incident.

2. The whole pipeline is wedged; the same batch retries forever. Root cause: A poison record with default settings (infinite retries, no bisect) — the batch fails, the checkpoint never advances, it retries indefinitely. Confirm: Function invocation errors loop on the same batch; iterator age climbs across shards. Fix: BisectBatchOnFunctionError to isolate the record by halving, a finite MaximumRetryAttempts, and an on-failure destination so the isolated record lands in a DLQ. Combine with ReportBatchItemFailures so you rarely reach bisect at all.

3. The search index double-applies a change or shows a stale document. Root cause: A non-idempotent write, or an out-of-order replay overwriting a newer document during reprocessing. Confirm: OpenSearch _version regresses; row counts in the warehouse drift from the table. Fix: Conditional write keyed on SequenceNumber (attribute_not_exists(_seq) OR _seq < :seq); for OpenSearch use _version_type=external with the sequence-derived version so a lower version is rejected.

4. Records vanished and were never processed. Root cause: They aged past the 24-hour retention before anyone redrove them. Confirm: ApproximateAgeOfOldestRecord was approaching 86,400 seconds. Fix: Set MaximumRecordAgeInSeconds well below 24h (e.g. 21,600) so poison records hit the DLQ while still readable; alarm on iterator age at minutes, not hours.

6. One slow sink stalls all sinks. Root cause: A single mega-Lambda fans out to every sink, so one slow target (OpenSearch under merge pressure) stalls the shard for all of them. Confirm: Iterator age climbs precisely when one sink is slow; the others are healthy but starved. Fix: Land changes on an EventBridge bus (via a Pipe) and give each sink its own rule, retry budget, and DLQ. The projector’s slowness no longer blocks the lake.

Iterator age is the master signal, but its shape tells you which failure you have. Read the pattern, not just the number:

Best practices

• Pick the delivery surface before tuning. Native Stream for strict per-key order and a short window; Kinesis Data Streams for DynamoDB for long retention or many fan-out consumers. You cannot tune your way out of the wrong surface.
• Enable NEW_AND_OLD_IMAGES unless you have a concrete reason for a leaner view. The diff is what lets you route on transitions and reconstruct deletes.
• Filter TTL-driven REMOVE events explicitly (or in the Pipe filter) so housekeeping is not replicated downstream.
• Ship ReportBatchItemFailures and BisectBatchOnFunctionError together on every ESM — the former for precise checkpointing, the latter as the safety net for unhandled exceptions.
• Always report the earliest real failure in a partial-batch response. Reporting a late one while dropping an earlier one silently skips data.
• Set MaximumRecordAgeInSeconds below 24 hours (e.g. 6h) so poison records are evicted to a DLQ while still readable in the stream.
• Make every downstream write idempotent — conditional-on-SequenceNumber, or a deterministic pk#sk#seq key with the sink’s natural upsert. Treat replay as a non-event.
• Derive a monotonic version from SequenceNumber and use external versioning (OpenSearch _version) so a stale replay cannot overwrite a newer doc.
• Fan out at an EventBridge bus/Pipe, never one mega-Lambda, so each sink owns its retries and DLQ and one slow target can’t stall the rest.
• Use ParallelizationFactor to drain hot shards during a spike, but fix chronic hot keys in the data model.
• Keep S3 (via Firehose) as the immutable system of record — the only thing that survives the 24-hour window and lets you rebuild any sink.
• Alarm on IteratorAge / ApproximateAgeOfOldestRecord, function errors/throttles, and DLQ depth before go-live — a silent stall you didn’t alarm on is the worst case.
• Make redrives deliberate and reviewed. Read shardId + sequence range from the DLQ and re-read at AT_SEQUENCE_NUMBER; never auto-retry a poison record back into the pipeline.

Security notes

DynamoDB Streams inherit the table’s encryption and are governed by their own IAM action set; the consumer needs least-privilege access to read the stream and write the DLQ, nothing more.

• Encrypt at rest with a customer-managed KMS key. A stream encrypts with the same key as its table; using a KMS CMK (rather than the AWS-owned key) gives you key policy control, rotation, and an auditable grant to the consumer’s role. See AWS KMS in Depth: Multi-Region Keys, Envelope Encryption, Key Policies, and Grants.
• Scope the consumer’s IAM to the four stream actions. The Lambda/Pipe role needs only DescribeStream, GetRecords, GetShardIterator, and ListStreams on the specific stream ARN — never dynamodb:*.
• Separate read-from-stream from write-to-sink. Grant sqs:SendMessage to the DLQ, the specific OpenSearch/S3 write actions to the sinks, and nothing broader. Cross-account fan-out should use scoped assume-role — see Secure Cross-Account Access: Assume-Role Patterns, External ID, Confused Deputy, and Session Policies.
• Don’t leak PII into the lake unfiltered. The NewImage/OldImage carry the full item; mask or drop sensitive attributes in the enrichment/processor before they land in S3 or a search index.
• Lock the DLQ down. Failure metadata names shard IDs and sequence ranges; restrict sqs:ReceiveMessage to the redrive role and encrypt the queue with SSE-KMS.

The minimal consumer policy — what to grant and why each action is needed:

data "aws_iam_policy_document" "cdc_consumer" {
  statement {
    actions   = ["dynamodb:DescribeStream", "dynamodb:GetRecords",
                 "dynamodb:GetShardIterator", "dynamodb:ListStreams"]
    resources = [aws_dynamodb_table.orders.stream_arn]
  }
  statement {
    actions   = ["sqs:SendMessage"]
    resources = [aws_sqs_queue.cdc_dlq.arn]
  }
  statement {
    actions   = ["kms:Decrypt"]
    resources = [aws_kms_key.table.arn]
  }
}

Cost & sizing

The bill for a CDC pipeline is dominated by the consumers and sinks, not the stream itself — enabling a stream is free, and you pay only for the GetRecords calls the managed poller makes. What actually drives the cost and how to right-size it:

• GetRecords requests are billed per call (DynamoDB Streams read-request units); the ESM batches efficiently, so this is small unless you set a tiny batch size with a zero batching window (many under-full reads). Use a sensible BatchSize (100) and a short MaximumBatchingWindowInSeconds (1–5s) to keep calls few and full.
• Lambda is the usual largest line — invocations × duration × memory. Bigger batches amortize invocation overhead; right-size memory to the per-record work.
• OpenSearch is often the single most expensive sink (cluster hours regardless of change volume) — size it to query load, and remember it is rebuildable from S3, so you can run it leaner.
• Firehose + S3 are cheap per GB; the win is avoiding the small-object tax by letting Firehose batch and compress, which keeps Athena fast and S3 PUTs low.
• Kinesis Data Streams for DynamoDB adds shard-hour (provisioned) or per-GB (on-demand) cost plus the longer retention — only pay this when you genuinely need >24h replay or many consumers.

A rough monthly picture for a mid-size pipeline (600 writes/s projected to OpenSearch + a lake):

The lesson MeridianStock learned: most “we need a bigger pipeline” instincts are actually “we mis-tuned the poller” or “we coupled our sinks.” Fix the configuration and the fan-out topology first; the bill usually goes down, not up. There is no free tier for streams specifically, but the per-GetRecords cost is small enough that the consumers dominate every realistic bill.

Interview & exam questions

1. What ordering guarantee does a DynamoDB Stream actually provide? Strict ordering per partition key, within a shard lineage — records for a given key are delivered in commit order. There is no global/table-wide ordering. Consumers must only assume per-key order; this is what makes ParallelizationFactor > 1 safe.

2. Is delivery exactly-once? How do you achieve exactly-once downstream? No — delivery is at-least-once; redrives and retries can replay a record. Exactly-once downstream is an idempotent write you design: a conditional write keyed on SequenceNumber in DynamoDB, external versioning in OpenSearch, or a deterministic pk#sk#seq key with the sink’s natural upsert.

3. A single poison record has stalled a shard for hours. Why, and what two settings fix it? With defaults (infinite retries, no bisect) the failing batch never checkpoints, so it retries forever and blocks everything behind it. Fix with BisectBatchOnFunctionError (isolate the record by halving) plus an on-failure destination, and ReportBatchItemFailures so good records checkpoint immediately. Bound it with a finite MaximumRetryAttempts and a MaximumRecordAgeInSeconds below 24h.

4. What does ParallelizationFactor do, and what does it cost you? It runs up to 10 concurrent invocations per shard, draining a hot shard without resharding. Per-key order is preserved (each key hashes to one invocation); you relax ordering across keys within a shard — almost always acceptable since cross-key order was never guaranteed.

5. How does the ReportBatchItemFailures checkpoint contract work, and what’s the trap? You return the failed sequence numbers; the poller treats the earliest reported failure as the checkpoint — everything at or after it is retried, everything before is done. The trap: reporting a late failure while dropping an earlier record silently skips data. Always report the earliest real failure.

6. When do you choose EventBridge Pipes over a Lambda ESM? When the work is route and reshape — filter, optionally enrich, deliver — with no poller code to own. Keep a Lambda ESM when the work is genuine compute: non-trivial transformation needing libraries, transactional writes to multiple sinks, or windowed aggregation (tumbling windows are ESM-only).

7. Why is a 24-hour stream not enough, and what do you pair with it? Records age out after 24 hours and are gone; you cannot backfill history from the stream. Pair it with S3 (via Firehose) as the immutable, replayable system of record, and a one-time Scan-and-load for data older than the window. Set MaximumRecordAgeInSeconds below 24h so poison records hit the DLQ while still readable.

8. How do you keep an OpenSearch projection from being corrupted by an out-of-order replay? Derive a monotonic version from the record’s SequenceNumber and index with _version_type=external, so OpenSearch rejects any write with a lower version. Set _id to the item key so writes are idempotent upserts and a REMOVE is a delete by the same _id.

9. What’s the difference between native DynamoDB Streams and Kinesis Data Streams for DynamoDB? Native Streams: 24-hour retention, strict per-key order, at-least-once, ~2 efficient readers per shard. Kinesis for DynamoDB: retention up to a year, best-effort order (reorder on sequence), possible duplicates, and many fan-out consumers. Pick Kinesis for long replay or many consumers; native for strict order and a short window.

10. The on-failure destination has a message but no record body. Why, and how do you recover the records? The destination receives batch metadata (shardId, failing sequence range, stream ARN), not the records — they still live in the stream for 24h. Recover by GetShardIterator at AT_SEQUENCE_NUMBER with the range from the message and re-reading the offending records for a deliberate redrive.

11. How do you handle TTL deletes you don’t want to replicate? A TTL deletion arrives as REMOVE with userIdentity.principalId=dynamodb.amazonaws.com. Filter those explicitly (in the function or the Pipe filter pattern) so housekeeping expiries are not mirrored to downstream sinks.

12. Iterator age is climbing on exactly one shard while others are flat. What does that tell you? A hot partition key is funneling writes into that shard and the poller is falling behind (often compounded by a poison record). Confirm with per-shard IteratorAge; mitigate with ParallelizationFactor and ReportBatchItemFailures, and fix the root cause in the data model (key design).

These map to the AWS Certified Developer – Associate (DVA-C02) — develop event-driven solutions, Lambda event source mappings, DynamoDB Streams — and the Solutions Architect – Associate / Professional for the integration and fan-out patterns. A compact cert mapping:

Quick check

1. A redrive replays a batch and your OpenSearch index now shows an older document than before. What single mechanism prevents this, and what value do you derive it from?
2. One shard’s IteratorAge is at 30 minutes while every other shard is near zero. What is the most likely root cause, and which ESM knob buys you time?
3. True or false: setting MaximumRetryAttempts infinite (the default) with no bisect is a safe configuration for a CDC processor.
4. You need to replay changes from eight days ago and have three independent teams consuming the same feed. Native Stream or Kinesis Data Streams for DynamoDB — and why?
5. Your on-failure SQS message contains a DDBStreamBatchInfo block but none of the actual records. How do you get the records to fix them?

Answers

1. External versioning derived from the record’s SequenceNumber: index into OpenSearch with _version set to the sequence (cast to a number) and _version_type=external, so a write with a lower version is rejected and a stale replay can’t overwrite a newer doc. In DynamoDB the equivalent is a conditional write attribute_not_exists(_seq) OR _seq < :seq.
2. A hot partition key is funneling writes into that one shard, so the single poller per shard falls behind (often with a poison record compounding it). Raise ParallelizationFactor (up to 10) to drain the shard across concurrent invocations — per-key order still holds — and enable ReportBatchItemFailures. The durable fix is the data model.
3. False. With infinite retries and no bisect, a single poison record fails the batch, the checkpoint never advances, and it retries forever — stalling every record behind it. Use BisectBatchOnFunctionError, a finite MaximumRetryAttempts, MaximumRecordAgeInSeconds below 24h, and an on-failure DLQ.
4. Kinesis Data Streams for DynamoDB. The native Stream caps retention at 24 hours (so eight days is impossible) and supports only ~2 efficient readers per shard. Kinesis offers retention up to a year and clean multi-consumer fan-out — at the cost of best-effort ordering and possible duplicates, so dedupe and reorder on a sequence number.
5. The destination carries batch metadata, not records — the records still live in the stream for 24 hours. Read the shardId and startSequenceNumber/endSequenceNumber from the message, call GetShardIterator at AT_SEQUENCE_NUMBER, and re-read the offending records for a deliberate, reviewed redrive (provided they haven’t aged past 24h).

Glossary

• DynamoDB Stream — an ordered, append-only log of item-level changes on a table, retained for 24 hours and sharded to match the table’s partitions.
• Shard — an ordered partition of the stream; the unit of both ordering and consumer parallelism.
• StreamViewType — what each record carries: KEYS_ONLY, NEW_IMAGE, OLD_IMAGE, or NEW_AND_OLD_IMAGES.
• SequenceNumber — a monotonic, per-shard record identifier; the anchor for idempotency and external versioning.
• Event source mapping (ESM) — the managed Lambda poller AWS runs for you; you tune BatchSize, ParallelizationFactor, retries, and failure handling.
• ParallelizationFactor — 1–10 concurrent invocations per shard; drains a hot shard while preserving per-key order.
• ReportBatchItemFailures — the partial-batch response contract; you return failed sequence numbers so good records checkpoint and aren’t reprocessed.
• Bisect-on-error — BisectBatchOnFunctionError; on a thrown error the poller splits the batch and retries halves to isolate the offending record.
• Iterator age — GetRecords.IteratorAgeMilliseconds / ApproximateAgeOfOldestRecord; how far behind the consumer is — the leading indicator of a stall.
• On-failure destination / DLQ — an SQS/SNS target that receives failure metadata (shard + sequence range) after retries are exhausted; the records stay in the stream for 24h.
• EventBridge Pipes — a managed source→filter→enrich→target integration; filter-enrich-route with no poller code.
• Poison record — an item that will never process successfully (unparseable schema, constraint violation); must be evicted so the shard keeps moving.
• Idempotent write — a downstream write designed so replay is a no-op (conditional-on-seq, upsert by id, deterministic pk#sk#seq key).
• Kinesis Data Streams for DynamoDB — an alternative change feed with retention up to a year and many fan-out consumers, at the cost of best-effort ordering and possible duplicates.
• TTL delete — an item expiry that arrives as REMOVE with userIdentity.principalId=dynamodb.amazonaws.com; filter it to avoid replicating housekeeping.
• System of record (S3) — the immutable, replayable change log (via Firehose) that survives the 24-hour stream window and lets you rebuild any sink.

Next steps

You can now build a CDC pipeline off DynamoDB Streams that survives replay, isolates poison, preserves per-key order, and fans out cleanly. Build outward:

• Next: Designing Event-Driven Architectures with Amazon EventBridge: Buses, Rules, Schemas, and Archive/Replay — the bus you fan out to, with archive/replay that complements the 24-hour stream window.
• Related: Resilient Messaging with SQS and SNS: Fan-Out, FIFO Ordering, DLQs, and Poison-Message Handling — the DLQ and poison-handling patterns this article leans on, in depth.
• Related: DynamoDB Single-Table Design: Modeling Access Patterns, GSIs, and Hot Partition Avoidance — fix hot keys at the source so no consumer tuning is needed.
• Related: A Structured Logging Pipeline on AWS: JSON Logs, CloudWatch Metric Filters, and Firehose to OpenSearch — the Firehose→S3→OpenSearch sink half, built out.
• Related: Event-Driven Order Processing with the Saga Pattern on AWS — where CDC events become saga steps in a real workflow.