Event-Driven Architecture on AWS: EventBridge, SQS, SNS, Lambda and Step Functions

It is 19:58 on the first night of a festival sale and your checkout API is a synchronous chain: order service calls inventory, which calls payments, which calls the loyalty service, which calls the email provider. The email provider’s p99 climbs to 9 seconds, every thread pool upstream fills with waiters, and by 20:04 customers cannot check out at all — not because anything your team owns is down, but because everything is welded together at request time. Event-driven architecture (EDA) is the structural answer: services record facts (“OrderPlaced”) onto a durable intermediary, and downstream consumers react on their own schedule, with their own retries, their own failures, and their own scaling. The email provider being slow becomes an email-queue backlog, not a checkout outage.

On AWS the intermediary is not one service but a toolbox: Amazon EventBridge (a serverless event bus with content-based routing), Amazon SNS (pub/sub fan-out), Amazon SQS (durable queues that buffer and meter work), AWS Lambda (the default consumer compute, wired in through event source mappings), and AWS Step Functions (orchestration when a reaction is a multi-step transaction that must complete or compensate). Each solves a different slice of the problem, each has different delivery guarantees, ordering behaviour, retry machinery and failure modes — and gluing them together naively produces the classic asynchronous horrors: lost events, duplicate side effects, poison messages blocking a FIFO group, and a DLQ nobody alarmed on quietly ageing out after 14 days.

This article is the full engineering treatment. You will learn the mental model (events vs commands, choreography vs orchestration), each service option-by-option with its real limits and defaults, the decision matrix between SNS, SQS and EventBridge, how Lambda’s pollers actually batch and retry, how to build sagas with compensation in Step Functions, why exactly-once is a marketing word and what idempotency plus the transactional outbox actually buy you, how to put a dead-letter queue (DLQ) on every hop, and how to trace one business event across five services with AWS X-Ray. Everything comes with real aws CLI, EventBridge pattern JSON and Terraform, and the reference tables are designed to be kept open during design reviews and incidents alike.

What problem this solves

Synchronous microservice chains fail collectively: availability multiplies (five 99.9% dependencies in series ≈ 99.5%), latency adds, and load spikes propagate instantly to the weakest component. Teams also couple at build time — checkout must know the loyalty service’s endpoint, payload, auth and SLA, so adding a tenth consumer of “an order happened” means changing the order service for the tenth time. Event-driven design inverts the dependency: the producer publishes a fact once; consumers subscribe without the producer knowing they exist.

Without this architecture — or with a half-understood version of it — production teams hit a predictable set of failures:

The trade is new failure classes — duplicates, reordering, eventual consistency, invisible drops — which is exactly what the rest of this article weaponises you against. It bites hardest on teams decomposing a monolith into microservices, platform teams building a central integration bus, and anyone wiring SaaS webhooks (payments, CRM) into internal systems at scale.

Learning objectives

By the end of this article you can:

• Distinguish events, commands and queries, and choose choreography (EventBridge) vs orchestration (Step Functions) per workflow with defensible reasoning.
• Design EventBridge buses, write content-filtering event patterns (prefix, numeric, anything-but, $or, wildcard), configure per-target retry policy and DLQ, and use archive/replay, pipes, scheduler and the schema registry appropriately.
• Apply the SNS vs SQS vs EventBridge decision matrix and defend it in a design review.
• Configure SQS correctly: standard vs FIFO, visibility timeout sized to the consumer, maxReceiveCount and redrive, long polling, and high-throughput FIFO with message group design.
• Tune Lambda event source mappings: batch size, batching window, maximum concurrency, filters, and partial batch responses (ReportBatchItemFailures) so one bad record doesn’t recycle a whole batch.
• Build a Step Functions saga with Retry/Catch and compensation states, and choose Standard vs Express by duration, semantics and cost.
• Make every consumer idempotent (DynamoDB conditional writes, Lambda Powertools) and publish reliably with the transactional outbox pattern.
• Put a DLQ on every hop, alarm on the right CloudWatch metrics (ApproximateAgeOfOldestMessage, DeadLetterInvocations), and trace one event end-to-end with X-Ray.

Prerequisites & where this fits

You should be comfortable with core Lambda concepts (handlers, concurrency, timeouts), IAM policies and roles, basic DynamoDB, and reading Terraform. If Lambda triggers are new, start with AWS Lambda Event-Driven Patterns — this article assumes those basics and goes several levels deeper. For the compute landscape around it, see AWS Compute: EC2, Lambda, ECS and EKS; for the data stores these pipelines land in, RDS vs DynamoDB vs Aurora.

This is an architecture-track article: it is the AWS counterpart to GCP Pub/Sub Event-Driven Architecture (useful contrast: Pub/Sub merges SNS-style fan-out and SQS-style pull into one service) and pairs directly with CQRS Read-Model Projection Pipelines, which consumes the events designed here to build query-side views. Certification relevance: this material maps to SAA-C03 (Domain 1: resilient architectures), SAP-C02 (Domain 1/3: multi-tier and integration design) and DVA-C02 (event sources, idempotency, Step Functions) — the SNS/SQS/EventBridge triangle is among the most-tested topics.

Core concepts

Before any service detail, four ideas carry the whole discipline: what an event is, who owns the flow, what delivery is actually guaranteed, and what the envelope looks like.

Events, commands and queries are different contracts

An event is an immutable statement that something happened — past tense, owned by the producer, with zero expectation about who reacts. A command is an instruction to do something — imperative, addressed to a specific handler, where the sender cares about completion. A query requests data now. Mixing these up is the root of most bad async designs: a “SendWelcomeEmail” message on a shared event bus is a command wearing an event’s clothes, and it silently couples the producer to the email team’s implementation.

The practical rule: facts on the bus, work on a queue. EventBridge carries events; each consumer that must do work about the event gets its own SQS queue subscribed to the bus, so retries, backlog and DLQ policy are per-consumer, not shared.

Choreography vs orchestration

Choreography means each service reacts to events and emits its own; the flow is emergent, no one owns it. Orchestration means one component (a Step Functions state machine) explicitly calls each step, holds the state, and handles branch/retry/compensation. Neither is “correct” — they solve different shapes of flow, and mature systems use both: choreography between bounded contexts, orchestration inside a business transaction.

A useful smell test: if you find yourself building “flow tracking” tables and timeout sweepers around choreographed events, that flow wanted to be a state machine. Conversely, if a state machine’s steps are just “notify five unrelated teams”, that wanted to be a bus.

Delivery guarantees: the at-least-once reality

Every hop in this stack is at-least-once by default. Networks duplicate, pollers time out and redeliver, retries re-execute. “Exactly-once” appears with heavy qualifiers — SQS FIFO offers exactly-once processing within a 5-minute deduplication window on the producer side, and Step Functions Standard offers exactly-once workflow state transitions — and neither absolves your consumer of receiving a message twice. The design consequence is non-negotiable: every consumer must be idempotent (covered in depth later).

The event envelope

EventBridge imposes a standard envelope; adopt its philosophy even for payloads on SNS/SQS. A production-grade event carries identity (for idempotency), lineage (for tracing) and versioning (for evolution):

{
  "version": "0",
  "id": "9d7bd4f2-3a11-4e8a-b3a0-0d2a6a4d61f1",
  "detail-type": "OrderPlaced",
  "source": "com.meridiankart.orders",
  "account": "111122223333",
  "time": "2026-07-07T14:28:11Z",
  "region": "ap-south-1",
  "resources": ["arn:aws:dynamodb:ap-south-1:111122223333:table/orders"],
  "detail": {
    "eventVersion": "1.2",
    "orderId": "ord_01J9ZK7QW3",
    "idempotencyKey": "ord_01J9ZK7QW3#placed",
    "correlationId": "req_7f3a2b",
    "customerId": "cus_88412",
    "amount": { "value": 4299.00, "currency": "INR" },
    "items": [{ "sku": "SKU-1181", "qty": 2 }],
    "occurredAt": "2026-07-07T14:28:10.812Z"
  }
}

Keep events thin-ish: enough data that common consumers don’t have to call back to the producer (avoiding re-coupling), but never the full aggregate — the 256 KB entry limit, PII minimisation and schema stability all push against fat events. If a consumer occasionally needs more, include the resource ARN/ID and let it fetch; if the payload is genuinely large (documents, images), use the claim-check pattern: put the object in S3 and publish the pointer.

EventBridge deep dive: buses, rules, patterns, pipes, scheduler

EventBridge is the routing brain of AWS event architectures: a serverless bus that receives events, matches them against rules with content-based patterns, transforms and delivers to targets with per-target retry and DLQ. It evolved from CloudWatch Events — which is why the Terraform resources are still named aws_cloudwatch_event_* and the metrics live in the AWS/Events namespace.

Buses: default, custom, partner

Cross-account delivery works by making another account’s bus a target of your rule, with the receiving bus’s resource policy authorising the sender. Keep the rule-of-thumb: producers publish to their bus; a forwarding rule sends to the hub; consumers subscribe in their own accounts. For multi-region resilience, global endpoints give you a Route 53-health-checked endpoint that fails PutEvents over to a secondary region with optional replication (at-least-once — replicated events will duplicate; your idempotency layer absorbs it).

Rules and event patterns: the routing language

A rule matches events with an event pattern — a JSON document where every specified field must match (implicit AND), and arrays inside a field mean OR. Since content filtering launched, patterns are a small query language:

A real pattern that routes high-value Indian orders to a fraud-review queue:

{
  "source": ["com.meridiankart.orders"],
  "detail-type": ["OrderPlaced"],
  "detail": {
    "amount": {
      "currency": ["INR"],
      "value": [{ "numeric": [">=", 50000] }]
    },
    "paymentMethod": [{ "anything-but": ["giftcard"] }]
  }
}

Test patterns before you trust them — the single most valuable EventBridge CLI command:

aws events test-event-pattern \
  --event-pattern file://pattern.json \
  --event file://sample-event.json
# => { "Result": true }   # false means your rule silently drops this event

Create the bus, rule and a target with production-grade retry and DLQ settings:

aws events create-event-bus --name orders-prod

aws events put-rule \
  --name orders-high-value --event-bus-name orders-prod \
  --event-pattern file://pattern.json --state ENABLED

aws events put-targets --rule orders-high-value --event-bus-name orders-prod \
  --targets '[{
    "Id": "fraud-queue",
    "Arn": "arn:aws:sqs:ap-south-1:111122223333:fraud-review",
    "RetryPolicy": { "MaximumRetryAttempts": 185, "MaximumEventAgeInSeconds": 86400 },
    "DeadLetterConfig": { "Arn": "arn:aws:sqs:ap-south-1:111122223333:fraud-review-eb-dlq" }
  }]'

aws events put-events --entries '[{
  "EventBusName": "orders-prod",
  "Source": "com.meridiankart.orders",
  "DetailType": "OrderPlaced",
  "Detail": "{\"orderId\":\"ord_01J9ZK7QW3\",\"amount\":{\"value\":52999,\"currency\":\"INR\"}}"
}]'

Note the last command’s response contains FailedEntryCount — PutEvents is a batch API (up to 10 entries) that can partially fail; code that ignores per-entry ErrorCode loses events under throttling and believes it didn’t.

Targets, input transformation, retry and DLQ

Each rule fans out to at most 5 targets (hard limit — fan out wider via SNS/SQS or multiple rules). Delivery is per-target: one target failing doesn’t affect the others.

API destinations deserve a mention: they make any HTTPS endpoint a target, with a connection object handling auth (API key, Basic, OAuth client credentials) and per-endpoint rate limiting — replacing a whole class of “Lambda that calls a webhook” glue.

EventBridge Pipes: point-to-point without glue Lambdas

EventBridge Pipes connect one source to one target with optional filtering and enrichment — the managed replacement for “Lambda that reads SQS, calls an API, writes to somewhere”. Pipes are also how you get DynamoDB Streams or Kinesis data onto a bus cleanly.

The canonical use in this article’s architecture: DynamoDB Stream → Pipe (filter INSERTs) → EventBridge bus, which turns a table write into a published event — the serverless outbox relay (see the outbox section).

EventBridge Scheduler vs scheduled rules

Old habit: cron rules on the default bus. New default: EventBridge Scheduler, a separate serverless scheduler purpose-built for it.

Schema registry, archive and replay

The schema registry can infer schemas from events flowing on a bus (schema discovery), version them, and generate code bindings (Java/Python/TypeScript). Turn discovery on in non-prod continuously and on prod temporarily when auditing drift — discovery bills per ingested event after the free tier (first 5M discovered events/month), and exported schemas belong in your CI contract tests.

Archive and replay is your async time machine: an archive attached to a bus records events (optionally filtered by a pattern) with a retention you choose (N days or indefinite); a replay re-publishes a time-slice of the archive onto the bus, where rules process them again. Three properties matter operationally: replayed events carry a replay-name field (so consumers can branch or ignore), replay is not order-preserving relative to original arrival, and replay pressure hits every matching rule unless you scope the replay to specific rules — so design consumers to tolerate replay from day one, which idempotency already gives you.

aws events create-archive --archive-name orders-prod-all \
  --event-source-arn arn:aws:events:ap-south-1:111122223333:event-bus/orders-prod \
  --retention-days 90

aws events start-replay --replay-name backfill-fraud-2026-07-06 \
  --event-source-arn arn:aws:events:ap-south-1:111122223333:archive/orders-prod-all \
  --event-start-time 2026-07-06T00:00:00Z --event-end-time 2026-07-06T23:59:59Z \
  --destination '{"Arn":"arn:aws:events:ap-south-1:111122223333:event-bus/orders-prod","FilterArns":["arn:aws:events:ap-south-1:111122223333:rule/orders-prod/orders-high-value"]}'

Limits and quotas that shape designs

Terraform: the whole routing layer as code

resource "aws_cloudwatch_event_bus" "orders" {
  name = "orders-prod"
}

resource "aws_cloudwatch_event_rule" "high_value" {
  name           = "orders-high-value"
  event_bus_name = aws_cloudwatch_event_bus.orders.name
  event_pattern = jsonencode({
    source        = ["com.meridiankart.orders"]
    "detail-type" = ["OrderPlaced"]
    detail = {
      amount = {
        currency = ["INR"]
        value    = [{ numeric = [">=", 50000] }]
      }
    }
  })
}

resource "aws_cloudwatch_event_target" "fraud_queue" {
  rule           = aws_cloudwatch_event_rule.high_value.name
  event_bus_name = aws_cloudwatch_event_bus.orders.name
  arn            = aws_sqs_queue.fraud_review.arn

  retry_policy {
    maximum_retry_attempts       = 185
    maximum_event_age_in_seconds = 86400
  }
  dead_letter_config {
    arn = aws_sqs_queue.fraud_review_eb_dlq.arn
  }
}

# The EB->SQS permission lives on the QUEUE policy, not IAM:
resource "aws_sqs_queue_policy" "fraud_review" {
  queue_url = aws_sqs_queue.fraud_review.id
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Service = "events.amazonaws.com" }
      Action    = "sqs:SendMessage"
      Resource  = aws_sqs_queue.fraud_review.arn
      Condition = { ArnEquals = { "aws:SourceArn" = aws_cloudwatch_event_rule.high_value.arn } }
    }]
  })
}

SNS vs SQS vs EventBridge: the routing decision

The most-asked design question on AWS integration, and the answer is a matrix, not a slogan. The one-line versions: SQS is a buffer (one logical consumer, durable, metered), SNS is a megaphone (push fan-out to many, huge scale, thinner filtering), EventBridge is a router (content-based rules, SaaS/AWS integration, schema/archive machinery, modest throughput ceilings and higher latency).

Read the matrix with three tie-breakers in mind: react to AWS service state changes on the default bus (those events already arrive there, free); take strict per-entity ordering to SQS FIFO (message groups are the only ordering primitive in this trio); and keep sub-10 ms hot paths out of all three — bus and queue hops cost tens to hundreds of milliseconds by design.

The composite pattern — EventBridge for routing, SQS per consumer for execution — appears in almost every mature AWS event architecture, and it is exactly what the reference architecture later in this article shows. SNS earns its slot when you need its delivery channels (mobile push, SMS, email) or when fan-out counts exceed EventBridge’s 5-targets-per-rule comfort zone, such as broadcasting to thousands of tenant queues.

SQS deep dive: standard vs FIFO, visibility, DLQs and redrive

SQS is the workhorse: a fully managed queue with 14-day retention, per-message locking, and the redrive machinery every other service borrows for its DLQs. Its knobs are few but every one of them causes production incidents when mis-set.

Standard vs FIFO

The FIFO trap that catches everyone: parallelism equals the number of distinct active message groups. A FIFO queue where every message carries MessageGroupId=orders is a single-file line — one worker, 300 msg/s ceiling, head-of-line blocking on any poison message. Key groups on the entity whose order matters (orderId, accountId, deviceId), never on a constant.

The settings matrix

Limits worth memorising

Visibility timeout mechanics — where duplicates are born

When a consumer receives a message, SQS doesn’t delete it; it hides it for the visibility timeout. If the consumer finishes and calls DeleteMessage in time, done. If not — crash, timeout, GC pause, Lambda function timeout — the message reappears and another worker processes it again. This is the single biggest source of duplicates in AWS event systems, and it’s tunable:

• Rule of thumb with Lambda: queue visibility timeout ≥ 6× the function timeout (the official guidance), because the poller may hold batches briefly before invoking, and throttled invocations return batches late.
• Long jobs: heartbeat with ChangeMessageVisibility to extend while work progresses; on failure, set visibility to 0 to make the message immediately retryable.
• ApproximateReceiveCount rides on every received message — log it; a value >1 tells you this is a retry, which your idempotency layer should treat accordingly.

DLQs and redrive — the poison-message escape hatch

A redrive policy moves a message to the DLQ after maxReceiveCount failed receives. Operational rules that separate teams who survive incidents from teams who don’t:

1. DLQ retention = 14 days, always. The expiry clock started when the message was first sent to the source queue; a 4-day DLQ can silently expire messages that spent 3 days retrying.
2. Alarm on ApproximateNumberOfMessagesVisible > 0 on every DLQ. A DLQ without an alarm is a data-loss device with extra steps.
3. Redrive is now a button (and an API). After fixing the consumer bug, move messages back:

# Create queue + DLQ with redrive, long polling and sane visibility
aws sqs create-queue --queue-name orders-email-dlq \
  --attributes MessageRetentionPeriod=1209600

aws sqs create-queue --queue-name orders-email \
  --attributes '{
    "VisibilityTimeout": "180",
    "ReceiveMessageWaitTimeSeconds": "20",
    "MessageRetentionPeriod": "345600",
    "RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:ap-south-1:111122223333:orders-email-dlq\",\"maxReceiveCount\":\"5\"}"
  }'

# After the fix: drain the DLQ back to the source at a controlled rate
aws sqs start-message-move-task \
  --source-arn arn:aws:sqs:ap-south-1:111122223333:orders-email-dlq \
  --max-number-of-messages-per-second 50
aws sqs list-message-move-tasks \
  --source-arn arn:aws:sqs:ap-south-1:111122223333:orders-email-dlq

The rate limit on start-message-move-task matters: redriving 200k messages at full speed into a consumer that just recovered is how you cause the second outage of the day.

Lambda event source mappings: batching, scaling, partial batch failure

Lambda consumes queues and streams through an event source mapping (ESM) — a fleet of pollers Lambda runs for you that receives batches, invokes your function synchronously, and manages deletes/checkpoints. Understanding who retries what — the ESM, the service, or Lambda’s async queue — is the difference between designed behaviour and mystery duplicates.

Who retries what

Two structural consequences: with SQS, your function must finish inside the visibility timeout or you get concurrent duplicate processing; with async sources (EventBridge/SNS), Lambda’s async event age (default 6 h, max) and 2 retries mean sustained throttling can drop events to the on-failure destination — configure one:

aws lambda put-function-event-invoke-config \
  --function-name notify-customer \
  --maximum-retry-attempts 2 --maximum-event-age-in-seconds 3600 \
  --destination-config '{"OnFailure":{"Destination":"arn:aws:sqs:ap-south-1:111122223333:notify-async-dlq"}}'

The ESM settings matrix

Scaling behaviour differs fundamentally: for SQS standard, Lambda starts with a handful of concurrent batches and scales up aggressively as backlog grows (hundreds of additional concurrency per minute), bounded by MaximumConcurrency, reserved concurrency and the account limit. For SQS FIFO, concurrency ≤ number of active message groups. For streams, concurrency = shards × parallelization factor — you scale by resharding, not by backlog.

Partial batch failure: the contract most teams get wrong

Default behaviour: your function throws on record 17 of 25 → the entire batch returns to the queue → 24 successfully processed messages get reprocessed (hello, duplicate emails) and will eventually dead-letter alongside the poison one. Partial batch responses fix this — but only if you both enable ReportBatchItemFailures on the ESM and return the right shape:

def handler(event, context):
    failures = []
    for record in event["Records"]:
        try:
            process(record)  # idempotent by design
        except Exception:
            failures.append({"itemIdentifier": record["messageId"]})
    # SQS: only these return to the queue; the rest are deleted.
    # Streams: checkpoint advances to just before the FIRST failure.
    return {"batchItemFailures": failures}

The semantics you must internalise:

aws lambda create-event-source-mapping \
  --function-name process-order \
  --event-source-arn arn:aws:sqs:ap-south-1:111122223333:orders-email \
  --batch-size 25 --maximum-batching-window-in-seconds 5 \
  --function-response-types ReportBatchItemFailures \
  --scaling-config MaximumConcurrency=50

resource "aws_lambda_event_source_mapping" "orders_email" {
  event_source_arn                   = aws_sqs_queue.orders_email.arn
  function_name                      = aws_lambda_function.process_order.arn
  batch_size                         = 25
  maximum_batching_window_in_seconds = 5
  function_response_types            = ["ReportBatchItemFailures"]

  scaling_config {
    maximum_concurrency = 50
  }
}

Step Functions: standard vs express, sagas and compensation

When the reaction to an event is itself a multi-step business transaction — reserve inventory, charge payment, create shipment — choreographing it as more events scatters the failure handling. AWS Step Functions makes the flow explicit: a state machine defined in ASL (Amazon States Language), with per-state retry, catch, timeouts, parallelism and human-readable execution history.

Standard vs Express

Two traps hide in that table. First, Express-async runs at-least-once: an Express workflow with non-idempotent side effects can execute those side effects twice — sagas with real money belong on Standard. Second, Standard’s 25,000-event history limit kills naive Map/loop designs over big datasets; use a Distributed Map (up to 10,000 concurrent child workflows, each with its own history) or chunk via S3.

Retry and Catch: the error-handling fields

Canonical error names you’ll match on: States.Timeout, States.TaskFailed, States.Permissions, States.HeartbeatTimeout, States.DataLimitExceeded (the 256 KB inter-state payload cap), plus the Lambda-specific transient trio Lambda.ServiceException, Lambda.AWSLambdaException, Lambda.SdkClientException, Lambda.TooManyRequestsException — retry those; don’t retry your own validation errors.

The saga pattern: compensate, don’t rollback

A saga decomposes a distributed transaction into local transactions, each with a compensating action that semantically undoes it. Step Functions expresses this naturally: forward steps chained with Catch routes into a compensation chain that runs in reverse order of what succeeded.

{
  "StartAt": "ReserveInventory",
  "States": {
    "ReserveInventory": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "reserve-inventory", "Payload.$": "$" },
      "TimeoutSeconds": 30,
      "Retry": [{
        "ErrorEquals": ["Lambda.ServiceException", "Lambda.TooManyRequestsException"],
        "IntervalSeconds": 2, "MaxAttempts": 4, "BackoffRate": 2.0, "JitterStrategy": "FULL"
      }],
      "Catch": [{ "ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "OrderFailed" }],
      "Next": "ChargePayment"
    },
    "ChargePayment": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "charge-payment", "Payload.$": "$" },
      "TimeoutSeconds": 60,
      "Retry": [{ "ErrorEquals": ["GatewayTimeout"], "IntervalSeconds": 3, "MaxAttempts": 3, "BackoffRate": 2.0 }],
      "Catch": [{ "ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "ReleaseInventory" }],
      "Next": "CreateShipment"
    },
    "CreateShipment": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "create-shipment", "Payload.$": "$" },
      "TimeoutSeconds": 45,
      "Catch": [{ "ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "RefundPayment" }],
      "Next": "OrderConfirmed"
    },
    "RefundPayment":     { "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "refund-payment", "Payload.$": "$" },
      "Retry": [{ "ErrorEquals": ["States.ALL"], "IntervalSeconds": 5, "MaxAttempts": 5, "BackoffRate": 2.0 }],
      "Next": "ReleaseInventory" },
    "ReleaseInventory":  { "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "release-inventory", "Payload.$": "$" },
      "Retry": [{ "ErrorEquals": ["States.ALL"], "IntervalSeconds": 5, "MaxAttempts": 5, "BackoffRate": 2.0 }],
      "Next": "OrderFailed" },
    "OrderFailed":    { "Type": "Fail", "Error": "OrderSagaFailed", "Cause": "Compensated" },
    "OrderConfirmed": { "Type": "Succeed" }
  }
}

Design rules for sagas that hold up: compensations must be idempotent and aggressively retried (a refund that fails is a lawsuit, not a log line); use semantic locks (mark the order PENDING so other flows don’t act on half-done state); and start the saga from the event queue with an idempotent execution name (order-{orderId}) so a duplicate OrderPlaced event cannot spawn a second saga — Standard workflows reject duplicate names for 90 days, one of the most underrated dedup mechanisms on AWS.

Wire the saga to the bus directly (EventBridge target type StartExecution) or — better for bursty load — EventBridge → SQS → Lambda → StartExecution, which lets you meter saga starts against the Standard start-rate quota.

Idempotency, ordering and the outbox pattern

Idempotency: the tax at-least-once charges

An operation is idempotent when applying it twice has the same effect as once. Every consumer in this architecture needs it. The strategies, strongest-first:

The DynamoDB pattern in six lines — the whole trick is that the check and the claim are one atomic operation:

import boto3, botocore, time
ddb = boto3.client("dynamodb")

def process_once(idempotency_key: str, event) -> bool:
    try:
        ddb.put_item(
            TableName="idempotency",
            Item={"pk": {"S": idempotency_key},
                  "expiresAt": {"N": str(int(time.time()) + 7 * 86400)}},
            ConditionExpression="attribute_not_exists(pk)")
    except botocore.exceptions.ClientError as e:
        if e.response["Error"]["Code"] == "ConditionalCheckFailedException":
            return False          # duplicate — skip side effects
        raise
    do_side_effects(event)        # charge, email, ship…
    return True

Key selection is the design decision: use a business key (orderId#placed), not the transport’s message ID — EventBridge assigns a new id on every retry-with-republish and replay, SQS assigns a new messageId when the same logical event is re-sent, so transport IDs dedup less than you think. And write the idempotency record with a TTL (7–30 days) or the table becomes an unbounded bill.

Ordering: where it exists and where it’s a myth

Because global ordering doesn’t exist, resilient consumers use version-aware writes instead of relying on arrival order: every event carries the aggregate version (or occurredAt), and the consumer applies it only if newer than what it has (ConditionExpression: version < :incoming). That single habit converts “strict ordering required” into “per-entity eventual convergence” — a far cheaper property to operate. Reserve true FIFO for flows where intermediate states matter (ledgers, inventory counts), not just final state.

The dual-write problem and the transactional outbox

The bug: your service writes the order to its database and then publishes OrderPlaced. Crash between the two → order exists, event never sent, downstream never knows. Reverse the order → event sent, DB write fails, downstream believes in a phantom order. Any design with two independent writes has this hole; retries don’t fix it, they just move it.

The transactional outbox closes it: write the state change and the event into the same atomic transaction (an outbox table/attribute alongside the business rows), then a separate relay reads committed outbox rows and publishes them. The event now exists if-and-only-if the state change committed.

The DynamoDB flavour is beautifully small — one Pipe replaces the entire relay:

resource "aws_pipes_pipe" "orders_outbox" {
  name     = "orders-outbox-relay"
  role_arn = aws_iam_role.pipe.arn
  source   = aws_dynamodb_table.orders.stream_arn
  target   = aws_cloudwatch_event_bus.orders.arn

  source_parameters {
    dynamodb_stream_parameters {
      starting_position                  = "LATEST"
      batch_size                         = 10
      maximum_retry_attempts             = 8
      dead_letter_config { arn = aws_sqs_queue.outbox_pipe_dlq.arn }
    }
    filter_criteria {
      filter { pattern = jsonencode({ eventName = ["INSERT"] }) }
    }
  }
  target_parameters {
    eventbridge_event_bus_parameters {
      source      = "com.meridiankart.orders"
      detail_type = "OrderPlaced"
    }
  }
}

Note what the outbox does not give you: it guarantees at-least-once publication, not exactly-once — the relay can crash after publishing and before checkpointing. Producer-side outbox + consumer-side idempotency is the complete, honest answer to “exactly-once on AWS”: effectively-once effects, built from two at-least-once halves.

Error handling and DLQ strategy per hop

Every hop needs an answer to three questions: who retries, for how long, and where does the event go when retries exhaust. Design it as a table — literally this table, filled in per pipeline, belongs in your runbook:

Three principles turn that table into policy. (1) Retryable vs terminal: retry throttles, timeouts and 5xx with backoff + jitter; never retry validation errors — dead-letter them immediately with a reason attribute. (2) DLQ contents differ per hop: an EventBridge target DLQ holds the original event plus error attributes (ERROR_CODE, RULE_ARN); a stream on-failure destination holds only pointers — fetch the records before stream retention expires. (3) Every DLQ gets an owner, an alarm and a redrive runbook. An unowned DLQ is where events go to die legally.

# The alarm that pages you before customers do: oldest message age on the work queue
aws cloudwatch put-metric-alarm \
  --alarm-name orders-email-backlog-age \
  --namespace AWS/SQS --metric-name ApproximateAgeOfOldestMessage \
  --dimensions Name=QueueName,Value=orders-email \
  --statistic Maximum --period 60 --evaluation-periods 5 \
  --threshold 600 --comparison-operator GreaterThanThreshold \
  --alarm-actions arn:aws:sns:ap-south-1:111122223333:oncall

Observability of async flows: tracing through events

A synchronous request shows up as one trace. An event-driven flow is five services, three queues and two retries — without deliberate correlation it is unobservable. You need three layers: traces (X-Ray), metrics (per-hop CloudWatch), and correlation IDs in structured logs.

X-Ray trace propagation: what connects and what breaks

The honest summary: X-Ray (or OpenTelemetry via ADOT exporting to X-Ray) will stitch most of the path, but batching and replay force you to also carry a correlationId inside the event detail, logged as a structured field by every consumer. Then CloudWatch Logs Insights answers “what happened to order ord_01J9ZK7QW3” across every log group at once:

fields @timestamp, @log, level, msg, detail.orderId as orderId
| filter correlationId = "req_7f3a2b" or detail.correlationId = "req_7f3a2b"
| sort @timestamp asc
| limit 200

The metrics that matter per hop

Throttling and backpressure: designing the flow control

Asynchronous doesn’t mean unlimited — it means you choose where the pressure accumulates. The design question for every pipeline: when input exceeds downstream capacity, which component absorbs, and which component sheds?

The mental model: SQS is the only component in this stack designed to hold pressure for days — so put a queue immediately upstream of every capacity-constrained consumer, cap that consumer’s concurrency at what its downstream survives, and alarm on queue age. Backpressure you designed is a backlog metric; backpressure you didn’t design is an outage.

Architecture at a glance

The reference architecture assembles every piece of this article into one order-processing pipeline. Read it left to right. Producers — an API Gateway-fronted order service and its DynamoDB outbox relay — publish OrderPlaced facts onto a custom EventBridge bus (orders-prod), where an archive records everything for replay. Rules content-match events and fan out: high-value orders to a fraud-review FIFO queue (grouped by orderId so per-order sequence holds), notification work to an SNS topic feeding per-channel queues, and every matched target carrying its own retry policy and DLQ. Consumers pull through Lambda event source mappings tuned with batching and partial batch responses, while the order saga runs in Step Functions Standard with compensation states. Side effects land behind an idempotency table in DynamoDB (conditional writes keyed on the business idempotency key), analytics events settle into S3, and the whole flow is stitched together by X-Ray trace propagation plus per-hop CloudWatch alarms. The numbered badges mark the six decision points this article drilled into: silent pattern non-match, FIFO group/throughput design, DLQ + redrive, partial batch failure, the idempotency gate, and archive replay semantics.

Trace one event through it: checkout writes the order row; the outbox pipe publishes OrderPlaced within a second; rules deliver to the FIFO queue and fan out via SNS; the fraud Lambda fails two records of a 25-record batch and returns only those via partial batch response; the saga reserves, charges and ships — or compensates in reverse; and every side effect checks the idempotency table first, so the duplicate delivery that will eventually happen becomes a no-op. Nothing in the pipeline depends on nothing failing — every hop assumes its neighbour will, eventually, and has a numbered answer for it.

Real-world scenario: MeridianKart’s flash-sale meltdown and rebuild

MeridianKart, a Pune-based marketplace doing ~1.2 million orders/day, ran its original “event-driven” pipeline as one SNS topic (prod-events) with nine Lambda subscribers. During a February flash sale peaking at 850 orders/second, three things failed at once. The invoice Lambda — non-idempotent — double-generated 41,000 invoices because SNS redelivered during a dependency brown-out. The inventory Lambda fell behind and, with no queue in front of it, Lambda’s async retries (2, then drop) silently discarded roughly 18,000 stock decrements once the on-failure destination — never configured — had nothing to catch. And the fraud service, which had moved to a FIFO queue “for safety”, throttled at exactly 300 messages/second: the team had set MessageGroupId to the constant string "orders", serialising the entire marketplace through one message group, and wasn’t batching sends. Recovery took 11 hours, most of it spent reconciling inventory from order history because there was no replayable record of events.

The rebuild followed the reference architecture above. Orders write to DynamoDB; a Pipe relays the stream onto a custom bus (orders-prod) — the outbox pattern ending the dual-write bug that had also been quietly dropping ~0.02% of events for months. Rules route to per-consumer SQS queues: standard queues for notifications and analytics, FIFO only for inventory (grouped by sku) and payments (grouped by orderId), with batched sends lifting the FIFO ceiling to 3,000 msg/s and high-throughput mode enabled as headroom. Every queue got a DLQ with maxReceiveCount=5, a 14-day retention and an age alarm; every consumer got ReportBatchItemFailures and an idempotency check against a DynamoDB table with 14-day TTL (write cost: about ₹450/month at their volume — the cheapest insurance they buy). The checkout saga moved to Step Functions Standard with compensation states and execution names of order-{orderId}, which — to the team’s surprise — found a duplicate-event bug in week one when 2,300 StartExecution calls failed with ExecutionAlreadyExists, each one a double-charge that hadn’t happened.

The next sale peaked at 1,100 orders/second. PutEvents throttled briefly (SDK adaptive retries absorbed it), the inventory queue aged to 4 minutes before draining — watched, not feared — and the DLQs collected 62 messages, all traced to one malformed seller SKU, redriven after a one-line fix with start-message-move-task at 50 msg/s. Total incident count: zero. The architecture didn’t remove failure; it gave every failure a place to sit and a runbook to follow.

Advantages and disadvantages

The disadvantages are not reasons to avoid EDA; they are the engineering bill for its advantages. The failure mode worth naming: teams adopt the topology (buses, queues) without the disciplines (idempotency, DLQ ownership, correlation IDs) and end up with distributed systems problems and none of the resilience payoff. Adopt the disciplines first — they’re cheap; the topology is easy afterwards.

Hands-on lab: bus → rule → queue → Lambda with DLQ, partial batch and replay

Free-tier friendly: a few hundred events cost fractions of a rupee (EventBridge bills ~$0.000001 per event; SQS/Lambda stay inside their free tiers). You need the AWS CLI v2 configured and permissions for Events, SQS, Lambda, IAM and CloudWatch. Region below is ap-south-1; adjust freely.

1. Create the bus, archive, queues.

export ACC=$(aws sts get-caller-identity --query Account --output text)
export REG=ap-south-1

aws events create-event-bus --name lab-orders
aws events create-archive --archive-name lab-orders-all \
  --event-source-arn arn:aws:events:$REG:$ACC:event-bus/lab-orders --retention-days 1

aws sqs create-queue --queue-name lab-orders-dlq \
  --attributes MessageRetentionPeriod=1209600
aws sqs create-queue --queue-name lab-orders-q --attributes "{
  \"VisibilityTimeout\": \"60\",
  \"ReceiveMessageWaitTimeSeconds\": \"20\",
  \"RedrivePolicy\": \"{\\\"deadLetterTargetArn\\\":\\\"arn:aws:sqs:$REG:$ACC:lab-orders-dlq\\\",\\\"maxReceiveCount\\\":\\\"2\\\"}\"
}"

2. Rule with a content filter, SQS target, target DLQ. Save as pattern.json:

{
  "source": ["lab.orders"],
  "detail-type": ["OrderPlaced"],
  "detail": { "amount": [{ "numeric": [">=", 500] }] }
}

aws events put-rule --name lab-high-value --event-bus-name lab-orders \
  --event-pattern file://pattern.json

# Allow EventBridge to send to the queue
aws sqs set-queue-attributes --queue-url https://sqs.$REG.amazonaws.com/$ACC/lab-orders-q \
  --attributes "{\"Policy\": \"{\\\"Version\\\":\\\"2012-10-17\\\",\\\"Statement\\\":[{\\\"Effect\\\":\\\"Allow\\\",\\\"Principal\\\":{\\\"Service\\\":\\\"events.amazonaws.com\\\"},\\\"Action\\\":\\\"sqs:SendMessage\\\",\\\"Resource\\\":\\\"arn:aws:sqs:$REG:$ACC:lab-orders-q\\\"}]}\"}"

aws events put-targets --rule lab-high-value --event-bus-name lab-orders \
  --targets "[{\"Id\":\"q1\",\"Arn\":\"arn:aws:sqs:$REG:$ACC:lab-orders-q\",
    \"RetryPolicy\":{\"MaximumRetryAttempts\":10,\"MaximumEventAgeInSeconds\":3600},
    \"DeadLetterConfig\":{\"Arn\":\"arn:aws:sqs:$REG:$ACC:lab-orders-dlq\"}}]"

3. A consumer that fails on purpose — records with amount == 999 fail, proving partial batch behaviour. Save as app.py, zip, deploy:

import json

def handler(event, context):
    failures = []
    for r in event["Records"]:
        body = json.loads(r["body"])
        amount = body.get("detail", {}).get("amount", 0)
        print(json.dumps({"msg": "processing", "amount": amount,
                          "receiveCount": r["attributes"]["ApproximateReceiveCount"]}))
        if amount == 999:
            failures.append({"itemIdentifier": r["messageId"]})
    return {"batchItemFailures": failures}

zip fn.zip app.py
aws iam create-role --role-name lab-eda-fn --assume-role-policy-document '{
  "Version":"2012-10-17","Statement":[{"Effect":"Allow",
  "Principal":{"Service":"lambda.amazonaws.com"},"Action":"sts:AssumeRole"}]}'
aws iam attach-role-policy --role-name lab-eda-fn \
  --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaSQSQueueExecutionRole

aws lambda create-function --function-name lab-eda-consumer \
  --runtime python3.12 --handler app.handler --zip-file fileb://fn.zip \
  --role arn:aws:iam::$ACC:role/lab-eda-fn --timeout 10

aws lambda create-event-source-mapping --function-name lab-eda-consumer \
  --event-source-arn arn:aws:sqs:$REG:$ACC:lab-orders-q \
  --batch-size 10 --function-response-types ReportBatchItemFailures

4. Publish test events — one below the filter (dropped by the rule), one good, one poison:

aws events put-events --entries "[
 {\"EventBusName\":\"lab-orders\",\"Source\":\"lab.orders\",\"DetailType\":\"OrderPlaced\",\"Detail\":\"{\\\"orderId\\\":\\\"o-1\\\",\\\"amount\\\":100}\"},
 {\"EventBusName\":\"lab-orders\",\"Source\":\"lab.orders\",\"DetailType\":\"OrderPlaced\",\"Detail\":\"{\\\"orderId\\\":\\\"o-2\\\",\\\"amount\\\":750}\"},
 {\"EventBusName\":\"lab-orders\",\"Source\":\"lab.orders\",\"DetailType\":\"OrderPlaced\",\"Detail\":\"{\\\"orderId\\\":\\\"o-3\\\",\\\"amount\\\":999}\"}]"
# Expect: "FailedEntryCount": 0

5. Observe the machinery work. o-1 never reaches the queue (pattern filtered — verify with aws events test-event-pattern). o-2 processes cleanly. o-3 fails, returns via partial batch response, retries once (watch receiveCount climb in the logs), then lands in the DLQ after maxReceiveCount=2:

aws logs tail /aws/lambda/lab-eda-consumer --since 10m --follow &
sleep 90
aws sqs get-queue-attributes \
  --queue-url https://sqs.$REG.amazonaws.com/$ACC/lab-orders-dlq \
  --attribute-names ApproximateNumberOfMessagesVisible
# => "1"  — the poison message, quarantined, with the good ones unharmed

6. Replay from the archive — the payoff step. Pretend the consumer had a bug; re-run the last hour:

aws events start-replay --replay-name lab-replay-1 \
  --event-source-arn arn:aws:events:$REG:$ACC:archive/lab-orders-all \
  --event-start-time $(date -u -v-1H +%Y-%m-%dT%H:%M:%SZ) \
  --event-end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) \
  --destination "{\"Arn\":\"arn:aws:events:$REG:$ACC:event-bus/lab-orders\"}"
aws events describe-replay --replay-name lab-replay-1 --query State

o-2 and o-3 process again — which is exactly why the consumer logged receiveCount and why production consumers check an idempotency table.

7. Teardown.

aws lambda delete-function --function-name lab-eda-consumer
aws events remove-targets --rule lab-high-value --event-bus-name lab-orders --ids q1
aws events delete-rule --name lab-high-value --event-bus-name lab-orders
aws events delete-archive --archive-name lab-orders-all
aws events delete-event-bus --name lab-orders
aws sqs delete-queue --queue-url https://sqs.$REG.amazonaws.com/$ACC/lab-orders-q
aws sqs delete-queue --queue-url https://sqs.$REG.amazonaws.com/$ACC/lab-orders-dlq
aws iam detach-role-policy --role-name lab-eda-fn \
  --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaSQSQueueExecutionRole
aws iam delete-role --role-name lab-eda-fn

Common mistakes & troubleshooting

The playbook, ordered by how often each one pages someone:

Errors you’ll meet in logs and CloudTrail, decoded:

Best practices

1. Facts on the bus, work on queues: EventBridge routes; every real consumer owns an SQS queue with its own DLQ, retry posture and concurrency valve.
2. Standardise the envelope: versioned detail-type (or eventVersion in detail), idempotencyKey, correlationId, occurredAt — enforced by a shared publisher library, documented in the schema registry.
3. Additive-only schema evolution: add optional fields freely; renames/removals/semantic changes get a new detail-type version (OrderPlaced.v2) with a deprecation window for old-rule consumers.
4. DLQ everything, alarm everything: every EventBridge target, SNS subscription, SQS queue and async Lambda gets a dead-letter destination, a > 0 alarm, an owner and a redrive runbook.
5. Idempotency by default: the consumer template includes the DynamoDB (or Powertools) idempotency gate; reviewers treat its absence like a missing auth check.
6. Outbox for anything that matters: if losing the event costs money, publish via DynamoDB Streams/CDC relay, never as a second write after the transaction.
7. Test event patterns in CI: golden sample events + test-event-pattern assertions catch the silent-non-match class before production does.
8. Size FIFO deliberately: group by the entity whose order matters; batch sends; prove peak throughput ≥ 2× forecast in a load test, or use standard + version-aware writes.
9. Set TimeoutSeconds on every Step Functions Task and retry only transient errors — with jitter (JitterStrategy: FULL).
10. Alarm on age, not depth: ApproximateAgeOfOldestMessage and IteratorAge measure SLA risk; depth alone doesn’t.
11. Archive from day one: a 90-day archive on every prod bus costs little and converts “we lost events” into “we replayed 40 minutes.”
12. Load-test against your region’s quotas (PutEvents TPS, FIFO throughput, Lambda concurrency, SFN start rate) and file Service Quotas raises before launch week, not during it.

Security notes

Least privilege per direction. Producers get events:PutEvents scoped to one bus ARN — and you can pin what they publish with IAM condition keys (events:source, events:detail-type), so the payments service physically cannot emit com.meridiankart.orders events. Consumers get sqs:ReceiveMessage/DeleteMessage/GetQueueAttributes on their queue only. The ESM uses the function’s execution role for SQS access — a common review miss.

Resource policies are the cross-service glue — and the attack surface. SQS queue policies must allow events.amazonaws.com/sns.amazonaws.com to SendMessage, but always with aws:SourceArn (and aws:SourceAccount) conditions to block the confused deputy — without them, anyone’s rule or topic can inject messages into your queue. Audit bus policies for over-broad Principal: "*" grants; cross-account access should enumerate account IDs or use aws:PrincipalOrgID.

Encryption. SQS defaults to SSE-SQS (free); use SSE-KMS with a CMK when compliance demands key control, and extend the key policy to every producing service principal (the #1 silent-delivery-failure cause). Custom EventBridge buses support CMK encryption at rest; SNS topics support KMS similarly. In transit is TLS throughout; for private compute, use VPC interface endpoints (com.amazonaws.region.sqs, .events, .states) so queue/bus traffic never crosses the public internet, and pin them with endpoint policies.

Data minimisation beats encryption. Events fan out to consumers you don’t control yet — keep PII out of detail (send customerId, not the address), tokenise where consumers rarely need raw values, and treat the archive as a regulated data store with retention matching your privacy obligations (a 90-day archive of PII-laden events is a GDPR/DPDP discovery problem). CloudTrail records the control plane (PutRule, PutTargets, policy changes — alert on these; a malicious rule is silent exfiltration) while data-plane visibility comes from bus logging and per-service metrics.

Cost & sizing

What drives the bill, with us-east-1 list prices (ap-south-1 within a few percent):

A worked month at MeridianKart scale — 50M business events, each fanning to ~2.4 targets, 5M Lambda invocations (batch 10), 1M sagas averaging 12 transitions:

The lesson in that table: Step Functions Standard dominates — which is why you orchestrate only the flows that need compensation semantics (here, if only 20% of orders truly needed the saga, the SFN line drops to $60), push high-volume simple reactions to plain queue+Lambda choreography, and consider Express for short internal workflows. Right-sizing levers in order of impact: batch everywhere (SQS requests and Lambda invokes both drop ~10×), sample traces, use 64 KB-conscious event payloads, and cap Lambda memory to measured need. At small scale the whole architecture is effectively free: a startup doing 100k events/month pays under $2 — less than ₹200 — for the same resilience posture.

Interview & exam questions

Q1. When would you pick EventBridge over SNS for fan-out? When you need content-based routing on the event body, SaaS/AWS-service event sources, schema registry, or archive/replay. SNS wins on raw fan-out scale (millions of subscriptions), push delivery channels (mobile/SMS/email), and lower per-message latency. Exam tell: “route by payload attributes to different targets” → EventBridge; “notify millions of endpoints” → SNS.

Q2. Why put SQS between EventBridge and Lambda instead of invoking Lambda directly? The queue adds durable buffering (14 days vs 24 h of retries), consumer-controlled pacing (MaximumConcurrency to protect downstreams), batch economics, per-consumer DLQ with redrive, and survives consumer outages longer. Direct invocation is fine for cheap, fast, idempotent reactions.

Q3. Does SQS FIFO give exactly-once delivery? No — exactly-once processing semantics within a 5-minute deduplication window, scoped to producer-side duplicate sends. A consumer that crashes after its side effect but before DeleteMessage still reprocesses the message. End-to-end effectively-once = FIFO dedup + consumer idempotency.

Q4. How do you size an SQS visibility timeout for a Lambda consumer? At least 6× the function timeout (AWS guidance), because the poller can hold batches before invoking and throttles delay processing. Too short → concurrent duplicate processing; too long → slow recovery after consumer crashes. For long jobs, heartbeat with ChangeMessageVisibility.

Q5. What does ReportBatchItemFailures change, and what must the handler do? Without it, one failed record fails the whole batch, reprocessing every good record. With it, the handler returns batchItemFailures with the failed itemIdentifiers; SQS re-delivers only those, and stream sources checkpoint up to the first failure. It must be enabled on the ESM and implemented in the response — either alone does nothing.

Q6. Standard vs Express Step Functions for a payment saga? Standard: exactly-once state transitions, execution-name dedup (90 days), 1-year duration, callbacks, full history — everything money needs. Express is at-least-once (async), 5-minute cap, cheaper per volume — right for high-throughput enrichment, wrong for non-idempotent financial steps.

Q7. Explain the saga pattern vs two-phase commit. 2PC holds locks across services for atomic commit — unavailable/unscalable across microservices and managed services. A saga is a sequence of local transactions, each with a compensating action executed in reverse on failure; consistency is eventual, availability is preserved. Step Functions encodes it with Catch routes to compensation states, which must be idempotent and retried.

Q8. What problem does the transactional outbox solve? The dual-write inconsistency: DB commit and event publish are two non-atomic operations, so a crash between them loses events or creates phantoms. Outbox writes the event in the same transaction as state, and a relay (DynamoDB Streams → Pipe → bus, or CDC) publishes after commit — at-least-once publication tied to committed state.

Q9. Where can events silently disappear in EventBridge, and how do you defend each spot? (1) PutEvents partial failures — check FailedEntryCount; (2) pattern non-match — test-event-pattern in CI + MatchedEvents alarm; (3) target failures past 24 h/185 retries — per-target DLQ; (4) Lambda async exhaustion — on-failure destination; (5) ESM filters deleting SQS messages — filter with care. Archive as the last-resort recovery for all of them.

Q10. How do you trace one business transaction across bus, queues and functions? Enable X-Ray on producers, state machines and functions — EventBridge and SNS propagate trace context; SQS→Lambda appears as linked traces. Because batching/replay break single-trace views, also carry a correlationId in the event detail, log it as a structured field everywhere, and query with Logs Insights.

Cert mapping: Q1/Q2/Q9 are SAA-C03 staples; Q3/Q4/Q5 recur in DVA-C02; Q6/Q7/Q8 are SAP-C02 scenario territory.

Quick check

1. An event arrives at EventBridge but no rule fires and nothing errors. What’s the most likely cause and the one command to confirm it?
2. Your Lambda takes up to 90 s; the SQS visibility timeout is 60 s. What failure will you see?
3. Which combination gives effectively-once side effects: (a) FIFO alone, (b) outbox alone, © outbox + consumer idempotency, (d) exactly-once delivery flag?
4. A Step Functions Express workflow charges cards and sometimes double-charges. Why?
5. What should you alarm on to detect a consumer falling behind its SLA — queue depth or queue age?

Answers

1. The rule’s event pattern doesn’t match (typo, type mismatch). Confirm with aws events test-event-pattern --event-pattern file://p.json --event file://e.json — false means silent drop.
2. Duplicate concurrent processing: at 60 s the message reappears and a second worker processes it while the first still runs; receive counts climb and side effects double. Set visibility ≥ 6× function timeout.
3. ©. Delivery is at-least-once everywhere; the outbox guarantees publication, idempotency makes duplicates harmless. There is no exactly-once delivery flag.
4. Express asynchronous executions run at-least-once — the whole workflow may execute twice. Money flows belong on Standard (exactly-once transitions + execution-name dedup) with idempotent tasks.
5. Age (ApproximateAgeOfOldestMessage). Depth says how much work exists; age says how late you are — a deep-but-fresh queue can be healthy, a shallow-but-old one is an SLA breach.

Glossary

• Event bus — EventBridge router that receives events and evaluates rules against them.
• Event pattern — JSON predicate (AND across fields, OR within arrays, operators like prefix/numeric/anything-but) selecting events for a rule.
• Rule target — destination (≤5 per rule) receiving matched events, with per-target input transform, retry policy and DLQ.
• EventBridge Pipes — managed point-to-point connector: source → filter → enrichment → target.
• Archive & replay — bus-attached event recording plus re-publication of a time slice; replayed events carry replay-name.
• Visibility timeout — period a received SQS message stays hidden; expiry without delete causes redelivery.
• Redrive policy / DLQ — after maxReceiveCount failed receives, SQS moves a message to the dead-letter queue; StartMessageMoveTask drains it back.
• Message group — FIFO ordering unit; strict order within, parallelism across.
• Event source mapping (ESM) — Lambda’s managed poller for SQS/streams, owning batching, filters, scaling and partial-batch semantics.
• Partial batch response — handler contract (batchItemFailures) reporting per-record failures so only those retry.
• Saga — distributed transaction as local steps plus compensating actions, orchestrated here by Step Functions.
• Idempotency key — stable business identifier that makes duplicate processing a no-op via conditional writes.
• Transactional outbox — pattern committing events atomically with state, relayed to the bus afterwards (DynamoDB Streams → Pipe is the serverless form).
• Backpressure — deliberately locating where excess load accumulates (queues) and metering consumers so downstreams survive.

Next steps

• Contrast the design space with GCP Pub/Sub Event-Driven Architecture — one service where AWS uses three, with different ordering and replay trade-offs.
• Feed these events into query-side views with CQRS Read-Model Projection Pipelines & Eventual Consistency — the natural consumer of the bus you just built.
• Go deeper on consumer patterns in AWS Lambda Event-Driven Patterns, including destinations and function URLs.
• Pick the right state stores for projections and idempotency tables in RDS vs DynamoDB vs Aurora.
• Audit the control plane you now depend on — rules, targets and policies — with AWS CloudTrail & Config for Audit and Compliance.