Amazon SNS Hands-On: Topics, Fan-Out, Message Filtering & Subscriptions

An order gets placed on your e-commerce site, and five things need to happen: the warehouse system needs to reserve stock, the billing service needs to charge the card, the notification service needs to email the buyer, the analytics pipeline needs to record the event, and the fraud team’s function needs to score it. The naive design has your checkout code call all five services in a loop — which means checkout is now coupled to five systems, slows to the speed of the slowest one, and breaks entirely when the analytics endpoint is down. Amazon SNS (Simple Notification Service) exists to break exactly that coupling. Your checkout code publishes one message to a topic and moves on; SNS pushes a copy to every interested subscriber. Checkout never knows — and never needs to know — who is listening.

That pattern is publish/subscribe (pub/sub), and SNS is AWS’s managed, serverless implementation of it. A publisher sends a message to a topic (a named channel); zero or more subscribers attach to that topic through a subscription, each naming a protocol — an SQS queue, a Lambda function, an HTTP/S webhook, an email address, an SMS number, a mobile push endpoint, or a Kinesis Data Firehose stream. When you publish once, SNS fans out the message to all of them in parallel, in milliseconds, with no code on your side to manage the list. Add a subscriber next quarter and no publisher changes. That decoupling — one-to-many, push-based, publisher-blind — is the entire value proposition, and it is the backbone of most event-driven architectures on AWS.

By the end of this article you will have built the real thing with your own hands: a topic that fans out to two SQS queues with different filter policies (so each queue receives only the events it asked for) plus a Lambda function (that receives everything), with a subscription dead-letter queue catching anything SNS can’t deliver — first with the aws CLI, then the identical stack as Terraform, then torn down. Along the way you will learn why fronting Lambda with SQS beats subscribing Lambda directly, when you actually need a FIFO topic, how filter policies and raw message delivery change what a subscriber sees, how the topic policy and KMS encryption gate cross-account publishing, and — critically — a symptom-to-fix troubleshooting playbook for the dozen ways a fan-out silently drops messages. Read the prose once; keep the tables open when your own topic misbehaves.

What problem this solves

Without a pub/sub broker, every producer must know every consumer. That is fine with one consumer and unbearable with five: the producer holds a hard-coded list of endpoints, retries each one, handles each one’s outages, and redeploys every time the list changes. Worse, the producer’s latency becomes the sum of all downstream calls, and a single slow or failing consumer can stall or crash the producer. This is temporal coupling (everyone must be up at once) layered on implementation coupling (the producer knows everyone’s address), and it is the reason monoliths are so hard to split.

SNS deletes both couplings at once. The publisher’s only dependency is the topic ARN; it publishes and returns in a few milliseconds regardless of how many subscribers exist or whether they are healthy. SNS owns the fan-out, the parallel delivery, the retries, and the durability. A subscriber that is down does not affect the publisher or the other subscribers — SNS retries it and, if you configured one, drops the undeliverable message into a DLQ for later replay rather than losing it. New subscribers self-serve: they subscribe to the topic (subject to the topic policy) without touching the producer.

Who hits the pain this solves: anyone wiring more than one consumer to an event, anyone splitting a monolith into services, anyone building “when X happens, do A and B and C” workflows. And SNS bites beginners in a very specific set of places — the SQS queue that stays empty because its queue policy doesn’t allow the topic; the subscriber that receives nothing because its filter policy is too strict; the email that never arrives because the subscription is unconfirmed; the consumer that chokes because it’s parsing the SNS envelope instead of turning on raw delivery; the events that vanish under an outage because no DLQ was attached. Get those five right and fan-out is boringly reliable. Here is the whole field on one screen — every piece you will meet, what it is, and the classic trap attached to it:

Learning objectives

By the end of this article you can:

• Explain the pub/sub model — publishers, topics, subscriptions, subscribers — and why it decouples producers from consumers in both time and implementation.
• Name every subscription protocol (SQS, Lambda, HTTP/S, email, email-json, SMS, mobile push/application, Kinesis Data Firehose) and know each one’s confirmation, payload shape and retry behaviour.
• Build the fan-out pattern — one publish to many SQS queues — and articulate exactly why SNS→SQS beats subscribing Lambda directly for durability, retry, backpressure and replay.
• Choose between Standard and FIFO topics by their ordering, deduplication, throughput and subscriber constraints (FIFO topic → FIFO queue).
• Write filter policies scoped to message attributes or the message body, using string, numeric, prefix, anything-but and exists operators, so each subscriber receives only what it wants.
• Use message attributes, per-protocol message structure, and raw message delivery correctly, and attach a subscription DLQ (redrive policy) so undeliverable messages are captured, not lost.
• Encrypt a topic with KMS, write a topic access policy, and wire cross-account publish and subscribe (including the SQS queue policy that lets a topic deliver into it).
• Decide SNS vs SQS vs EventBridge for a given integration, and run a symptom → cause → confirm → fix troubleshooting playbook for a dozen classic fan-out failures.

Prerequisites & where this fits

You need an AWS account with permission to create SNS topics, SQS queues, IAM roles and Lambda functions (a personal or dev sandbox account — never straight into production). Install and configure the AWS CLI v2 (aws configure or aws sso login), and for the IaC half, Terraform ≥ 1.5. You should be comfortable reading JSON and running shell commands; you do not need prior SNS experience. Everything in the lab sits inside the always-free SNS tier (1 million requests/month) and free-tier SQS and Lambda, so the running cost is effectively ₹0 — the only line items that could bill a few paise are CloudWatch Logs and, if you send them, SMS/email deliveries (the lab uses neither).

Where this sits: SNS is one of the three application-integration primitives you wire everything else with. It pairs constantly with SQS (the durable queue you fan out into) and sits alongside EventBridge (the content-routing event bus). This article is the SNS-focused, hands-on companion to the broader Event-Driven Architecture on AWS: EventBridge, SQS, SNS, Lambda and Step Functions, which shows how all four fit together, and to AWS Lambda Patterns: Event-Driven Functions That Scale to Zero, which goes deep on the consumer side (idempotency, DLQs, batching). Two wave siblings drill into the neighbours this article only touches: a dedicated SQS Standard vs FIFO hands-on for the queue mechanics (visibility timeout, long polling, redrive) that the fan-out relies on, and an EventBridge rules & event bus hands-on for the content-routing alternative to SNS — reach for those when you need queue or bus depth beyond what you build here.

A quick map of who owns what, so when a fan-out misbehaves you look in the right place first:

Core concepts

Five ideas make everything later obvious. Read them once; the deep sections just expand each.

A topic is a channel; publishers and subscribers never meet. You create a topic and get back a topic ARN (arn:aws:sns:ap-south-1:1234:orders-topic). Publishers call Publish against that ARN; subscribers call Subscribe against it. Neither side holds a reference to the other. That indirection is the point: the topic is the single stable contract, and both sides evolve independently behind it.

Publish is one beat; delivery is many. A single Publish call is accepted, durably stored across multiple Availability Zones, and then delivered independently to every matching subscription. Each delivery has its own protocol, its own retry state, its own filter evaluation and its own success/failure. One publish can produce a dozen deliveries; nine can succeed while three retry. You reason about the publish and each delivery as separate things.

SNS is push; SQS is pull — and they compose. SNS pushes to endpoints (it calls them). SQS is a queue that consumers poll. The most important pattern in this whole article — fan-out — is SNS pushing into several SQS queues so each downstream system can pull at its own pace, durably, with replay. SNS gives you the one-to-many broadcast; SQS gives each consumer a durable buffer. Together they are the standard way to decouple a producer from many independent, resilient consumers.

Delivery is at-least-once (Standard), so consumers must be idempotent. A Standard topic optimises for throughput and availability, which means a message can occasionally be delivered more than once and ordering is best-effort, not guaranteed. If your subscriber charges a card or inserts a row, design it to tolerate seeing the same message twice (a dedupe key, a conditional write). A FIFO topic trades throughput for strict ordering and exactly-once processing within a group — use it only when you truly need order.

Every gate is a policy, and there are three of them. Who may publish to and subscribe to a topic is the topic policy (resource-based, on the topic). Whether SNS may deliver into an SQS queue is the SQS queue policy (on the queue). Whether SNS may invoke a Lambda is the Lambda resource policy (on the function). The console wires these for you when you click “subscribe”; the CLI and Terraform do not, which is why “the subscription exists but the queue is empty” is the single most common fan-out bug.

The vocabulary in one table

Pin every moving part down before the deep dive. The glossary at the end repeats these for lookup; this is the mental model side by side:

The publish → deliver contract

Everything SNS does is one of these four beats. Knowing which beat you’re debugging tells you where to look:

The pub/sub model and topics

A topic is the unit you create, secure, and publish to. It has a type (Standard or FIFO), a policy, optional encryption, and a set of subscriptions. Nothing about a topic references its subscribers — that asymmetry is what lets you add and remove consumers freely.

Standard vs FIFO topics

The single biggest topic decision is Standard vs FIFO, and beginners over-reach for FIFO. FIFO guarantees ordering and deduplication, but at roughly 300 messages/second per topic and with hard constraints on which subscribers it can serve. Standard is virtually unlimited and is the right default for the overwhelming majority of fan-outs — you just make consumers idempotent.

The rule: use Standard and make consumers idempotent unless you can name a concrete correctness bug caused by out-of-order or duplicate processing. FIFO exists for cases like “apply account debits in order” where reordering is genuinely wrong — and even then, ordering is only preserved end-to-end if the FIFO topic delivers into FIFO SQS queues (a FIFO topic can fan out to Standard queues too, but you lose the guarantee at that hop).

FIFO ordering and deduplication knobs

If you do go FIFO, two parameters govern behaviour and beginners routinely misuse them:

Topic attributes and limits

A topic carries a handful of attributes you set once and mostly forget — but each maps to a real failure mode:

Subscriptions and delivery protocols

A subscription binds one endpoint to a topic under one protocol. The protocol decides three things a beginner must know up front: whether the subscription needs confirmation, what payload shape the endpoint receives, and how retries/DLQ behave.

The protocol matrix

Two facts save hours. First, SQS, Lambda and Firehose subscriptions auto-confirm, but email, email-json and HTTP/S do not — until confirmed they show SubscriptionArn: PendingConfirmation and receive nothing. Second, Lambda always receives the full SNS envelope (you read event['Records'][0]['Sns']['Message']); raw message delivery is not available for Lambda, only for SQS, HTTP/S and Firehose.

Confirmation behaviour

Subscription attributes you actually set

Every subscription has attributes that turn features on. These four are the ones you’ll touch constantly:

The fan-out pattern (and why SNS→SQS beats SNS→Lambda direct)

Fan-out is the reason most teams adopt SNS: one publish, delivered to many independent subscribers, each processing on its own terms. The canonical, production-grade shape is SNS → several SQS queues, with each queue owned by one consumer (a Lambda, an ECS service, an EC2 worker). This is what you build in the lab.

Why put SQS between SNS and your compute?

You can subscribe a Lambda directly to a topic — it works, and it’s tempting because it’s one fewer resource. But a direct Lambda subscription is an asynchronous push with all the fragility that implies. Fronting the Lambda with an SQS queue turns a fragile push into a durable, replayable pull. This is one of the most important design lessons in AWS integration, so here it is in full:

The thesis: SNS gives you the broadcast; SQS gives each consumer durability, buffering and replay. If a consumer is down for an hour, the SQS-fronted version keeps its messages safe and drains them when it recovers; the direct-Lambda version depends entirely on SNS retries and a DLQ, and offers no easy replay. Use direct Lambda only for genuinely loss-tolerant notifications; use SNS→SQS for anything that matters.

Fan-out variations

Message filtering with filter policies

Fan-out delivers to every subscriber by default. Filter policies let each subscription say “only send me the messages I care about,” so SNS evaluates the policy per message and delivers only matches. This is how one topic serves many consumers with different interests without a firehose of irrelevant traffic — SQS-A gets order_placed, SQS-B gets premium-tier events, and neither pays to receive the other’s.

Filter policy scope: attributes vs body

A filter policy has a scope that decides what it inspects:

Attribute-scoped filtering is the classic approach (cheap, explicit); body-scoped filtering (newer) lets you route on the payload directly — handy when the publisher is a service you don’t control and can’t make attach attributes.

Filter operators

A filter policy is a JSON object mapping attribute (or body-field) names to an array of match conditions. The value must satisfy at least one entry in the array (OR within a key), and all keys must match (AND across keys). The operators:

Worked filtering examples

Here is how the two lab queues route differently from the same topic:

A single publish with attributes event=order_placed, tier=premium, amount=2500 matches SQS-A (event), SQS-B (tier) and Lambda (no filter) — three deliveries from one publish. A publish with event=order_shipped, tier=standard matches only Lambda. That’s filtered fan-out in one sentence.

Filter policy limits and gotchas

The number-one filtering bug: a policy that is correct but too strict, so the subscriber silently receives nothing. Always test a filter with a message you expect to match, and confirm with get-subscription-attributes.

Message attributes, structure and raw delivery

Three related concepts control what bytes a subscriber actually sees.

Message attributes

Message attributes are typed key/value pairs you attach beside the body. They exist for two reasons: to drive filtering (as above) and to carry metadata some protocols read (e.g. SMS sender id). They are part of the 256 KB total size.

Message structure: per-protocol payloads

By default, every subscriber gets the same --message string. But with --message-structure json, you send a JSON object whose keys are protocols, so each subscriber type gets a tailored payload — a short SMS, a rich email, a full JSON to SQS:

Example: {"default":"Order update","sms":"Order shipped!","email":"Your order has shipped. Track: ...","sqs":"{\"orderId\":\"1001\",\"status\":\"shipped\"}"} sends a terse SMS, a friendly email, and structured JSON to the queue — from one publish.

Raw message delivery: envelope vs raw

By default SNS wraps your message in a JSON envelope (metadata: Type, MessageId, TopicArn, Message, Timestamp, Signature, MessageAttributes). An SQS consumer then has to parse .Message out of that envelope. Raw message delivery strips the envelope so the subscriber receives exactly your body:

Turn raw delivery on for SQS subscriptions unless you specifically need the SNS metadata — it removes a whole class of “why is my JSON double-wrapped?” bugs.

Delivery retries and the subscription DLQ

SNS does not deliver once and forget. When an endpoint fails (a 500 from a webhook, a throttled Lambda, an SQS queue that briefly denies access), SNS retries on a policy that depends on the endpoint type. Whatever it still can’t deliver is either dropped or — if you configured a redrive policy — sent to a dead-letter queue.

The retry phases

SNS’s delivery retry policy has four phases, most relevant to HTTP/S where you can tune them:

Retry policy by endpoint type

The redrive policy (subscription DLQ)

A redrive policy on a subscription names an SQS queue that catches messages SNS could not deliver after all retries — instead of losing them, you keep them for inspection and replay. This is the “DLQ on a subscription” the lab attaches.

The rule mirrors Lambda’s: a subscription with no redrive policy silently drops messages it can’t deliver. Attach a DLQ to every subscription whose messages matter, and remember the DLQ’s own queue policy must let the topic write to it — a DLQ you can’t deliver into is no DLQ at all.

Encryption, topic policy and cross-account

KMS encryption at rest

SNS encrypts messages in transit over HTTPS always. For encryption at rest, enable server-side encryption (SSE) with a KMS key:

The classic encryption trap: you encrypt the topic with a CMK, and now SQS delivery fails silently because the SQS service (delivering on SNS’s behalf) can’t use the key. Fix: the CMK key policy must grant kms:Decrypt and kms:GenerateDataKey to the relevant service principals (and, for cross-account, the other account). Encrypting a topic is one line; making every consumer able to decrypt is the real work.

The topic access policy

The topic policy is a resource-based policy that answers “who may Publish and Subscribe?” The default policy allows only the topic owner’s account. You edit it to let another account or an AWS service publish:

Cross-account publish and subscribe

Cross-account fan-out (platform team owns the topic; app teams own the queues) has three moving policies. Get all three and it works; miss one and you get a specific, diagnosable failure:

SNS vs SQS vs EventBridge — which integration?

These three are constantly confused. The one-liner: SQS is a queue (one-to-one, pull, buffer), SNS is a broadcaster (one-to-many, push, fan-out), EventBridge is a router (content-based routing across many event types and sources).

A useful mental model: many mature architectures use all three — EventBridge routes diverse events by content, SNS fans a chosen event out to many consumers, and each consumer sits behind its own SQS queue for durability. They compose; they don’t compete.

Architecture at a glance

The diagram below is the exact stack you build in the lab, drawn as a left-to-right flow. A publisher calls Publish on the SNS topic with a message plus message attributes (event, tier, amount). SNS evaluates each subscription’s filter policy and fans the message out: SQS-A keeps only event=order_placed, SQS-B keeps only tier=premium (both with raw message delivery so consumers get the body, not the envelope), and the Lambda subscription has no filter so it receives everything. Any delivery SNS can’t complete after its retries lands in the subscription DLQ (an SQS queue) instead of being dropped, and the whole thing is gated by the topic policy and KMS.

The six numbered badges mark the six places a fan-out fails, and the legend narrates each as symptom · confirm · fix — the same map as the troubleshooting playbook, drawn onto the architecture so you can see where each failure lives.

Real-world scenario

KloudCart, a mid-sized marketplace in Pune, had a checkout service that, on every order, synchronously called five downstream systems in a loop: inventory, billing, email, analytics and fraud. It worked until Black Friday, when the analytics endpoint slowed to 4 seconds per call. Because checkout awaited every downstream call, checkout latency ballooned to 12 seconds and the analytics timeout began throwing exceptions that rolled back otherwise-successful orders. One slow, non-critical consumer was taking down the critical path. Their senior engineer, Rohan, was tasked with decoupling it in a week.

He introduced an SNS topic, kc-order-events, and made checkout do exactly one thing after persisting the order: Publish an order_placed message with attributes event=order_placed, tier, and amount. Checkout latency dropped to the single publish call — about 8 milliseconds. Then he fanned out. Inventory, billing and analytics each got their own SQS queue subscribed to the topic; each queue was consumed by that team’s existing worker. The analytics slowdown now just meant its queue backed up harmlessly while checkout and the other consumers ran at full speed — the durable SQS buffer absorbed the lag that used to crash the order.

Filtering earned its keep immediately. The billing queue only needed paid orders, so Rohan added {"event": ["order_placed"], "payment": ["card", "upi"]} and billing stopped seeing cash-on-delivery noise. The fraud team only cared about high-value orders, so their subscription carried {"amount": [{"numeric": [">", 50000]}]} — one topic, five consumers, each receiving exactly its slice. He turned on raw message delivery for every SQS subscription so the workers read the order JSON directly instead of unwrapping the SNS envelope, which let him migrate the consumers with zero code changes to their message parsing.

Two failures during rollout mapped exactly to this article’s playbook. First, the billing queue stayed empty even though the subscription showed as confirmed — the SQS queue policy didn’t allow sns.amazonaws.com to SendMessage (the console adds it, but Rohan had used Terraform without an aws_sqs_queue_policy). Second, during a deploy the email consumer’s webhook was down for 20 minutes; without a subscription DLQ those notifications would have been dropped after retries. Rohan had attached a redrive policy to a kc-sns-dlq queue, so the 340 undelivered messages sat safely in the DLQ and were re-driven once the webhook recovered — zero lost emails. Post-migration: checkout p99 fell from 12 s to under 50 ms, a single slow consumer could no longer affect the order, and adding a new “loyalty points” consumer next quarter was a five-minute self-service subscribe with no checkout change. Rohan’s wiki note became team doctrine: “Publish once, fan out to a queue per consumer, filter at the subscription, and put a DLQ on everything that matters.”

Advantages and disadvantages

When each side matters: SNS dominates for one-to-many broadcast and fan-out — notifications, event distribution, “when X, do A and B and C.” The disadvantages bite when you need durable per-consumer backlogs (add SQS), content routing across many event types with replay/schemas (use EventBridge), or strict ordering at high volume (FIFO’s throughput ceiling fights you). The honest rule: reach for SNS whenever one event must reach many consumers quickly; put SQS behind each consumer that needs durability; and switch to EventBridge when routing logic, source variety, or archive/replay become the point.

Hands-on lab

You will build the diagram: a Standard topic that fans out to two SQS queues with different filter policies (raw delivery on) and a Lambda (no filter), with a subscription DLQ on the Lambda subscription. First with the aws CLI, then the identical stack as Terraform, then a clean teardown. Everything is free-tier. Pick a region and stick to it (this lab uses ap-south-1, Mumbai).

⚠️ Cost note: SNS publishes/deliveries to SQS and Lambda are free at this volume (1M requests/mo free tier). We deliberately avoid SMS and email (those do cost). The only paise-level items are CloudWatch Logs; we delete everything at the end.

What you’ll create

Part A — the CLI path

Step 1 — Create the topic. A Standard topic; capture its ARN:

TOPIC_ARN=$(aws sns create-topic \
  --name kv-orders-topic \
  --region ap-south-1 \
  --query TopicArn --output text)
echo "$TOPIC_ARN"
# arn:aws:sns:ap-south-1:111122223333:kv-orders-topic

Step 2 — Create the three SQS queues and grab their ARNs and URLs.

for q in kv-sqs-a kv-sqs-b kv-sns-dlq; do
  aws sqs create-queue --queue-name "$q" --region ap-south-1 >/dev/null
done

get_url() { aws sqs get-queue-url --queue-name "$1" --region ap-south-1 --query QueueUrl --output text; }
get_arn() { aws sqs get-queue-attributes --queue-url "$1" --attribute-names QueueArn --region ap-south-1 --query 'Attributes.QueueArn' --output text; }

A_URL=$(get_url kv-sqs-a);   A_ARN=$(get_arn "$A_URL")
B_URL=$(get_url kv-sqs-b);   B_ARN=$(get_arn "$B_URL")
DLQ_URL=$(get_url kv-sns-dlq); DLQ_ARN=$(get_arn "$DLQ_URL")

Step 3 — Let the topic deliver into each queue (the step the console hides). Each queue’s policy must allow sns.amazonaws.com to SendMessage, scoped to this topic. Do it for all three (including the DLQ, which SNS also writes to):

set_queue_policy() {
  local url="$1" arn="$2"
  local policy=$(cat <<JSON
{"Version":"2012-10-17","Statement":[{
  "Sid":"AllowSNS","Effect":"Allow",
  "Principal":{"Service":"sns.amazonaws.com"},
  "Action":"sqs:SendMessage","Resource":"$arn",
  "Condition":{"ArnEquals":{"aws:SourceArn":"$TOPIC_ARN"}}
}]}
JSON
)
  aws sqs set-queue-attributes --queue-url "$url" \
    --attributes "{\"Policy\":$(jq -Rs . <<<"$policy")}" \
    --region ap-south-1
}
set_queue_policy "$A_URL" "$A_ARN"
set_queue_policy "$B_URL" "$B_ARN"
set_queue_policy "$DLQ_URL" "$DLQ_ARN"

Step 4 — Subscribe SQS-A with a filter policy + raw delivery. SQS subscriptions auto-confirm, so this is live immediately:

SUB_A=$(aws sns subscribe \
  --topic-arn "$TOPIC_ARN" \
  --protocol sqs \
  --notification-endpoint "$A_ARN" \
  --attributes '{"RawMessageDelivery":"true","FilterPolicy":"{\"event\":[\"order_placed\"]}"}' \
  --region ap-south-1 \
  --query SubscriptionArn --output text)

Step 5 — Subscribe SQS-B with a different filter policy.

SUB_B=$(aws sns subscribe \
  --topic-arn "$TOPIC_ARN" \
  --protocol sqs \
  --notification-endpoint "$B_ARN" \
  --attributes '{"RawMessageDelivery":"true","FilterPolicy":"{\"tier\":[\"premium\"]}"}' \
  --region ap-south-1 \
  --query SubscriptionArn --output text)

Step 6 — Create the Lambda and let SNS invoke it. A trivial handler that logs the event, an execution role for logs, then the resource-based permission SNS needs:

cat > notify.py <<'PY'
import json
def handler(event, context):
    for rec in event["Records"]:
        msg = rec["Sns"]["Message"]          # Lambda always gets the SNS envelope
        print(f"notify-fn received: {msg}")
    return {"ok": True}
PY
zip notify.zip notify.py

cat > trust.json <<'JSON'
{"Version":"2012-10-17","Statement":[{"Effect":"Allow",
 "Principal":{"Service":"lambda.amazonaws.com"},"Action":"sts:AssumeRole"}]}
JSON
aws iam create-role --role-name kv-notify-role \
  --assume-role-policy-document file://trust.json >/dev/null
aws iam attach-role-policy --role-name kv-notify-role \
  --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
sleep 10   # let the role propagate

ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
aws lambda create-function --function-name kv-notify-fn \
  --runtime python3.12 --architectures arm64 \
  --handler notify.handler --zip-file fileb://notify.zip \
  --role arn:aws:iam::${ACCOUNT_ID}:role/kv-notify-role \
  --region ap-south-1 >/dev/null

# The resource-based policy that lets SNS invoke the function:
aws lambda add-permission --function-name kv-notify-fn \
  --statement-id sns-invoke --action lambda:InvokeFunction \
  --principal sns.amazonaws.com --source-arn "$TOPIC_ARN" \
  --region ap-south-1

Step 7 — Subscribe the Lambda (no filter → receives everything) and attach a DLQ.

FN_ARN=arn:aws:lambda:ap-south-1:${ACCOUNT_ID}:function:kv-notify-fn
SUB_L=$(aws sns subscribe \
  --topic-arn "$TOPIC_ARN" --protocol lambda \
  --notification-endpoint "$FN_ARN" \
  --region ap-south-1 --query SubscriptionArn --output text)

# Subscription DLQ: undeliverable messages go to kv-sns-dlq instead of being dropped
aws sns set-subscription-attributes \
  --subscription-arn "$SUB_L" \
  --attribute-name RedrivePolicy \
  --attribute-value "{\"deadLetterTargetArn\":\"$DLQ_ARN\"}" \
  --region ap-south-1

Step 8 — Publish and watch filtering route the same topic three ways. First a message that matches both queues and the Lambda:

aws sns publish --topic-arn "$TOPIC_ARN" \
  --message '{"orderId":"1001","amount":2500}' \
  --message-attributes '{"event":{"DataType":"String","StringValue":"order_placed"},"tier":{"DataType":"String","StringValue":"premium"}}' \
  --region ap-south-1

Now one that matches neither queue (only the unfiltered Lambda):

aws sns publish --topic-arn "$TOPIC_ARN" \
  --message '{"orderId":"1002","status":"shipped"}' \
  --message-attributes '{"event":{"DataType":"String","StringValue":"order_shipped"},"tier":{"DataType":"String","StringValue":"standard"}}' \
  --region ap-south-1

Step 9 — Verify the routing. Read each queue: SQS-A should have only order 1001; SQS-B should have only order 1001; the Lambda logs should show both 1001 and 1002.

aws sqs receive-message --queue-url "$A_URL" --max-number-of-messages 10 --region ap-south-1 \
  --query 'Messages[].Body' --output text
# {"orderId":"1001","amount":2500}      <-- raw body, no SNS envelope

aws sqs receive-message --queue-url "$B_URL" --max-number-of-messages 10 --region ap-south-1 \
  --query 'Messages[].Body' --output text
# {"orderId":"1001","amount":2500}      <-- 1002 was filtered out (tier=standard)

aws logs tail /aws/lambda/kv-notify-fn --region ap-south-1 --since 5m
# notify-fn received: {"orderId":"1001","amount":2500}
# notify-fn received: {"orderId":"1002","status":"shipped"}   <-- Lambda has no filter

That contrast — SQS-A/B each hold only their matching message while the Lambda holds both — is filtered fan-out. Notice the SQS bodies are your raw JSON (raw delivery on), while the Lambda received the full envelope (rec["Sns"]["Message"]).

Step 10 — Tear down. Delete in dependency order:

aws sns unsubscribe --subscription-arn "$SUB_A" --region ap-south-1
aws sns unsubscribe --subscription-arn "$SUB_B" --region ap-south-1
aws sns unsubscribe --subscription-arn "$SUB_L" --region ap-south-1
aws sns delete-topic --topic-arn "$TOPIC_ARN" --region ap-south-1
for u in "$A_URL" "$B_URL" "$DLQ_URL"; do aws sqs delete-queue --queue-url "$u" --region ap-south-1; done
aws lambda delete-function --function-name kv-notify-fn --region ap-south-1
aws logs delete-log-group --log-group-name /aws/lambda/kv-notify-fn --region ap-south-1
aws iam detach-role-policy --role-name kv-notify-role \
  --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
aws iam delete-role --role-name kv-notify-role

Part B — the same stack as Terraform

The CLI teaches the moving parts; Terraform is how you keep them. Note the three things people forget in IaC that the console does automatically: the SQS queue policy (aws_sqs_queue_policy), the Lambda permission (aws_lambda_permission), and the filter policy / raw delivery / redrive on the subscription.

terraform {
  required_providers {
    aws     = { source = "hashicorp/aws", version = "~> 5.0" }
    archive = { source = "hashicorp/archive", version = "~> 2.4" }
  }
}
provider "aws" { region = "ap-south-1" }

data "aws_caller_identity" "me" {}

# --- Topic ---
resource "aws_sns_topic" "orders" {
  name = "kv-orders-topic"
}

# --- Queues (A, B, DLQ) ---
resource "aws_sqs_queue" "a"   { name = "kv-sqs-a" }
resource "aws_sqs_queue" "b"   { name = "kv-sqs-b" }
resource "aws_sqs_queue" "dlq" { name = "kv-sns-dlq" }

# --- Queue policies: allow THIS topic to SendMessage ---
data "aws_iam_policy_document" "allow_sns" {
  for_each = { a = aws_sqs_queue.a.arn, b = aws_sqs_queue.b.arn, dlq = aws_sqs_queue.dlq.arn }
  statement {
    effect    = "Allow"
    actions   = ["sqs:SendMessage"]
    resources = [each.value]
    principals { type = "Service", identifiers = ["sns.amazonaws.com"] }
    condition {
      test     = "ArnEquals"
      variable = "aws:SourceArn"
      values   = [aws_sns_topic.orders.arn]
    }
  }
}
resource "aws_sqs_queue_policy" "a"   { queue_url = aws_sqs_queue.a.id,   policy = data.aws_iam_policy_document.allow_sns["a"].json }
resource "aws_sqs_queue_policy" "b"   { queue_url = aws_sqs_queue.b.id,   policy = data.aws_iam_policy_document.allow_sns["b"].json }
resource "aws_sqs_queue_policy" "dlq" { queue_url = aws_sqs_queue.dlq.id, policy = data.aws_iam_policy_document.allow_sns["dlq"].json }

# --- SQS subscriptions with distinct filter policies + raw delivery ---
resource "aws_sns_topic_subscription" "a" {
  topic_arn            = aws_sns_topic.orders.arn
  protocol             = "sqs"
  endpoint             = aws_sqs_queue.a.arn
  raw_message_delivery = true
  filter_policy        = jsonencode({ event = ["order_placed"] })
  depends_on           = [aws_sqs_queue_policy.a]
}
resource "aws_sns_topic_subscription" "b" {
  topic_arn            = aws_sns_topic.orders.arn
  protocol             = "sqs"
  endpoint             = aws_sqs_queue.b.arn
  raw_message_delivery = true
  filter_policy        = jsonencode({ tier = ["premium"] })
  depends_on           = [aws_sqs_queue_policy.b]
}

# --- Lambda subscriber (no filter) + its permission + a subscription DLQ ---
data "archive_file" "fn" {
  type        = "zip"
  source_file = "${path.module}/notify.py"
  output_path = "${path.module}/notify.zip"
}
resource "aws_iam_role" "fn" {
  name = "kv-notify-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17",
    Statement = [{ Effect = "Allow", Principal = { Service = "lambda.amazonaws.com" }, Action = "sts:AssumeRole" }]
  })
}
resource "aws_iam_role_policy_attachment" "logs" {
  role       = aws_iam_role.fn.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
}
resource "aws_lambda_function" "fn" {
  function_name    = "kv-notify-fn"
  role             = aws_iam_role.fn.arn
  runtime          = "python3.12"
  architectures    = ["arm64"]
  handler          = "notify.handler"
  filename         = data.archive_file.fn.output_path
  source_code_hash = data.archive_file.fn.output_base64sha256
}
resource "aws_lambda_permission" "sns" {
  statement_id  = "sns-invoke"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.fn.function_name
  principal     = "sns.amazonaws.com"
  source_arn    = aws_sns_topic.orders.arn
}
resource "aws_sns_topic_subscription" "lambda" {
  topic_arn  = aws_sns_topic.orders.arn
  protocol   = "lambda"
  endpoint   = aws_lambda_function.fn.arn
  redrive_policy = jsonencode({ deadLetterTargetArn = aws_sqs_queue.dlq.arn })
  depends_on = [aws_lambda_permission.sns]
}

terraform init
terraform apply        # review the plan, type yes
# publish a test message the same way as Step 8, then:
terraform destroy      # one-command teardown

Common mistakes & troubleshooting

This is the section you’ll return to. Match your symptom, run the confirm command, apply the fix.

SNS / delivery error reference

Delivery status: how to actually see what SNS did

SNS delivery is invisible until you turn on delivery status logging — an IAM role plus a sample rate that writes success/failure to CloudWatch Logs. It is the single best tool for “did SNS even try, and what did the endpoint say?”

The three nastiest, explained

“Subscription confirmed but the queue is empty” (row 5) wastes the most hours because the subscription genuinely exists and looks healthy — so you debug SNS when the problem is the SQS queue policy. SNS delivers to SQS by calling sqs:SendMessage as the SNS service principal; if the queue’s resource policy doesn’t allow sns.amazonaws.com (scoped to your topic ARN), every delivery is denied and the queue stays empty. The console adds this policy when you subscribe; the CLI and Terraform do not. Always attach the queue policy, and use aws sqs get-queue-attributes --attribute-names Policy to confirm the SNS statement is there.

“The subscriber receives nothing” (rows 1–2) is almost always a filter policy that is correct-but-too-strict, or scoped to the wrong thing. Remember the semantics: AND across keys, OR within a key, and a missing attribute fails its key (unless you used exists:false). If you filter on MessageAttributes but publish only a body (or vice-versa via FilterPolicyScope), nothing matches. Debug by publishing a message you are certain should match, then dump get-subscription-attributes and compare field-by-field. A numeric filter silently failing is usually a String attribute where a Number was required.

“Messages disappear under an outage” (rows 8–9) catches teams who added fan-out but not resilience. SNS retries, but a Standard topic does not persist messages after retries exhaust — without a redrive policy the message is gone. And the DLQ itself is an SQS queue that SNS must be allowed to write to, so a DLQ whose queue policy doesn’t grant the topic SendMessage is a DLQ that also silently drops. Attach a redrive policy to every subscription that matters, give the DLQ the same SNS-allow policy as any other queue, and you turn “lost forever” into “sitting in the DLQ, ready to replay.”

Best practices

• Fan out into an SQS queue per consumer. Give each independent consumer its own queue subscribed to the topic. You get durability, backpressure, per-consumer replay and batching for free — the single most important SNS design rule.
• Prefer SNS→SQS over SNS→Lambda direct. Reserve direct Lambda subscriptions for genuinely loss-tolerant, low-volume notifications; put SQS in front of anything that must not be lost.
• Filter at the subscription, not in the consumer. A filter policy drops non-matches before delivery, so you don’t pay to deliver, store, poll and discard irrelevant messages. Move routing logic to the subscription.
• Turn on raw message delivery for SQS/HTTP subscribers. Unless you need the SNS signature/metadata, raw delivery gives consumers your payload directly and removes envelope-parsing bugs.
• Attach a DLQ (redrive policy) to every subscription that matters — and remember the DLQ’s own queue policy must let the topic write to it. A DLQ you can’t deliver into isn’t a DLQ.
• Make Standard-topic consumers idempotent. At-least-once means duplicates happen. Use a dedupe key or a conditional write; don’t reach for FIFO just to avoid designing for it.
• Use FIFO only when ordering is a correctness requirement — and pair a FIFO topic with FIFO queues to keep the guarantee end-to-end. Expect ~300 msg/s and pick a MessageGroupId that reflects your real ordering scope.
• Encrypt with KMS, and fix the key policy for every consumer. Enabling SSE is one line; letting SQS/Lambda/cross-account principals decrypt is the actual work — do it at the same time.
• Scope the topic policy tightly. Grant Publish/Subscribe to specific principals with aws:SourceArn/aws:SourceAccount conditions; never leave a topic open to *.
• Deploy with IaC, not the console. The console silently adds the queue policy and Lambda permission; IaC does not. Encode them explicitly so your stack is reproducible and reviewable.
• Turn on delivery status logging in production. The success/failure feedback roles are the only way to see what endpoints actually returned — invaluable when a subscriber “isn’t getting messages.”

Security notes

SNS’s security surface is small but every piece gates delivery. Get these right from the start:

The two you’ll get wrong first: encrypting a topic with a CMK and forgetting to let SQS decrypt (silent delivery failure), and leaving a topic policy that allows Subscribe too broadly (someone subscribes their own endpoint to your event stream). Lock the policy to named principals with conditions.

Cost & sizing

SNS pricing is refreshingly simple: you pay per request (a publish or a delivery) plus per-delivery fees that depend on the protocol, and the free tier covers most beginner and small-production usage. SQS/Lambda deliveries are the cheap, high-value path.

A worked example to make it concrete (rough INR at ~₹84/USD):

The takeaways: SQS and Lambda deliveries are free, which is exactly why the SNS→SQS fan-out is both the most robust and the cheapest pattern; email and SMS are the expensive protocols (price them before you fan out to them at volume); and 1M requests/month free means small apps run at ₹0. To right-size, batch publishes where you can, filter at the subscription so you don’t pay to deliver noise, and keep consumers in-region to avoid egress.

Interview & exam questions

1. What is Amazon SNS in one sentence, and how does it differ from SQS? SNS is a managed pub/sub service: publishers send a message to a topic and SNS pushes a copy to every subscriber (one-to-many, push). SQS is a queue: one producer, one consumer group, and the consumer pulls (one-to-one, pull, durable buffer). SNS broadcasts; SQS buffers. (CLF-C02, SAA-C03)

2. Explain the fan-out pattern and why you’d put SQS between SNS and a consumer. Fan-out is one publish delivered to many subscribers. Fronting each consumer with its own SQS queue adds a durable buffer, backpressure, easy replay, and per-record retry/redrive — so a slow or down consumer doesn’t lose messages or affect others. A direct Lambda subscription has none of that beyond SNS retries and a DLQ. (SAA-C03, DVA-C02)

3. When do you need a FIFO topic, and what’s the cost? Only when strict ordering and exactly-once processing within a group are correctness requirements (e.g. financial ledgers). The cost is throughput — about 300 messages/second — and the constraint that end-to-end ordering requires FIFO SQS queues as subscribers. Otherwise use Standard and make consumers idempotent. (SAA-C03, DVA-C02)

4. How does message filtering work, and what are the two scopes? Each subscription can carry a filter policy that SNS evaluates per message, delivering only matches. The scope is either MessageAttributes (filter on typed attributes attached at publish) or MessageBody (filter on JSON fields in the payload). Semantics are AND across keys, OR within a key. (DVA-C02)

5. Your SQS queue is subscribed and confirmed but receives nothing. First hypothesis? The SQS queue’s resource policy doesn’t allow sns.amazonaws.com to sqs:SendMessage for the topic ARN. The console adds this automatically; CLI/Terraform don’t. Confirm with sqs get-queue-attributes --attribute-names Policy. (DVA-C02, SCS)

6. What is raw message delivery and when do you enable it? By default SNS wraps the body in a JSON envelope (metadata + Message). Raw message delivery (SQS/HTTP/Firehose only) strips the envelope so the subscriber receives your body verbatim, with attributes mapped to native SQS attributes. Enable it unless you need the SNS signature/metadata. (DVA-C02)

7. How do you stop SNS from losing messages when an endpoint is down? Attach a redrive policy (subscription DLQ) pointing to an SQS queue; after retries exhaust, undeliverable messages go there instead of being dropped, ready to inspect and replay. Ensure the DLQ’s queue policy allows the topic to write to it. (DVA-C02, SAA-C03)

8. Describe delivery/retry differences between endpoint types. For AWS-managed endpoints (SQS, Lambda, Firehose), SNS uses an aggressive managed multi-phase retry policy. For HTTP/S you can customize the delivery policy (immediate, pre-backoff, backoff, post-backoff; default ~3 retries at 20s). Email/SMS/push retries are managed and have no DLQ. (DVA-C02)

9. How do you publish cross-account, and what three policies are involved? The topic policy must allow the publisher’s account SNS:Publish (and a subscriber account SNS:Subscribe); the subscriber’s SQS queue policy must allow sns.amazonaws.com to SendMessage; and if the topic uses a CMK, the key policy must allow the relevant principals to decrypt. (SCS, SAA-C03)

10. SNS vs EventBridge — when each? SNS for high-throughput, low-latency, one-to-many fan-out with simple attribute/body filtering. EventBridge for content-based routing across many event types and 200+ AWS/SaaS sources, with a schema registry and built-in archive/replay. Many designs use EventBridge to route and SNS to fan out. (SAA-C03, DVA-C02)

11. Why must Standard-topic consumers be idempotent? Standard topics deliver at-least-once, so a consumer can occasionally see the same message twice; ordering is best-effort too. If processing has side effects (charging, inserting), design for duplicates with a dedupe key or conditional write. (DVA-C02)

12. What’s the maximum SNS message size and how do you exceed it? 256 KB (body + attributes). For larger payloads, use the SNS Extended Client Library, which stores the payload in S3 and sends a reference through SNS (up to 2 GB). (DVA-C02)

Quick check

1. A publisher sends one message with attributes event=order_placed, tier=standard. SQS-A filters {"event":["order_placed"]}, SQS-B filters {"tier":["premium"]}, Lambda has no filter. Which endpoints receive it?
2. You subscribed an SQS queue via the CLI; it’s confirmed but empty. What single resource is almost certainly misconfigured?
3. Name two concrete advantages of SNS→SQS→Lambda over subscribing the Lambda directly.
4. Your consumer is receiving {"Type":"Notification","Message":"..."} and choking on it. What subscription setting fixes this?
5. You must guarantee ordered, exactly-once processing of account transactions. What topic type and which subscriber queue type do you use?

Answers

1. SQS-A (event matches) and Lambda (no filter). SQS-B does not — tier=standard fails its tier=premium filter.
2. The SQS queue’s resource policy — it must allow sns.amazonaws.com to sqs:SendMessage scoped to the topic ARN. The console adds this; the CLI/Terraform don’t.
3. Any two of: a durable buffer (up to 14 days) so a down consumer loses nothing; backpressure so spikes queue rather than overwhelm; easy replay by re-driving the DLQ; batching and partial-failure handling on the poll.
4. Turn on raw message delivery (RawMessageDelivery=true) so the consumer gets the body directly instead of the SNS envelope (or parse .Message out of the envelope).
5. A FIFO topic delivering to FIFO SQS queues, with a MessageGroupId that scopes ordering (e.g. per account) and deduplication enabled.

Glossary

Next steps

• See how SNS fits the bigger picture. Wire topics, queues, buses and functions together in Event-Driven Architecture on AWS: EventBridge, SQS, SNS, Lambda and Step Functions.
• Go deep on the consumer side. Learn idempotency, DLQs, batching and scale-to-zero patterns in AWS Lambda Patterns: Event-Driven Functions That Scale to Zero.
• Master the queue you fan out into. Follow the wave’s SQS Standard vs FIFO hands-on for visibility timeout, long polling and redrive mechanics that the fan-out depends on.
• Compare the routing alternative. Work through the wave’s EventBridge rules & event bus hands-on to see when content-based routing beats simple pub/sub filtering.
• Harden it for production. Add KMS encryption, tighten the topic policy, and turn on delivery status logging before you put a fan-out on your critical path.