Containerization Lesson 25 of 113

Configure Dapr on Kubernetes for Service Invocation, State, and Pub/Sub Building Blocks

Jun 10, 2026 31 min read Advanced Vinod

In a nutshell

Dapr (Distributed Application Runtime) is a helper process that runs next to every one of your services and hands each of them the same set of ready-made superpowers — call another service, save and read data, publish and receive events — through plain HTTP calls to localhost. Your code never imports a Redis client, a Kafka client, or a service-discovery library; it just talks to the little helper beside it, and the helper deals with the messy, vendor-specific plumbing. These ready-made powers are called building blocks.

The mental model that makes it click: think of Dapr as a universal travel adapter that clips onto each service. Your service speaks one simple “language” (HTTP to http://localhost:3500/...) into the adapter, and the adapter speaks whatever the outside world needs — the Kafka wire protocol, the Redis protocol, mutual-TLS handshakes, DNS lookups. Swap Redis for Postgres, or Kafka for RabbitMQ, and your application code does not change one line — you edit a small YAML file that reconfigures the adapter. That is the whole point: portable building blocks with no SDK lock-in. The same checkout binary runs unchanged on Azure with Redis + Confluent and on a laptop with in-memory stores.

This “helper beside your app” is a sidecar — a second container that Kubernetes runs inside the same Pod as your app, so the two share a network and reach each other over localhost. You do not write the sidecar; Dapr injects it for you the moment you add one annotation to your Deployment. The rest of this lesson stands up the Dapr control plane on a Kubernetes cluster and wires the three most-used building blocks the way you would run them in production: mTLS service invocation, a Redis state store, and Kafka pub/sub.

Level: Advanced (with a beginner on-ramp) · Time: ~31 min · You’ll wire: service invocation, state, and pub/sub on a real cluster

A logistics company is breaking a monolithic order-management system into a dozen Go and .NET microservices, and the platform team keeps re-solving the same three problems in every service: how does the checkout service call the inventory service securely and find it without hardcoding a DNS name; where does a long-running saga stash its state so a pod restart does not lose an in-flight order; and how does a shipment-created event reach the four services that care about it without each one growing a bespoke Kafka client. Rewriting that plumbing per service — per language — is where the velocity went. Dapr (Distributed Application Runtime) solves exactly this: a sidecar that exposes service invocation, state management, and pub/sub as plain HTTP/gRPC APIs, so a service calls http://localhost:3500/v1.0/invoke/inventory/method/reserve and the sidecar handles discovery, mTLS, retries, and the broker wire protocol. This guide stands up the Dapr control plane on Kubernetes and wires all three building blocks — mTLS service invocation, a Redis state store, and Kafka pub/sub — the way you would run them in production.

Prerequisites:

After this lesson you can:

The building blocks, in plain terms

Before the mechanics, hold the three big building blocks in your head as “a job your app wants done” on the left and “the boring stuff the sidecar does for you” on the right. In every case the app makes one HTTP call to its own sidecar on port 3500, and the sidecar owns the rest.

Building block What your app says (to localhost:3500) What the sidecar quietly handles The API you call
Service invocation “Call inventory’s reserve method” Name resolution, mTLS, load-balancing, retries, tracing POST /v1.0/invoke/inventory/method/reserve
State management “Save / read this order” Talking to Redis/Postgres/Cosmos; key namespacing, ETags, TTL, transactions POST /v1.0/state/statestore
Pub/sub “Announce shipment-created” Broker wire protocol, CloudEvents envelope, at-least-once delivery, dead-letter POST /v1.0/publish/orderpubsub/shipment-created

The second thing to internalise: on Kubernetes, everything you configure for Dapr is a Kubernetes resource — a Custom Resource, to be precise — that the Dapr operator reconciles and hands to the sidecars. You never SSH anywhere or edit a config file inside a pod. There are four kinds you will meet (they are Custom Resources managed by an operator, the same controller pattern Kubernetes uses everywhere):

Kubernetes resource Kind (apiVersion) What it declares
Component Component (dapr.io/v1alpha1) One backing resource: a state store, a pub/sub broker, a secret store, or a binding
Subscription Subscription (dapr.io/v2alpha1) Which topic on which broker routes to which app HTTP route
Configuration Configuration (dapr.io/v1alpha1) Cross-cutting sidecar settings: mTLS, tracing sampling, access-control policy
Resiliency Resiliency (dapr.io/v1alpha1) Timeouts, retries, and circuit breakers per target app/component/actor

Keep those two tables in view and the rest of the lesson is just filling them in.

Target topology

The picture below shows the two halves of a Dapr install. Read it as: a small set of shared “brain” pods (the control plane) that you install once, and your own app pods (the data plane), each of which quietly grows a sidecar.

Configure Dapr on Kubernetes for Service Invocation, State, and Pub/Sub Building Blocks — topology

The picture has two planes. The control plane is a set of Dapr system pods in the dapr-system namespace: dapr-operator (watches Component and Configuration CRDs and reconciles them to sidecars), dapr-sidecar-injector (a mutating webhook that injects the daprd sidecar into any pod annotated dapr.io/enabled: "true"), dapr-sentry (the CA that issues and rotates the X.509 SVIDs used for mTLS between sidecars), and dapr-placement (the actor placement service — present even if you do not use actors today). The data plane is your application pods in the apps namespace, each running your container plus an injected daprd sidecar. Service-to-service calls go pod → its own sidecar → (mTLS) → target sidecar → target app; state and pub/sub calls go pod → its own sidecar → Redis or Kafka. Everything you configure — the Redis state store, the Kafka pub/sub broker, the mTLS policy — is declared as Kubernetes resources that the operator hands to the sidecars at runtime.

1. Install the Dapr control plane

Install with Helm so the deployment is declarative and lands cleanly in GitOps. Add the repo, create the namespace, and install with high availability (three replicas of each control-plane service) and mTLS enabled from the start.

helm repo add dapr https://dapr.github.io/helm-charts/
helm repo update

helm upgrade --install dapr dapr/dapr \
  --version 1.14.4 \
  --namespace dapr-system \
  --create-namespace \
  --set global.ha.enabled=true \
  --set global.mtls.enabled=true \
  --set global.mtls.workloadCertTTL=24h \
  --set dapr_sentry.logLevel=info \
  --wait

helm upgrade --install is idempotent — running it again reconciles the release rather than erroring, which is exactly what you want under GitOps. Confirm the control plane is healthy. The CLI reports each service and the mTLS root-cert expiry:

dapr status -k

  NAME                   NAMESPACE    HEALTHY  STATUS   REPLICAS  VERSION  AGE
  dapr-operator          dapr-system  True     Running  3         1.14.4   2m
  dapr-sentry            dapr-system  True     Running  3         1.14.4   2m
  dapr-sidecar-injector  dapr-system  True     Running  3         1.14.4   2m
  dapr-placement-server  dapr-system  True     Running  3         1.14.4   2m

(Output representative.) Recent Dapr versions also run a dapr-scheduler-server in dapr-system — the Scheduler service that backs the Jobs API, actor reminders, and durable workflow scheduling. You do not touch it directly for the three building blocks in this lesson, but it is part of a modern control plane and appears in dapr status -k on a fresh 1.14+ install.

For production, do not let Sentry self-generate its CA. Issue the trust bundle from HashiCorp Vault’s PKI secrets engine and provide it to the chart (dapr_sentry.tls.issuer*), so the mTLS root chains to your corporate PKI and rotates on Vault’s schedule rather than a Helm value. The 24-hour workloadCertTTL means leaf SVIDs roll automatically and a leaked cert is short-lived.

2. Enable application-level mTLS service invocation

Create the application namespace and a Dapr Configuration that turns on tracing and pins the mTLS settings the sidecars enforce.

kubectl create namespace apps

# dapr-config.yaml
apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
  name: appconfig
  namespace: apps
spec:
  mtls:
    enabled: true
    workloadCertTTL: 24h
    allowedClockSkew: 15m
  tracing:
    samplingRate: "0.1"          # 10% sampled to keep cost sane
    otel:
      endpointAddress: "otel-collector.observability:4317"
      isSecure: true
      protocol: grpc

What the annotation actually triggers. When a pod carrying dapr.io/enabled: "true" is admitted to the API server, the dapr-sidecar-injector mutating webhook rewrites the pod spec on the fly — it appends a second container (daprd), mounts the SPIFFE trust bundle, and sets a few env vars. Your Deployment YAML on disk stays a single container; the running pod has two. Because this happens at admission time, only pods created after the injector is healthy get a sidecar — an existing pod must be restarted to gain one.

Annotate each workload so the injector adds a sidecar and binds this config. Here is the inventory service (the callee) and the checkout service (the caller):

# inventory-deploy.yaml (snippet)
spec:
  template:
    metadata:
      annotations:
        dapr.io/enabled: "true"
        dapr.io/app-id: "inventory"
        dapr.io/app-port: "8080"
        dapr.io/config: "appconfig"
        dapr.io/log-as-json: "true"

The dapr.io/app-id is the stable logical name every other service uses to reach this one — think of it as a phone number that never changes even when pods come and go. Once both are deployed, the checkout service invokes inventory purely by app-id — no service URL, no client-side TLS code. The call is name-resolved by Dapr (mDNS locally, Kubernetes DNS in-cluster) and encrypted sidecar-to-sidecar:

# from inside the checkout pod, calling its own sidecar on 3500
curl -s -X POST http://localhost:3500/v1.0/invoke/inventory/method/reserve \
  -H "Content-Type: application/json" \
  -d '{"sku":"SKU-4471","qty":2}'

Read the URL as three parts: /v1.0/invoke/ (the building block) + inventory (the target app-id) + /method/reserve (the path on the target app). The caller never knows the callee’s IP, port, or namespace DNS — only its app-id.

To restrict who may call what — so only checkout can hit inventory/reserve — add an access-control policy to the callee’s Configuration. This is Dapr’s app-level authorization, evaluated against the verified SPIFFE identity in the peer’s certificate:

  accessControl:
    defaultAction: deny
    trustDomain: "public"
    policies:
      - appId: checkout
        defaultAction: deny
        trustDomain: "public"
        namespace: "apps"
        operations:
          - name: /reserve
            httpVerb: ["POST"]
            action: allow

Two things are happening here, and it is worth naming them separately: mTLS gives you authentication (the peer really is checkout, proven by its certificate), and the access-control policy gives you authorization (this authenticated peer is allowed to call this method). defaultAction: deny means anything not explicitly allowed is refused — the safe default.

3. Wire the Redis state store

Pull the Redis password from Vault rather than a literal. Install the Dapr Vault secret-store component (so other components can reference Vault), then declare the state store referencing that secret. (For how the pod’s service account authenticates to Vault in the first place, see Configure Vault OIDC/JWT & Kubernetes auth for workload secrets.)

# secretstore-vault.yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: vault
  namespace: apps
spec:
  type: secretstores.hashicorp.vault
  version: v1
  metadata:
    - name: vaultAddr
      value: "https://vault.security.internal:8200"
    - name: enginePath
      value: "secret"
    - name: vaultKubernetesMountPath
      value: "kubernetes"          # Vault Kubernetes auth role bound to the pod SA

# statestore-redis.yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: statestore
  namespace: apps
spec:
  type: state.redis
  version: v1
  metadata:
    - name: redisHost
      value: "orders-redis.redis.cache.windows.net:6380"
    - name: enableTLS
      value: "true"
    - name: redisPassword
      secretKeyRef:
        name: orders-redis-password   # key path inside Vault
        key: password
    - name: actorStateStore
      value: "true"                    # also serve actor state if needed
auth:
  secretStore: vault

Notice the shape of a Component: a type (here state.redis, one of ~100 pluggable component types) plus a metadata list of key/value settings. Point type at state.postgresql or state.azure.cosmosdb instead and the app-facing API stays identical — that is the portability payoff. Apply and verify the operator accepted both components:

kubectl apply -f secretstore-vault.yaml -f statestore-redis.yaml
dapr components -k -n apps

The checkout saga now persists and reads state through its sidecar — no Redis client in the app, and key is automatically namespaced as <app-id>||<key> so services cannot collide:

# save state
curl -s -X POST http://localhost:3500/v1.0/state/statestore \
  -H "Content-Type: application/json" \
  -d '[{"key":"order-9912","value":{"status":"reserving","items":2}}]'

# read it back
curl -s http://localhost:3500/v1.0/state/statestore/order-9912

That <app-id>||<key> prefix is invisible to your code — you write and read order-9912 — but in Redis the actual key is checkout||order-9912, which is why two different services can both use the key order-9912 without ever overwriting each other. Use Dapr’s ETag-based optimistic concurrency and explicit consistency for saga steps — pass concurrency=first-write and consistency=strong so two pods racing on the same order do not clobber each other:

curl -s -X POST "http://localhost:3500/v1.0/state/statestore" \
  -d '[{"key":"order-9912","value":{"status":"shipped"},"etag":"3","options":{"concurrency":"first-write","consistency":"strong"}}]'

An ETag is a version stamp the store returns each time you read. With concurrency: first-write, the write only succeeds if the ETag you send still matches the current one; if another pod wrote in the meantime, you get an HTTP 409 and re-read before retrying. That is optimistic concurrency: no locks, just “check the version at write time.”

4. Configure Kafka pub/sub

Declare the pub/sub component pointing at your Kafka brokers, with SASL credentials again sourced from Vault. This wires the orderpubsub broker that every service will publish to and subscribe from.

# pubsub-kafka.yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: orderpubsub
  namespace: apps
spec:
  type: pubsub.kafka
  version: v1
  metadata:
    - name: brokers
      value: "pkc-xxxxx.westeurope.azure.confluent.cloud:9092"
    - name: authType
      value: "password"            # SASL/PLAIN over TLS
    - name: saslUsername
      secretKeyRef:
        name: kafka-sasl
        key: username
    - name: saslPassword
      secretKeyRef:
        name: kafka-sasl
        key: password
    - name: maxMessageBytes
      value: "1048576"
    - name: consumerGroup
      value: "{appID}"             # per-service consumer group
    - name: initialOffset
      value: "newest"
auth:
  secretStore: vault

The consumerGroup: "{appID}" templating is doing quiet but important work: every subscribing service gets its own Kafka consumer group, so each service receives a full copy of the stream (fan-out), rather than several services accidentally sharing one group and splitting the messages between them. A subscriber declares its interest with a Subscription resource — Dapr routes matching events to the app’s HTTP route, so the app never opens a Kafka connection:

# subscription-shipment.yaml
apiVersion: dapr.io/v2alpha1
kind: Subscription
metadata:
  name: shipment-created-sub
  namespace: apps
spec:
  topic: shipment-created
  pubsubname: orderpubsub
  routes:
    default: /on-shipment-created
scopes:
  - notifications
  - billing

The scopes list is the switch that says “this subscription applies only to the notifications and billing apps” — other apps in the namespace ignore it. Publishing is a single sidecar call from any producer; Dapr wraps the payload in a CloudEvents envelope so consumers get a typed, traceable event:

curl -s -X POST http://localhost:3500/v1.0/publish/orderpubsub/shipment-created \
  -H "Content-Type: application/json" \
  -d '{"orderId":"order-9912","carrier":"DHL","eta":"2026-06-13"}'

The notifications and billing services each receive a POST to /on-shipment-created and must answer 200 (or return {"status":"RETRY"} to redeliver, DROP to dead-letter). Add a dead-letter topic on the subscription (deadLetterTopic: shipment-created-dlq) so poison messages park instead of blocking the partition.

5. Deploy via GitOps and IaC

Do not kubectl apply these by hand in production. The cluster, node pools, Vault, and the managed Redis/Kafka are provisioned with Terraform (with Ansible configuring any self-managed broker VMs/virtual appliances that sit outside Kubernetes). Every Dapr Component, Configuration, and Subscription lives in a git repo and is reconciled by Argo CD, so the desired state is auditable and a bad component is a revert, not a hotfix. A GitHub Actions (or Jenkins) pipeline lints and validates the manifests, runs dapr-bot/conformance checks, and gates merges. Scan the manifests and images in CI with Wiz Code (IaC and container-image misconfiguration scanning — it flags a state store left unencrypted or a sidecar over-privileged before it ships), while Wiz runs runtime CSPM over the live cluster. A failed gate or a drifted component auto-raises a ServiceNow change/incident ticket so platform changes carry an approval trail.

# argocd-app.yaml (snippet)
spec:
  source:
    repoURL: https://github.com/acme/dapr-platform.git
    path: components/apps
    targetRevision: main
  destination:
    namespace: apps
  syncPolicy:
    automated: { prune: true, selfHeal: true }

Validation

Walk these checks top to bottom; each isolates one building block.

# Control plane + mTLS root health
dapr status -k
dapr mtls -k                      # expects: "Mutual TLS is enabled"
dapr mtls expiry -k               # root-cert expiry; alert if < 30 days

# Components are registered to the sidecars
dapr components -k -n apps        # statestore, orderpubsub, vault present

# Service invocation works end to end (run from the checkout pod)
kubectl exec -n apps deploy/checkout -c checkout -- \
  curl -s -X POST http://localhost:3500/v1.0/invoke/inventory/method/reserve \
  -d '{"sku":"SKU-4471","qty":1}'

# Sidecar health/readiness on the data plane
kubectl exec -n apps deploy/checkout -c daprd -- \
  wget -qO- http://localhost:3500/v1.0/healthz && echo OK

# State round-trip and pub/sub delivery show in the sidecar logs
kubectl logs -n apps deploy/notifications -c daprd | grep -i "shipment-created"

In Dynatrace (OneAgent on the node pools, plus the Dapr sidecars exporting OpenTelemetry traces to the OTel collector you set in step 2), confirm a single distributed trace spans checkout → inventory invocation and the publish → subscribe hop, and that the Dapr-emitted metrics (dapr_http_server_request_count, sidecar latency, retry counts) are flowing. Teams on Datadog instead point the same OTLP exporter at the Datadog Agent’s OTLP intake — the Dapr instrumentation is vendor-neutral. Watch for dapr_component_loaded and any component init errors as the canary that a secret reference is wrong.

Going deeper

Everything above gets the three building blocks running. This section is for when you own the platform and need to reason about how Dapr behaves under load, under failure, and under scrutiny.

The sidecar (daprd) and how injection really works

daprd is a single Go binary — the same one whether you use one building block or ten. It runs in your pod, shares the pod’s network namespace (hence localhost), and exposes a fixed set of ports: 3500 (HTTP API), 50001 (gRPC API for your app), 9090 (Prometheus metrics), and an internal gRPC port (50002) that sidecars use to talk to each other for service invocation. At startup it loads every Component scoped to its app-id, opens the connections those components describe (Redis, Kafka, Vault), registers declarative subscriptions, and begins probing your app on dapr.io/app-port to know when it is ready.

Injection is a MutatingWebhookConfiguration named dapr-sidecar-injector. Because it mutates pods at admission time, three consequences follow that trip people up: (1) pods created before the injector was healthy have no sidecar — roll them; (2) the annotations must be on the pod template (spec.template.metadata.annotations), not the Deployment’s own metadata, because the webhook only sees the pod spec; and (3) sidecar resource sizing, extra volumes, and app-health settings are all driven by dapr.io/* annotations the webhook reads at that same moment.

The control plane itself is four (now often five) small services, each doing one job: dapr-operator runs the reconcile loop that watches Component/Configuration/Resiliency/Subscription CRDs and pushes them to sidecars; dapr-sentry is the certificate authority minting SPIFFE SVIDs; dapr-placement-server maps virtual actors to host pods; and dapr-scheduler-server (1.14+) stores and fires scheduled jobs, actor reminders, and workflow timers. If you never use actors, workflows, or the Jobs API, placement and scheduler simply sit idle.

Service invocation: resiliency, retries, and identity

The full call path for checkout → inventory is: your app → local sidecar (HTTP/gRPC) → name resolution (in Kubernetes, DNS resolves the app-id to the target’s sidecar) → target sidecar over mTLS on the internal gRPC port → target app on its app-port. Two guarantees ride along: the mTLS handshake proves each peer’s SPIFFE identity (spiffe://public/ns/apps/inventory), and Dapr applies a built-in default resiliency policy to invocation — it retries transient connection-level failures with backoff even if you never write a policy. That default is why a target pod restarting mid-deploy usually looks like a brief blip, not an error, to the caller.

For anything beyond the default, author a Resiliency CRD. It separates policies (reusable definitions) from targets (what they apply to), and the three policy families compose — a circuit breaker wraps retries, which wrap a timeout:

# resiliency.yaml
apiVersion: dapr.io/v1alpha1
kind: Resiliency
metadata:
  name: invocation-resiliency
  namespace: apps
spec:
  policies:
    timeouts:
      fast: 2s
    retries:
      invokeRetry:
        policy: exponential
        maxInterval: 10s
        maxRetries: 5
        matching:
          httpStatusCodes: "429,500-599"    # only retry these
    circuitBreakers:
      inventoryCB:
        maxRequests: 1
        interval: 30s
        timeout: 60s
        trip: consecutiveFailures >= 5
  targets:
    apps:
      inventory:
        timeout: fast
        retry: invokeRetry
        circuitBreaker: inventoryCB

One caution that matters in production: retries are only safe on idempotent operations. Retrying a non-idempotent POST /charge can double-charge. Use matching to retry only server-side/transient status codes, keep the operation idempotent (idempotency keys), and let the circuit breaker trip open after repeated failures so you stop hammering a sick dependency. Targets can also be components (with inbound/outbound scopes) and actors.

State: consistency, concurrency, and transactions

Dapr’s state API exposes four dials worth understanding:

Dial Options What it means
Concurrency first-write, last-write first-write enforces the ETag (optimistic lock); last-write blindly overwrites
Consistency strong, eventual strong = read-your-writes / quorum where the store supports it; eventual = faster, may lag
ETag opaque version string Returned on read; sent back on write to detect races → 409 on mismatch
TTL ttlInSeconds metadata Auto-expire ephemeral state (sessions, locks)

For multi-key atomicity, stores that implement the transactional interface (Redis, PostgreSQL, Cosmos DB, and more) support an all-or-nothing transaction — commit two writes and a delete as one unit, so a saga step never leaves half-applied state:

curl -s -X POST http://localhost:3500/v1.0/state/statestore/transaction \
  -H "Content-Type: application/json" \
  -d '{
    "operations": [
      {"operation":"upsert","request":{"key":"order-9912","value":{"status":"shipped"}}},
      {"operation":"delete","request":{"key":"reservation-9912"}}
    ]
  }'

Setting actorStateStore: "true" on the component lets the same store back virtual actors, which require a store that supports both transactions and ETags — not every state store qualifies, so check the component’s capabilities before enabling it.

Pub/sub: delivery guarantees and CloudEvents

The single most important fact: Dapr pub/sub is at-least-once, not exactly-once. On the happy path a message is delivered once, but if a consumer crashes after processing but before acknowledging, the broker redelivers on restart. Therefore consumers must be idempotent — dedupe on the CloudEvents id. Nobody gives you exactly-once end-to-end here; the outbox pattern (Dapr’s transactional outbox writes state and publishes atomically) closes the producer gap, and idempotent handlers close the consumer gap.

Every message Dapr publishes is wrapped in a CloudEvents 1.0 envelope, which is what makes cross-service events typed and traceable. Your data sits inside a standard structure:

{
  "specversion": "1.0",
  "type": "com.acme.shipment-created",
  "source": "shipment",
  "id": "a1b2c3d4-0001",
  "time": "2026-06-11T09:14:22Z",
  "datacontenttype": "application/json",
  "pubsubname": "orderpubsub",
  "topic": "shipment-created",
  "traceparent": "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01",
  "data": {"orderId": "order-9912", "carrier": "DHL", "eta": "2026-06-13"}
}

The traceparent field is the W3C trace-context header that stitches the publish and the subscribe into one distributed trace. Interoperating with a non-Dapr producer that writes raw JSON? Set rawPayload: "true" on the subscription so Dapr does not expect an envelope. On ordering: Dapr preserves the broker’s per-partition ordering (Kafka orders within a partition) as long as consumer concurrency does not exceed partitions — it does not add a global ordering guarantee. For throughput, bulkPublish/bulkSubscribe batch messages; for poison messages, deadLetterTopic parks them so one bad event never wedges a partition.

The rest of the catalogue: bindings, actors, workflow, jobs

Service invocation, state, and pub/sub are the entry points, but the same sidecar exposes more building blocks you will reach for later:

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: nightly-reconcile
  namespace: apps
spec:
  type: bindings.cron
  version: v1
  metadata:
    - name: schedule
      value: "@every 1h"
scopes:
  - billing

Observability, end to end

Dapr is instrumented out of the box on all three signals. Traces: sidecars create spans for each invocation and pub/sub hop and propagate W3C trace context, exported over OTLP to any collector (sampling set in the Configuration). Metrics: Prometheus on :9090 exposes dapr_http_server_request_count, gRPC counters, component-load gauges, sidecar latency histograms, and resiliency-activation counts. Logs: JSON structured logs (dapr.io/log-as-json: "true") from both your app and daprd, correlated by app-id and trace id. Because it is all OpenTelemetry, the same wiring feeds Dynatrace, Datadog, Grafana Tempo, or Jaeger without vendor-specific code.

Dapr vs a service mesh

The most common architecture question: “isn’t this just a service mesh?” No — they overlap only on mTLS. A mesh (Istio, Linkerd) is transparent infrastructure: it intercepts all L7 network traffic your app already emits and applies traffic policy, telemetry, and mTLS without the app knowing. Dapr is opt-in application tooling: your code deliberately calls the sidecar’s API to get building blocks a mesh does not have (state, pub/sub, bindings, actors, workflow).

Dimension Dapr Service mesh (Istio / Linkerd)
Model App opts in by calling localhost:3500 APIs Transparent L7 proxy; app is unaware
Primary scope Application building blocks (state, pub/sub, bindings, actors, workflow) Network concerns (traffic mgmt, mTLS, telemetry)
mTLS Yes (Sentry, SPIFFE) Yes (mesh CA, SPIFFE)
Rich traffic shaping (canary, mirroring, L7 routing) Limited Rich
Backing-infra portability (swap store/broker via config) Yes N/A
Can they coexist? Yes — commonly run together Yes

Rule of thumb: reach for a mesh when you want network-level policy across all traffic uniformly (see Istio ambient mesh mTLS & traffic management); reach for Dapr when your code wants portable building blocks. Plenty of production platforms run both — Dapr for application APIs, a mesh for L7 traffic policy — and let each do what it is best at.

Rollback / teardown

Because everything is declarative, rollback is removing manifests or reverting the Argo CD revision — never deleting live broker data by hand.

# Remove app-scoped Dapr resources (Argo CD will also prune on revert)
kubectl delete -f subscription-shipment.yaml -f pubsub-kafka.yaml \
  -f statestore-redis.yaml -f secretstore-vault.yaml -n apps

# Drop the sidecar from a workload: remove the dapr.io/* annotations and roll
kubectl rollout restart deploy/checkout -n apps

# Uninstall the control plane (Helm), then the namespace
helm uninstall dapr -n dapr-system
kubectl delete namespace dapr-system

# Or via the CLI, which also cleans CRDs
dapr uninstall -k --all

The managed Redis and Kafka outlive the Dapr install — tearing down Dapr does not touch your data. If you provisioned them with Terraform, destroy them deliberately with terraform destroy -target on those resources only.

Common pitfalls

Common beginner mistakes

These are misconceptions rather than symptoms — the mental model is wrong, so the fix is to re-frame, not just to patch a flag.

Security notes

mTLS is on for all sidecar traffic from step 1, giving every service a SPIFFE identity issued by Sentry (ideally chained to Vault PKI), so service-to-service traffic is encrypted and authenticated without app code. Layer least-privilege on top: scope each component with scopes: so only the apps that need a broker or store can load it, and use the access-control policies in step 2 to allow only specific app-ids and methods — default-deny. Keep all credentials in Vault referenced via the Dapr secret store; nothing sensitive belongs in a Component literal or a Kubernetes Secret. Human access to the cluster and the Dapr dashboard runs through Okta → Entra ID SSO backing Kubernetes RBAC, not a shared kubeconfig, so every operator action is attributable. CrowdStrike Falcon sensors on the node pool provide runtime threat detection for the sidecars and your app containers, feeding the SOC, while Wiz Code catches misconfigured components (an unencrypted state store, an over-broad access policy) in CI before they reach a cluster. Edge ingress to any externally exposed service sits behind Akamai for TLS termination, WAF, and bot mitigation before traffic reaches the mesh.

Cost notes

Dapr itself is open-source and free; the cost is the sidecar footprint and the backing infrastructure. Each daprd sidecar adds roughly 50–100 MiB of memory and a small CPU slice per pod — set explicit sidecar resource requests/limits via dapr.io/sidecar-memory-request annotations and right-size them, because at a few hundred pods the aggregate is real. The HA control plane (three replicas of four services) is a fixed, modest overhead worth paying in production. The larger line items are the managed Redis (size for state throughput, not peak RAM — Standard/Premium tier with TLS) and managed Kafka (priced on partitions, throughput, and retention — keep retention tight and partition counts matched to real consumer parallelism). Sample tracing at 10% (step 2) rather than 100% to cut Dynatrace/Datadog ingest cost without losing signal. Run the in-cluster Bitnami Redis / Strimzi Kafka options for non-production to avoid paying for managed brokers in dev, and let Terraform tear those environments down on a schedule.

Practice challenges

Work these in order — each builds on the last. Try before opening the solution.

Challenge 1 (beginner) — Get a sidecar injected. You have a payments Deployment with a single container listening on 8080. Add the minimum annotations so Dapr injects a sidecar bound to appconfig, then prove the pod has two containers.

<details> <summary>Solution</summary>

Put the annotations on the pod template, not the Deployment metadata:

spec:
  template:
    metadata:
      annotations:
        dapr.io/enabled: "true"
        dapr.io/app-id: "payments"
        dapr.io/app-port: "8080"
        dapr.io/config: "appconfig"

Roll and verify: kubectl get pod -n apps -l app=payments -o jsonpath='{.items[0].spec.containers[*].name}' should list payments daprd. Why: the injector is an admission webhook that only reads the pod template, so a fresh pod is required and the annotations must live under spec.template.metadata. </details>

Challenge 2 (beginner) — Round-trip state and see the namespacing. From the payments pod, save {"balance":100} under key acct-1, read it back, then find the actual key in Redis.

<details> <summary>Solution</summary>

kubectl exec -n apps deploy/payments -c payments -- \
  curl -s -X POST http://localhost:3500/v1.0/state/statestore \
  -H "Content-Type: application/json" \
  -d '[{"key":"acct-1","value":{"balance":100}}]'

kubectl exec -n apps deploy/payments -c payments -- \
  curl -s http://localhost:3500/v1.0/state/statestore/acct-1

In Redis the key is payments||acct-1. Why: Dapr prefixes every key with <app-id>|| so two services never collide on the same logical key. </details>

Challenge 3 (intermediate) — Declarative subscribe + publish. Route the topic payment-captured on orderpubsub to the ledger app’s /on-payment route, restricted to ledger only, then publish a test event and confirm delivery.

<details> <summary>Solution</summary>

apiVersion: dapr.io/v2alpha1
kind: Subscription
metadata:
  name: payment-captured-sub
  namespace: apps
spec:
  topic: payment-captured
  pubsubname: orderpubsub
  routes:
    default: /on-payment
scopes:
  - ledger

curl -s -X POST http://localhost:3500/v1.0/publish/orderpubsub/payment-captured \
  -H "Content-Type: application/json" -d '{"orderId":"order-9912","amount":42}'
kubectl logs -n apps deploy/ledger -c daprd | grep -i payment-captured

Why: scopes limits the subscription to the ledger app; the sidecar delivers a CloudEvents-wrapped POST to /on-payment, so ledger never opens a Kafka connection. </details>

Challenge 4 (intermediate) — Lock invocation down to one caller. Only checkout should be allowed to call payments’s /capture method over POST; everything else denied. Add the access-control policy to payments’s Configuration.

<details> <summary>Solution</summary>

spec:
  accessControl:
    defaultAction: deny
    trustDomain: "public"
    policies:
      - appId: checkout
        defaultAction: deny
        trustDomain: "public"
        namespace: "apps"
        operations:
          - name: /capture
            httpVerb: ["POST"]
            action: allow

Why: defaultAction: deny refuses everything not explicitly allowed; the policy is evaluated against the caller’s verified SPIFFE identity, so only an mTLS-authenticated checkout calling POST /capture gets through. </details>

Challenge 5 (advanced) — Add resiliency + scope the store. Give calls to payments a 2s timeout, up to 4 exponential retries on 5xx, and a circuit breaker after 5 consecutive failures. Separately, scope the statestore component so only checkout and payments can load it.

<details> <summary>Solution</summary>

apiVersion: dapr.io/v1alpha1
kind: Resiliency
metadata: { name: payments-resiliency, namespace: apps }
spec:
  policies:
    timeouts: { quick: 2s }
    retries:
      pay:
        policy: exponential
        maxInterval: 8s
        maxRetries: 4
        matching: { httpStatusCodes: "500-599" }
    circuitBreakers:
      payCB: { maxRequests: 1, interval: 30s, timeout: 60s, trip: consecutiveFailures >= 5 }
  targets:
    apps:
      payments: { timeout: quick, retry: pay, circuitBreaker: payCB }

Then add to statestore-redis.yaml (top level, sibling to spec):

scopes:
  - checkout
  - payments

Why: policies compose (breaker → retries → timeout) and apply only to the named target; scopes turns the store from namespace-wide into an allow-list, so no other app loads it. Retrying only 5xx keeps non-idempotent failures from being replayed blindly. </details>

Challenge 6 (advanced) — Atomic saga step. In one call, mark order-9912 shipped and delete its reservation-9912 record, all-or-nothing.

<details> <summary>Solution</summary>

curl -s -X POST http://localhost:3500/v1.0/state/statestore/transaction \
  -H "Content-Type: application/json" \
  -d '{"operations":[
        {"operation":"upsert","request":{"key":"order-9912","value":{"status":"shipped"}}},
        {"operation":"delete","request":{"key":"reservation-9912"}}
      ]}'

Why: the /transaction endpoint commits both operations atomically on stores that support it (Redis, Postgres, Cosmos), so a saga step never leaves the order shipped but the reservation still held. </details>

Glossary

DaprKubernetesMicroservicesPub/SubService MeshPlatform
Need this built for real?

Vinod is a Senior Cloud Architect (22+ yrs) — available for Azure / AWS / GCP architecture, landing zones, and migrations.

Work with me
VH

Written by Vinod

Senior Cloud Architect with 22+ years of experience. Sharing practical knowledge from production environments.

Comments