Configure Dapr on Kubernetes for Service Invocation, State, and Pub/Sub Building Blocks

A logistics company is breaking a monolithic order-management system into a dozen Go and .NET microservices, and the platform team keeps re-solving the same three problems in every service: how does the checkout service call the inventory service securely and find it without hardcoding a DNS name; where does a long-running saga stash its state so a pod restart does not lose an in-flight order; and how does a shipment-created event reach the four services that care about it without each one growing a bespoke Kafka client. Rewriting that plumbing per service — per language — is where the velocity went. Dapr (Distributed Application Runtime) solves exactly this: a sidecar that exposes service invocation, state management, and pub/sub as plain HTTP/gRPC APIs, so a service calls http://localhost:3500/v1.0/invoke/inventory/method/reserve and the sidecar handles discovery, mTLS, retries, and the broker wire protocol. This guide stands up the Dapr control plane on Kubernetes and wires all three building blocks — mTLS service invocation, a Redis state store, and Kafka pub/sub — the way you would run them in production.

Prerequisites:

• A Kubernetes cluster, v1.27+ (AKS, EKS, GKE, or on-prem), with kubectl context set and cluster-admin for the install.
• helm v3.12+ and the Dapr CLI (dapr) v1.14+ installed locally.
• A Redis instance reachable from the cluster (managed — Azure Cache for Redis / ElastiCache — or in-cluster Bitnami Redis) for the state store.
• A Kafka cluster reachable from the cluster (managed — Confluent Cloud / MSK / Event Hubs Kafka endpoint — or Strimzi in-cluster) for pub/sub.
• HashiCorp Vault reachable for component secrets (broker SASL password, Redis password), so no credential is committed to git.
• An IdP for the dashboard and operator access: Okta as the workforce IdP federated to Microsoft Entra ID, which backs Kubernetes RBAC (AKS AAD integration) so engineers authenticate with corporate SSO, not a shared kubeconfig.

Target topology

The picture has two planes. The control plane is a set of Dapr system pods in the dapr-system namespace: dapr-operator (watches Component and Configuration CRDs and reconciles them to sidecars), dapr-sidecar-injector (a mutating webhook that injects the daprd sidecar into any pod annotated dapr.io/enabled: "true"), dapr-sentry (the CA that issues and rotates the X.509 SVIDs used for mTLS between sidecars), and dapr-placement (the actor placement service — present even if you do not use actors today). The data plane is your application pods in the apps namespace, each running your container plus an injected daprd sidecar. Service-to-service calls go pod → its own sidecar → (mTLS) → target sidecar → target app; state and pub/sub calls go pod → its own sidecar → Redis or Kafka. Everything you configure — the Redis state store, the Kafka pub/sub broker, the mTLS policy — is declared as Kubernetes resources that the operator hands to the sidecars at runtime.

1. Install the Dapr control plane

Install with Helm so the deployment is declarative and lands cleanly in GitOps. Add the repo, create the namespace, and install with high availability (three replicas of each control-plane service) and mTLS enabled from the start.

helm repo add dapr https://dapr.github.io/helm-charts/
helm repo update

helm upgrade --install dapr dapr/dapr \
  --version 1.14.4 \
  --namespace dapr-system \
  --create-namespace \
  --set global.ha.enabled=true \
  --set global.mtls.enabled=true \
  --set global.mtls.workloadCertTTL=24h \
  --set dapr_sentry.logLevel=info \
  --wait

Confirm the control plane is healthy. The CLI reports each service and the mTLS root-cert expiry:

dapr status -k

  NAME                   NAMESPACE    HEALTHY  STATUS   REPLICAS  VERSION  AGE
  dapr-operator          dapr-system  True     Running  3         1.14.4   2m
  dapr-sentry            dapr-system  True     Running  3         1.14.4   2m
  dapr-sidecar-injector  dapr-system  True     Running  3         1.14.4   2m
  dapr-placement-server  dapr-system  True     Running  3         1.14.4   2m

For production, do not let Sentry self-generate its CA. Issue the trust bundle from HashiCorp Vault’s PKI secrets engine and provide it to the chart (dapr_sentry.tls.issuer*), so the mTLS root chains to your corporate PKI and rotates on Vault’s schedule rather than a Helm value. The 24-hour workloadCertTTL means leaf SVIDs roll automatically and a leaked cert is short-lived.

2. Enable application-level mTLS service invocation

Create the application namespace and a Dapr Configuration that turns on tracing and pins the mTLS settings the sidecars enforce.

kubectl create namespace apps

# dapr-config.yaml
apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
  name: appconfig
  namespace: apps
spec:
  mtls:
    enabled: true
    workloadCertTTL: 24h
    allowedClockSkew: 15m
  tracing:
    samplingRate: "0.1"          # 10% sampled to keep cost sane
    otel:
      endpointAddress: "otel-collector.observability:4317"
      isSecure: true
      protocol: grpc

Annotate each workload so the injector adds a sidecar and binds this config. Here is the inventory service (the callee) and the checkout service (the caller):

# inventory-deploy.yaml (snippet)
spec:
  template:
    metadata:
      annotations:
        dapr.io/enabled: "true"
        dapr.io/app-id: "inventory"
        dapr.io/app-port: "8080"
        dapr.io/config: "appconfig"
        dapr.io/log-as-json: "true"

Once both are deployed, the checkout service invokes inventory purely by app-id — no service URL, no client-side TLS code. The call is name-resolved by Dapr (mDNS locally, Kubernetes DNS in-cluster) and encrypted sidecar-to-sidecar:

# from inside the checkout pod, calling its own sidecar on 3500
curl -s -X POST http://localhost:3500/v1.0/invoke/inventory/method/reserve \
  -H "Content-Type: application/json" \
  -d '{"sku":"SKU-4471","qty":2}'

To restrict who may call what — so only checkout can hit inventory/reserve — add an access-control policy to the callee’s Configuration. This is Dapr’s app-level authorization, evaluated against the verified SPIFFE identity in the peer’s certificate:

  accessControl:
    defaultAction: deny
    trustDomain: "public"
    policies:
      - appId: checkout
        defaultAction: deny
        trustDomain: "public"
        namespace: "apps"
        operations:
          - name: /reserve
            httpVerb: ["POST"]
            action: allow

3. Wire the Redis state store

Pull the Redis password from Vault rather than a literal. Install the Dapr Vault secret-store component (so other components can reference Vault), then declare the state store referencing that secret.

# secretstore-vault.yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: vault
  namespace: apps
spec:
  type: secretstores.hashicorp.vault
  version: v1
  metadata:
    - name: vaultAddr
      value: "https://vault.security.internal:8200"
    - name: enginePath
      value: "secret"
    - name: vaultKubernetesMountPath
      value: "kubernetes"          # Vault Kubernetes auth role bound to the pod SA

# statestore-redis.yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: statestore
  namespace: apps
spec:
  type: state.redis
  version: v1
  metadata:
    - name: redisHost
      value: "orders-redis.redis.cache.windows.net:6380"
    - name: enableTLS
      value: "true"
    - name: redisPassword
      secretKeyRef:
        name: orders-redis-password   # key path inside Vault
        key: password
    - name: actorStateStore
      value: "true"                    # also serve actor state if needed
auth:
  secretStore: vault

Apply and verify the operator accepted both components:

kubectl apply -f secretstore-vault.yaml -f statestore-redis.yaml
dapr components -k -n apps

The checkout saga now persists and reads state through its sidecar — no Redis client in the app, and key is automatically namespaced as <app-id>||<key> so services cannot collide:

# save state
curl -s -X POST http://localhost:3500/v1.0/state/statestore \
  -H "Content-Type: application/json" \
  -d '[{"key":"order-9912","value":{"status":"reserving","items":2}}]'

# read it back
curl -s http://localhost:3500/v1.0/state/statestore/order-9912

Use Dapr’s ETag-based optimistic concurrency and explicit consistency for saga steps — pass concurrency=first-write and consistency=strong so two pods racing on the same order do not clobber each other:

curl -s -X POST "http://localhost:3500/v1.0/state/statestore" \
  -d '[{"key":"order-9912","value":{"status":"shipped"},"etag":"3","options":{"concurrency":"first-write","consistency":"strong"}}]'

4. Configure Kafka pub/sub

Declare the pub/sub component pointing at your Kafka brokers, with SASL credentials again sourced from Vault. This wires the orderpubsub broker that every service will publish to and subscribe from.

# pubsub-kafka.yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: orderpubsub
  namespace: apps
spec:
  type: pubsub.kafka
  version: v1
  metadata:
    - name: brokers
      value: "pkc-xxxxx.westeurope.azure.confluent.cloud:9092"
    - name: authType
      value: "password"            # SASL/PLAIN over TLS
    - name: saslUsername
      secretKeyRef:
        name: kafka-sasl
        key: username
    - name: saslPassword
      secretKeyRef:
        name: kafka-sasl
        key: password
    - name: maxMessageBytes
      value: "1048576"
    - name: consumerGroup
      value: "{appID}"             # per-service consumer group
    - name: initialOffset
      value: "newest"
auth:
  secretStore: vault

A subscriber declares its interest with a Subscription resource — Dapr routes matching events to the app’s HTTP route, so the app never opens a Kafka connection:

# subscription-shipment.yaml
apiVersion: dapr.io/v2alpha1
kind: Subscription
metadata:
  name: shipment-created-sub
  namespace: apps
spec:
  topic: shipment-created
  pubsubname: orderpubsub
  routes:
    default: /on-shipment-created
scopes:
  - notifications
  - billing

Publishing is a single sidecar call from any producer; Dapr wraps the payload in a CloudEvents envelope so consumers get a typed, traceable event:

curl -s -X POST http://localhost:3500/v1.0/publish/orderpubsub/shipment-created \
  -H "Content-Type: application/json" \
  -d '{"orderId":"order-9912","carrier":"DHL","eta":"2026-06-13"}'

The notifications and billing services each receive a POST to /on-shipment-created and must answer 200 (or return {"status":"RETRY"} to redeliver, DROP to dead-letter). Add a dead-letter topic on the subscription (deadLetterTopic: shipment-created-dlq) so poison messages park instead of blocking the partition.

5. Deploy via GitOps and IaC

Do not kubectl apply these by hand in production. The cluster, node pools, Vault, and the managed Redis/Kafka are provisioned with Terraform (with Ansible configuring any self-managed broker VMs/virtual appliances that sit outside Kubernetes). Every Dapr Component, Configuration, and Subscription lives in a git repo and is reconciled by Argo CD, so the desired state is auditable and a bad component is a revert, not a hotfix. A GitHub Actions (or Jenkins) pipeline lints and validates the manifests, runs dapr-bot/conformance checks, and gates merges. Scan the manifests and images in CI with Wiz Code (IaC and container-image misconfiguration scanning — it flags a state store left unencrypted or a sidecar over-privileged before it ships), while Wiz runs runtime CSPM over the live cluster. A failed gate or a drifted component auto-raises a ServiceNow change/incident ticket so platform changes carry an approval trail.

# argocd-app.yaml (snippet)
spec:
  source:
    repoURL: https://github.com/acme/dapr-platform.git
    path: components/apps
    targetRevision: main
  destination:
    namespace: apps
  syncPolicy:
    automated: { prune: true, selfHeal: true }

Validation

Walk these checks top to bottom; each isolates one building block.

# Control plane + mTLS root health
dapr status -k
dapr mtls -k                      # expects: "Mutual TLS is enabled"
dapr mtls expiry -k               # root-cert expiry; alert if < 30 days

# Components are registered to the sidecars
dapr components -k -n apps        # statestore, orderpubsub, vault present

# Service invocation works end to end (run from the checkout pod)
kubectl exec -n apps deploy/checkout -c checkout -- \
  curl -s -X POST http://localhost:3500/v1.0/invoke/inventory/method/reserve \
  -d '{"sku":"SKU-4471","qty":1}'

# Sidecar health/readiness on the data plane
kubectl exec -n apps deploy/checkout -c daprd -- \
  wget -qO- http://localhost:3500/v1.0/healthz && echo OK

# State round-trip and pub/sub delivery show in the sidecar logs
kubectl logs -n apps deploy/notifications -c daprd | grep -i "shipment-created"

In Dynatrace (OneAgent on the node pools, plus the Dapr sidecars exporting OpenTelemetry traces to the OTel collector you set in step 2), confirm a single distributed trace spans checkout → inventory invocation and the publish → subscribe hop, and that the Dapr-emitted metrics (dapr_http_server_request_count, sidecar latency, retry counts) are flowing. Teams on Datadog instead point the same OTLP exporter at the Datadog Agent’s OTLP intake — the Dapr instrumentation is vendor-neutral. Watch for dapr_component_loaded and any component init errors as the canary that a secret reference is wrong.

Rollback / teardown

Because everything is declarative, rollback is removing manifests or reverting the Argo CD revision — never deleting live broker data by hand.

# Remove app-scoped Dapr resources (Argo CD will also prune on revert)
kubectl delete -f subscription-shipment.yaml -f pubsub-kafka.yaml \
  -f statestore-redis.yaml -f secretstore-vault.yaml -n apps

# Drop the sidecar from a workload: remove the dapr.io/* annotations and roll
kubectl rollout restart deploy/checkout -n apps

# Uninstall the control plane (Helm), then the namespace
helm uninstall dapr -n dapr-system
kubectl delete namespace dapr-system

# Or via the CLI, which also cleans CRDs
dapr uninstall -k --all

The managed Redis and Kafka outlive the Dapr install — tearing down Dapr does not touch your data. If you provisioned them with Terraform, destroy them deliberately with terraform destroy -target on those resources only.

Common pitfalls

• Sidecar not injected. The pod has no daprd container. Almost always the dapr.io/enabled annotation is on the Deployment metadata, not the pod template metadata — it must be on spec.template.metadata.annotations. Verify the injector webhook is healthy with kubectl get mutatingwebhookconfiguration dapr-sidecar-injector.
• app-port mismatch. dapr.io/app-port must equal the port your container actually listens on, or invocation and subscription POSTs hit nothing and the sidecar logs connection refused.
• mTLS clock skew. Cross-node SVID validation fails if node clocks drift beyond allowedClockSkew. Keep NTP healthy; 15m skew is forgiving but not infinite.
• Secret reference resolves empty. A secretKeyRef that points at a Vault path the pod’s service account is not bound to comes back blank, and the component fails to init with a quiet log line. Check the Vault Kubernetes auth role binds the right namespace + service account.
• Consumer group collisions. Two services sharing a Kafka consumerGroup will split partitions and each miss half the events. Use consumerGroup: "{appID}" so every app gets its own group.
• No dead-letter topic. Without one, a single un-processable message blocks its partition and backs up the topic. Always set deadLetterTopic.
• mTLS root expiry crept up. The Sentry root cert has a finite life; if it lapses, every invocation fails at once. Alert on dapr mtls expiry -k < 30 days and rotate via Vault PKI ahead of time.

Security notes

mTLS is on for all sidecar traffic from step 1, giving every service a SPIFFE identity issued by Sentry (ideally chained to Vault PKI), so service-to-service traffic is encrypted and authenticated without app code. Layer least-privilege on top: scope each component with scopes: so only the apps that need a broker or store can load it, and use the access-control policies in step 2 to allow only specific app-ids and methods — default-deny. Keep all credentials in Vault referenced via the Dapr secret store; nothing sensitive belongs in a Component literal or a Kubernetes Secret. Human access to the cluster and the Dapr dashboard runs through Okta → Entra ID SSO backing Kubernetes RBAC, not a shared kubeconfig, so every operator action is attributable. CrowdStrike Falcon sensors on the node pool provide runtime threat detection for the sidecars and your app containers, feeding the SOC, while Wiz Code catches misconfigured components (an unencrypted state store, an over-broad access policy) in CI before they reach a cluster. Edge ingress to any externally exposed service sits behind Akamai for TLS termination, WAF, and bot mitigation before traffic reaches the mesh.

Cost notes

Dapr itself is open-source and free; the cost is the sidecar footprint and the backing infrastructure. Each daprd sidecar adds roughly 50–100 MiB of memory and a small CPU slice per pod — set explicit sidecar resource requests/limits via dapr.io/sidecar-memory-request annotations and right-size them, because at a few hundred pods the aggregate is real. The HA control plane (three replicas of four services) is a fixed, modest overhead worth paying in production. The larger line items are the managed Redis (size for state throughput, not peak RAM — Standard/Premium tier with TLS) and managed Kafka (priced on partitions, throughput, and retention — keep retention tight and partition counts matched to real consumer parallelism). Sample tracing at 10% (step 2) rather than 100% to cut Dynatrace/Datadog ingest cost without losing signal. Run the in-cluster Bitnami Redis / Strimzi Kafka options for non-production to avoid paying for managed brokers in dev, and let Terraform tear those environments down on a schedule.