Self-Managed Kafka on Kubernetes with Strimzi for a Trading Platform

A mid-tier proprietary trading firm — call it the kind of shop that runs a few hundred million dollars of equities and futures strategies out of a single co-located rack near the exchange — gives its platform team an ultimatum. The market-data fan-out bus that feeds every strategy engine, the order-state journal, and the post-trade risk feed all run on a managed streaming service, and during the open and the close the firm is watching its tail latency blow out from a comfortable two milliseconds to forty, with occasional throttle-induced stalls that show up directly as missed fills on the desk. A missed fill is real money, and the head of trading does not care that the cloud provider’s SLA was technically met. The mandate: own the streaming layer, get the p99 down and keep it there, prove it with numbers, and do it without hiring a five-person Kafka operations team. This article is the reference architecture for that — a self-managed, self-healing Apache Kafka platform on Kubernetes, operated by Strimzi, tuned for a low-latency trading feed, and instrumented so the firm’s CTO can see the SLO is holding in real time.

The pressures in trading stack differently from a typical enterprise. Latency is the product, not a feature — the entire reason to leave a managed service is that you cannot tune a broker you do not control, cannot pin it to the right instance type, and cannot eliminate the noisy-neighbor throttle that costs you fills. Durability still matters because the order journal is a regulated record under MiFID II / SEC 17a-4 retention, so “fast but lossy” is not on the table. Availability means surviving the loss of an Availability Zone mid-session without a trading halt. And operational leverage means a two-person platform team has to run this, which is the single fact that makes Strimzi — not a hand-rolled Kafka install — the only sane answer.

In a nutshell

Strip away the trading jargon and this lesson is about one tool: Strimzi, the Kubernetes operator for Apache Kafka. Two ideas unlock the whole thing.

First, Kafka is a durable, replayable message log — a giant append-only tape that producers write events onto and many consumers read from at their own pace. It runs as a cluster of brokers (the servers that hold the data), and every stream of messages is a topic split into partitions that are copied (replicated) across brokers so no single machine losing power loses your data. That is the entire product a trading desk is buying: a fast, lossless firehose of market data and order events.

Second, an operator is a piece of software you install into your Kubernetes cluster that behaves like a tireless human administrator for one specific system. Think of Strimzi as a robot Kafka admin you hire once and it never sleeps. You do not run kafka-topics.sh scripts or hand-edit server configs. Instead you write a few YAML files describing what you want — “a 6-broker cluster spread across three data-centre zones,” “a topic called orders.journal with 24 partitions,” “a user allowed to write only to the quote feed” — and Strimzi’s controller watches those files and makes reality match, forever. Those YAML files are custom resources (Kafka, KafkaNodePool, KafkaTopic, KafkaUser, and friends): Kubernetes objects that Strimzi taught the cluster to understand, exactly like the operator and CRD pattern generalizes. Provisioning brokers, rolling upgrades one at a time, renewing TLS certificates before they expire, replacing a dead broker at 3 a.m. — the operator does all of it so a two-person team does not have to.

The twist in this lesson is the word low-latency. Running Kafka is easy; running it so the time from “an order event is produced” to “a strategy engine consumes it” stays under a handful of milliseconds even during the market open is where you earn your keep. That is why so much of what follows is tuning — which machines the brokers sit on, how they use the disk and the CPU, how replicas are placed across zones, and what durability you trade for speed on each topic. Managed Kafka hides those knobs; a trading firm leaves managed Kafka precisely to turn them.

If you hold four phrases, you have the spine of the lesson: Kafka = a replicated event log. Broker = a server holding the data. Operator (Strimzi) = a robot admin driven by YAML. Low-latency = tune every layer so the desk gets its fills. Everything below is the detail behind those four.

Level: Advanced — with a beginner on-ramp · Time: ~30 min

Before this lesson, it helps to be comfortable with the operator and CRD / controller pattern and with StatefulSets: stable identity, storage & ordering (Strimzi runs brokers as identity-stable, disk-sticky pods). Each stands alone if you are not. If you want the contrasting operator, the sibling Confluent Platform Kafka on Kubernetes builds the same system with a different vendor’s operator.

After this lesson you will be able to:

• Explain, to a skeptic, why a latency-critical trading feed self-manages Kafka with Strimzi instead of using a managed service — and when that trade is wrong.
• Name the core Strimzi custom resources (Kafka, KafkaNodePool, KafkaTopic, KafkaUser, KafkaMirrorMaker2, KafkaRebalance) and say what each one provisions.
• Reason about the durability/latency trade behind acks, min.insync.replicas, replication factor, and rack awareness — and set them per topic on purpose.
• Describe how KRaft controllers, tiered storage, follower fetching, and JVM/GC tuning each shave latency or cost, and where the microseconds actually go.
• Spot the classic self-managed-Kafka footguns — shared nodes, no rack awareness, GC pauses, slow PVs, acks=0 — before they page you.

---

Why self-manage, and why an operator

The honest first question is whether to self-manage Kafka at all. Managed Kafka (MSK, Confluent Cloud) is the right default for most teams, and naming why this firm leaves it matters because someone will rightly challenge the decision.

The firm leaves managed Kafka for exactly three reasons that do not apply to a normal SaaS backend. First, broker placement and instance type — a low-latency feed wants brokers pinned to specific high-clock, network-optimized instances in the same AZ as the strategy engines, with local NVMe for the commit log, and a managed service abstracts that away. Second, kernel and JVM tuning — the page-cache behavior, the vm.dirty_ratio, the G1 pause targets, the NIC interrupt affinity all matter at single-digit-millisecond p99, and you cannot touch them on a managed broker. Third, throttle control — managed services protect themselves with quotas that surface as latency spikes precisely at the open and close, which is the worst possible time for this workload.

But self-managing Kafka the old way — Ansible playbooks, hand-rolled failover runbooks, 3 a.m. pages to manually reassign partitions when a broker dies — is exactly the five-person-team cost the firm is trying to avoid. Strimzi resolves the contradiction. It is a Kubernetes operator that turns Kafka into declarative custom resources: you describe the cluster you want (Kafka, KafkaNodePool, KafkaTopic, KafkaUser) in YAML, and the operator continuously reconciles reality to match — provisioning brokers, rolling upgrades one broker at a time with controlled-shutdown, regenerating TLS certs before they expire, and recovering a failed broker onto a healthy node without a human in the loop. The firm gets the control of self-managed Kafka with something close to the operational cost of a managed one. The whole platform runs on Amazon EKS so the operator, the brokers, and the firm’s own tooling share one control plane and one IAM story.

The Strimzi model: Kafka as custom resources

Before the architecture, understand how Strimzi represents Kafka, because everything downstream is one of these objects. Strimzi installs a Cluster Operator — a controller pod that runs a never-ending reconcile loop (observe the desired custom resources, diff against the live cluster, act to close the gap). You never kubectl exec into a broker to create a topic; you apply a YAML object and let the operator do it. The full set of Strimzi custom resources is small enough to memorize:

Two of these — KafkaTopic and KafkaUser — are reconciled by the Entity Operator, a companion the Cluster Operator deploys alongside the cluster; it contains the Topic Operator (turns KafkaTopic objects into real topics) and the User Operator (turns KafkaUser objects into real users, ACLs, and, for mTLS, a client certificate delivered as a Kubernetes Secret). This is the payoff of the operator model: a topic and a permission are GitOps artifacts, reviewed in a pull request, not imperative commands typed at a shell. A topic and the user allowed to write it look like this:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: orders.journal
  labels:
    strimzi.io/cluster: trading-bus     # which Kafka this topic belongs to
spec:
  partitions: 24                        # unit of consumer parallelism
  replicas: 3                           # replication factor across brokers/AZs
  config:
    min.insync.replicas: "2"            # a write needs 2 in-sync copies to ack
    retention.ms: "220903200000"        # ~7 years total; the tail lives in tiered storage
    cleanup.policy: delete

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
  name: order-management-system
  labels:
    strimzi.io/cluster: trading-bus
spec:
  authentication:
    type: tls                           # mTLS: Strimzi issues this client its own cert
  authorization:
    type: simple                        # Kafka ACLs, deny by default
    acls:
      - resource:
          type: topic
          name: orders.journal
          patternType: literal
        operations: [Write, Describe]   # least privilege: write, nothing else
        host: "*"

How the brokers actually run. A broker is stateful — it owns a specific slice of the log on a specific disk — so it cannot be an anonymous, disposable pod. Each broker needs a stable identity and its own persistent volume that re-attaches when the pod reschedules, which is exactly what a StatefulSet provides. Strimzi originally used StatefulSets and now manages the pods with its own StrimziPodSet controller (which superseded StatefulSets in modern Strimzi) — same guarantees of stable name and sticky disk, but with the fine-grained, one-broker-at-a-time rolling control the operator needs for safe upgrades. You do not write the StrimziPodSet; the operator generates it from your Kafka and KafkaNodePool.

Listeners, TLS, and auth in one breath. A Kafka resource exposes one or more listeners — named ports with a type (internal for in-cluster clients; nodeport, loadbalancer, ingress, or cluster-ip for external ones), a TLS flag, and an authentication mode (tls for mutual-TLS client certs, scram-sha-512 for username/password, or oauth for token-based access). This platform uses a single TLS-encrypted internal listener with authentication.type: tls, so every producer and consumer must present a Strimzi-issued client certificate before it can open a session. The KafkaUser above is what mints that certificate and attaches the ACLs. Put together, the operator model means the whole security posture — encryption, identity, permissions — is declared in YAML and rotated automatically, which is the single biggest reason a two-person team can run this safely.

Architecture overview

The platform has two planes that share infrastructure but live on different timescales: a hot data plane — producers slamming market data and order events through brokers to consumers, where every microsecond is measured — and a control plane — the Strimzi operator, certificate rotation, MirrorMaker, and observability, which runs continuously but off the critical latency path. Keeping these separate in your head is the first step to operating this well.

The defining property of the topology is rack awareness mapped to AWS Availability Zones. The Kafka cluster runs across three AZs in one region. Strimzi labels each broker with its zone via the broker.rack config sourced from the node’s topology.kubernetes.io/zone label, and Kafka’s rack-aware replica placement then guarantees that the replicas of any partition land in different AZs. Lose an entire AZ and every partition still has a surviving in-sync replica in another zone — no data loss, no trading halt, just a leader election that completes in well under a second.

Hot path, following the data flow:

1. Producers — the market-data gateway that normalizes the exchange feed, and the order-management system that journals every order state transition — run as pods in the same AZ as the partition leaders they write to, so the hot write path never crosses a zone boundary and never pays the inter-AZ network hop. They connect over the internal listener with mTLS.
2. The write lands on the partition leader broker. For the order journal, the producer runs with acks=all and the topic has min.insync.replicas=2 across a replication factor of 3 — the write is only acknowledged once a second AZ has it, which is what makes the journal durable enough for a regulated record. For the ephemeral market-data fan-out, where a dropped tick is re-sent on the next quote and durability is worthless, producers run acks=1 to shave the acknowledgement latency.
3. Kafka uses KRaft mode — no ZooKeeper — so cluster metadata and leader election live in a dedicated quorum of controller nodes, removing an entire failure-prone dependency and cutting failover time. Strimzi provisions the controllers as their own KafkaNodePool, isolated from the brokers that carry the hot data.
4. Consumers — the strategy engines, the real-time risk aggregator, the order-state cache — read from the leader (or from a rack-local follower via follower fetching, so a consumer reads from a replica in its own AZ and avoids the cross-zone read hop). They commit offsets back to Kafka.
5. The commit log for the latency-sensitive topics sits on local NVMe instance storage on the broker nodes for the lowest possible write latency, with tiered storage offloading older closed log segments to Amazon S3 so the brokers keep only the recent hot data on fast local disk and the long retention required for compliance lives cheaply in object storage.

Control plane, continuous and off the hot path: the Strimzi Cluster Operator watches the custom resources and reconciles the cluster. MirrorMaker 2 (run by Strimzi) replicates the order journal and other durable topics to a second AWS region for disaster recovery. Prometheus scrapes JMX metrics from every broker via the Strimzi metrics exporter, and Grafana renders the latency SLO dashboards the desk and the CTO watch.

Component breakdown

A few of these choices deserve the why, because they are the ones teams get wrong.

Why tiered storage instead of just big local disks. A trading firm’s order journal must be retained for years, but the brokers only ever serve the recent window at low latency. Putting years of cold segments on local NVMe is ruinously expensive and forces enormous brokers whose recovery (re-replicating a full disk after a node loss) takes hours. Tiered storage keeps only a few hours of hot data on local NVMe and transparently offloads closed segments to S3; a broker’s local footprint stays small, so recovery is fast, while retention is effectively unlimited and cheap. The trade is that a consumer reading far back in history pays an S3 fetch — irrelevant for a real-time desk, acceptable for the occasional compliance replay.

Why KRaft, not ZooKeeper. ZooKeeper was a second distributed system to operate, secure, and recover, and its loss could freeze the cluster. KRaft folds metadata management into a Kafka controller quorum, removing that dependency entirely, shrinking failover time during a broker loss, and simplifying the security surface — one fewer thing for a two-person team to run and for Wiz to have to assess.

Why mTLS everywhere, not just at the edge. In a flat Kubernetes network, any compromised pod can reach a broker port. mTLS means every producer and consumer presents a client certificate the broker verifies, so an unauthorized pod cannot even open a session, and all traffic is encrypted in transit — table stakes for order data. Strimzi runs its own certificate authority, issues per-client certs through the KafkaUser resource, and rotates them automatically before expiry, which removes the classic self-managed-Kafka footgun of a cluster-wide outage when a hand-managed cert silently expires.

Implementation guidance

Provision the substrate with Terraform, then hand the cluster to GitOps. The split matters: Terraform owns the slow-moving cloud substrate (VPC across three AZs, the EKS cluster, node groups on the right instance families, the S3 tiered-storage bucket, IAM/IRSA roles), and Argo CD owns the fast-moving Kafka definition (the Strimzi CRs), so a topic or broker-config change is a reviewed Git commit that Argo syncs — never a kubectl apply from someone’s laptop.

A trimmed Kafka custom resource communicates the intent — KRaft, rack-aware, tuned listeners:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: trading-bus
  annotations:
    strimzi.io/node-pools: enabled
    strimzi.io/kraft: enabled
spec:
  kafka:
    replicas: 6
    rack:
      topologyKey: topology.kubernetes.io/zone   # spread replicas across AZs
    listeners:
      - name: tls
        port: 9093
        type: internal
        tls: true
        authentication:
          type: tls                               # mTLS for every client
    config:
      default.replication.factor: 3
      min.insync.replicas: 2
      offsets.topic.replication.factor: 3
      replica.selector.class: org.apache.kafka.common.replica.RackAwareReplicaSelector  # follower fetch

And the broker node pool, pinned to local-NVMe instances with JBOD storage:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: brokers
  labels:
    strimzi.io/cluster: trading-bus
spec:
  replicas: 6
  roles: [broker]
  storage:
    type: jbod
    volumes:
      - id: 0
        type: ephemeral          # local NVMe instance store for the hot commit log
        sizeLimit: 800Gi

The CI that applies this runs in GitHub Actions, authenticating to AWS via OIDC federation so there is no stored access key to leak, and Wiz Code scans the Terraform and Helm/CR manifests in the pull request — flagging a publicly readable S3 bucket or an over-broad security group before it ever reaches a cluster. Argo CD then reconciles the merged CRs onto EKS.

Tune for the trading workload, because defaults are not for you. Pin brokers to network-optimized, local-NVMe instances (an i-family or comparable) so the commit log writes to local disk and the NIC has headroom for replication traffic. Set the JVM heap modestly (Kafka leans on the OS page cache, not a huge heap) with G1 pause targets tuned low. Increase num.network.threads and num.io.threads for the fan-out, and align producers and the partition leaders into the same AZ so the hot write path never crosses a zone. Use follower fetching (the RackAwareReplicaSelector above) so consumers read from an in-zone replica and avoid the cross-AZ read hop and its dollar cost. These are the knobs a managed service hides and the reason the firm left.

Configure tiered storage explicitly per topic. Enable remote storage on the durable topics and set a short local retention with a long remote one, so hot data stays on NVMe and cold data lands in S3:

remote.storage.enable=true
local.retention.ms=10800000        # 3 hours hot on local NVMe
retention.ms=220903200000          # ~7 years total, the tail living in S3

Enterprise considerations

Security & Zero Trust. The platform is Zero Trust by construction inside the cluster: mTLS authenticates and encrypts every client-broker session, and Kafka ACLs declared through KafkaUser resources enforce least privilege per service — the market-data gateway can Write only to the quote topics, a strategy engine can Read only the feeds it subscribes to, and everything else is denied by default. Layer on top: (a) HashiCorp Vault holds the few real secrets — the MirrorMaker peer-cluster credentials, third-party feed tokens, signing keys — leased dynamically and injected by the Vault Agent sidecar with IRSA-backed auth, so nothing sensitive sits in a plain Kubernetes Secret; (b) Wiz runs continuous CSPM across EKS and S3 and attack-path analysis, alerting the moment the tiered-storage bucket drifts toward public exposure or a node’s security group widens, with Wiz Code as the shift-left check in CI; © CrowdStrike Falcon sensors run as a DaemonSet on every broker and operator node for runtime threat detection, feeding the firm’s SOC; (d) human access to Grafana, the EKS console, and the platform tooling federates through Okta (brokered to Entra ID where Azure-side tooling needs a native token), so engineers authenticate once under conditional access rather than sharing a kubeconfig. An SLO breach or a Falcon detection auto-raises a ServiceNow incident so there is a ticket and an audit trail, not just a log line.

Cost optimization. Self-managed is not automatically cheaper; engineer it to be.

The inter-AZ alignment lever is the one teams forget: a chatty cross-zone consumer can quietly make data-transfer the largest bill on the platform, and follower fetching plus AZ-pinned producers is what keeps it down.

Scalability. Each tier scales independently. Brokers scale out by raising replicas on the KafkaNodePool; Strimzi provisions the new broker and you rebalance partitions onto it with Cruise Control (which Strimzi integrates) rather than a hand-built reassignment plan. Topic throughput scales with partition count — size it for the consumer parallelism the strategy engines need, since a partition is the unit of consumer concurrency. EKS node groups scale via the cluster autoscaler. The natural ceiling on a single cluster is metadata and replication overhead, which is why a firm at real scale shards by domain (a market-data cluster separate from the order-journal cluster) rather than one mega-cluster.

Failure modes, and what each one looks like. Name them before they page you.

• An AZ goes down mid-session — every partition led from that zone needs a new leader. Because replicas are rack-spread, a surviving in-sync replica in another AZ is elected in under a second; the desk sees a sub-second blip, not a halt. Mitigation: RF=3 across three AZs with min.insync.replicas=2, verified by periodic AZ-failure game days.
• A broker dies — Strimzi reschedules it onto a healthy node, and because tiered storage keeps the local footprint small, re-replication of the hot window finishes in minutes, not hours. Mitigation: tiered storage and Cruise Control-driven rebalancing.
• A cert silently expires — the classic self-managed outage. Mitigation: Strimzi’s CA auto-renews cluster and client certs ahead of expiry; an alert fires if renewal lags.
• Under-replicated partitions climb — replication is falling behind, the early warning of a saturated broker or network. Mitigation: alert on UnderReplicatedPartitions > 0 and on ISR shrink, long before it becomes data risk.
• Consumer lag blows out at the open — a strategy engine cannot keep up, so it acts on stale prices. Mitigation: alert on records-lag-max per consumer group against an SLO, scale partitions and consumer instances, and pre-warm before the open.
• Regional outage — see DR below.

Reliability & DR (RTO/RPO). Decide the numbers per tier. Within the region, the rack-aware three-AZ layout gives zero data loss and sub-second recovery for an AZ failure. For a full regional outage, MirrorMaker 2 asynchronously replicates the durable topics (the order journal above all) to a paired AWS region, syncing both the data and the consumer-group offsets so a failed-over consumer resumes near where it stopped. Async replication means the cross-region RPO is the replication lag — typically seconds — and the RTO is how fast you repoint producers and consumers at the DR cluster, which a runbook plus DNS makes minutes. A pragmatic target for this platform: in-region RTO under one minute and RPO zero; cross-region RTO ~15 minutes and RPO seconds. The order journal’s S3-backed tiered storage is the durable backstop — even a lost cluster is rebuildable from object storage.

Observability and the SLO contract. This is where self-managing earns its keep, because the whole project is justified by proving latency. The Strimzi JMX exporter exposes every broker metric to Prometheus, and Grafana renders the dashboards the desk and CTO watch. The SLOs are explicit and alerted:

Dynatrace sits above the Kafka-native metrics with full-stack distributed tracing and Davis anomaly detection, correlating a producer-side latency spike to a node, a network event, or a GC pause so the two-person team gets a root cause, not just a red graph — and surfacing a regression on its own before it trips the SLO. A sustained SLO breach auto-raises a ServiceNow incident.

Governance. Pin the Kafka and Strimzi versions explicitly and promote upgrades through a staging cluster — Strimzi rolls a version change one broker at a time with controlled shutdown, but you still gate it. Keep every KafkaTopic, KafkaUser, and broker config in Git, reviewed and instantly revertable, with Argo CD as the single applier so the live cluster never drifts from the repo. Route cluster changes through a ServiceNow change approval for an audit trail, and rely on Wiz as the independent check that the posture (no public S3, mTLS on, ACLs enforced) is actually holding.

Explicit tradeoffs

Accept these or do not build it. Self-managing Kafka — even with Strimzi doing the heavy lifting — means you now own broker tuning, capacity planning, partition rebalancing, and the on-call for a stateful distributed system, which a managed service carried for you. Strimzi shrinks that load dramatically but does not erase it: someone still has to understand KRaft quorums, ISR dynamics, and why a partition went under-replicated. Tiered storage adds an S3 dependency and a small fetch-latency cliff for historical reads. The rack-aware, in-AZ-aligned topology that delivers the latency and the data-transfer savings is more design effort than a single-zone cluster, and getting follower fetching and producer placement right is fiddly. And running this on EKS means the platform team owns both Kubernetes and Kafka — two deep systems — which is only worth it because the operator makes the Kafka half tractable for a small team.

The alternatives, and when they win. If latency is not your product and you just need durable streaming, managed Kafka (MSK or Confluent Cloud) is the right default — less to operate, and you give up exactly the tuning this firm needed. If your workload is simple queue-and-fan-out rather than a high-throughput replayable log, a managed message queue (SQS/SNS, or a lighter broker) is simpler than any Kafka at all. If you need stream processing on top — windowed aggregations, joins — add Kafka Streams or Flink as consumers rather than pushing logic into the brokers. And if you are a small team without Kubernetes expertise, running Kafka on dedicated VMs with Ansible trades the EKS learning curve for a more manual operational model — viable, but it gives back much of the self-healing that made Strimzi worth choosing.

The shape of the win

For the trading desk, the payoff is not “we run our own Kafka.” It is that during the open and the close — the moments that decide the firm’s P&L — the produce-to-consume p99 sits under eight milliseconds on a Grafana panel the head of trading can see, an AZ can fail without a single missed fill, the order journal is provably durable and retained for the regulator, and a two-person platform team runs the whole thing because Strimzi handles the broker lifecycle, certificate rotation, and self-healing that used to demand a dedicated ops squad. That combination — managed-service-level operational cost with self-managed-level control over latency — is the one that justifies the build. Everything upstream — the rack-aware AZ spread, the KRaft controllers, the tiered storage to S3, the mTLS and ACLs, the Vault-held secrets, the Wiz posture scanning, the Dynatrace anomaly detection, the MirrorMaker DR — exists so the desk gets its fills and the CTO can prove the SLO is holding. The architecture here is the destination; start with a single durable cluster if you must, but a regulated, latency-critical trading feed is where self-managed Kafka on Kubernetes has to land.

Going deeper

The body above is the blueprint. This section is for the engineer who has to operate it — where the microseconds live, and the internals a two-person team eventually has to understand.

KRaft: controllers, brokers, and the quorum

Modern Kafka runs KRaft mode, and as of Kafka 4.0 that is the only mode — ZooKeeper has been removed entirely, and Strimzi dropped ZooKeeper support in version 0.46. Metadata that ZooKeeper used to hold (topics, partitions, leaders, ACLs, configs) now lives in an internal Kafka log, __cluster_metadata, managed by a controller quorum that agrees on changes using the Raft consensus protocol. A node’s job is set by its KafkaNodePool role: controller nodes run the metadata quorum; broker nodes carry the actual topic data; a node can hold both roles (combined mode), but for a latency platform you separate them so a metadata election never competes for CPU or page cache with the hot data path. You need an odd number of controllers (3 is standard) so the quorum can always reach majority; three across three AZs survives losing one AZ. Controllers hold little data but need a durable metadata log, so they take a small persistent volume:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: controllers
  labels:
    strimzi.io/cluster: trading-bus
spec:
  replicas: 3
  roles: [controller]              # dedicated KRaft quorum, no topic data
  storage:
    type: persistent-claim
    size: 20Gi
    class: gp3

Why this matters for latency: removing ZooKeeper removes a network hop and an entire failure domain from leader election, so when a broker dies the new-leader decision is faster and the “blip” the desk sees is shorter. It also shrinks the surface a small team has to secure and patch.

Low-latency tuning: where the microseconds go

Latency is a stack, and you tune every layer. Working from the wire inward:

• Rack awareness and local reads. broker.rack (set from the node’s topology.kubernetes.io/zone label) makes Kafka place partition replicas in different AZs for durability. Pair it with the RackAwareReplicaSelector (follower fetching) so a consumer reads from a replica in its own AZ — no cross-zone hop, no inter-AZ data-transfer bill. Producers are pinned to the same AZ as the leaders they write to, so the hot write path never crosses a zone either.

• Producer batching: acks, linger.ms, batch.size. A producer with linger.ms=0 sends the instant it can (lowest latency, smaller batches); a few milliseconds of linger trades latency for throughput by filling bigger batches. acks picks durability: acks=all waits for min.insync.replicas copies (used on the order journal), acks=1 waits only for the leader (the ephemeral market-data fan-out), acks=0 waits for nothing and can silently drop data. Compression (lz4/zstd) shrinks the bytes on the wire at a small CPU cost.

• Page cache and disk. Kafka does not cache messages in its JVM heap — it writes to the OS page cache and lets the kernel flush to disk, and consumers reading recent data are served straight from RAM. So the two things that matter are lots of free RAM for page cache and a fast commit-log disk: local NVMe instance storage (an i-family node) for single-digit-microsecond writes, not a network EBS volume. Leave vm.dirty_ratio/vm.swappiness sane so the kernel does not stall on a flush.

• JVM and GC. Give the broker a modest heap (Kafka lives in the page cache, not the heap — 6 GB is plenty even on a 32 GB node) and target short G1 pauses, because a 200 ms stop-the-world GC pause is a 200 ms latency spike on every partition that broker leads:

  spec:
    kafka:
      resources:
        requests:
          cpu: "8"
          memory: 32Gi
        limits:
          cpu: "8"
          memory: 32Gi          # requests == limits => Guaranteed QoS, enabling CPU pinning
      jvmOptions:
        -Xms: 6g
        -Xmx: 6g                # small, fixed heap; the rest of RAM is page cache
        -XX:
          MaxGCPauseMillis: 20
          InitiatingHeapOccupancyPercent: 35
  

• Dedicated nodes, CPU pinning, hugepages, network. Brokers get dedicated nodes (taints + tolerations, node affinity) so no noisy neighbor steals CPU or cache. With Guaranteed QoS (requests == limits) and the kubelet’s static CPU Manager policy, broker threads are pinned to exclusive cores, killing scheduler jitter. Hugepages cut TLB misses for the JVM. On the network side, choose network-optimized instances, keep producers/leaders in-zone, and (on bare metal) pin NIC interrupts. Each of these is a knob a managed service hides — and the reason the firm left.

The storage story: per-broker PVs and retention

Each broker owns its log on its own volume. Strimzi supports three storage types: ephemeral (node-local instance store — fastest, lost if the pod moves, used here for the hot NVMe commit log), persistent-claim (a PVC from a StorageClass — durable, survives reschedule), and jbod (“just a bunch of disks” — multiple volumes per broker so Kafka spreads partitions across spindles). The retention story is two-layered: retention.ms/retention.bytes bound how long/large a topic’s log grows before old segments are deleted, while tiered storage (below) lets the local retention stay tiny even when the total retention is years. Watch PV IOPS as closely as capacity — an under-provisioned disk (a small gp3 without enough provisioned IOPS) throttles the commit log and shows up as producer-latency spikes long before it shows up as “disk full.”

Rebalancing with Cruise Control

When you add a broker, existing partitions do not move to it automatically — you would otherwise hand-write a partition-reassignment plan and babysit it. Strimzi integrates Cruise Control, which models the cluster against a set of goals (rack-awareness, disk/CPU/network capacity, even replica distribution) and generates a balanced plan you approve. You express it as a KafkaRebalance resource; modes are full (rebalance everything), add-brokers (drain replicas onto new brokers after scale-up), remove-brokers (drain off brokers before scale-down), and remove-disks (move data between a broker’s own JBOD volumes):

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaRebalance
metadata:
  name: add-broker-7
  labels:
    strimzi.io/cluster: trading-bus
spec:
  mode: add-brokers
  brokers: [6]                   # move replicas onto the newly added broker id 6
  goals:
    - RackAwareGoal              # never violate AZ spread while balancing
    - DiskCapacityGoal
    - NetworkInboundCapacityGoal
    - ReplicaDistributionGoal

You apply it, inspect the optimizationResult in the resource status, then annotate it strimzi.io/rebalance: approve to execute. Strimzi can also auto-trigger a rebalance on scale-up/down so a KafkaNodePool replica change is self-completing.

Tiered storage in Strimzi

Kafka tiered storage (generally available since Kafka 3.9) splits a topic’s log into a hot local tier and a cold remote tier: closed log segments are offloaded to object storage while brokers keep only the recent window on fast local NVMe. Enable it per topic (remote.storage.enable=true, a short local.retention.ms, a long retention.ms, already shown above) and configure the pluggable RemoteStorageManager on the Kafka resource — Strimzi uses a type: custom block pointing at an implementation such as Aiven’s S3 plugin:

spec:
  kafka:
    tieredStorage:
      type: custom
      remoteStorageManager:
        className: io.aiven.kafka.tieredstorage.RemoteStorageManager
        classPath: /opt/kafka/plugins/tiered-storage/*
        config:
          storage.backend.class: io.aiven.kafka.tieredstorage.storage.s3.S3Storage
          storage.s3.bucket.name: trading-bus-tiered
          storage.s3.region: us-east-1

The payoff is fast broker recovery (a replaced broker re-replicates only the small hot window, minutes not hours) and cheap, effectively unlimited retention for the regulated journal. The cost is an S3 fetch for anyone reading far back in history — a non-issue for a real-time desk, fine for a compliance replay. Confirm the maturity of tiered storage for your exact Strimzi/Kafka versions before you lean on it for a regulated record.

Security: mTLS, SCRAM, and OAuth

The default here is mTLS — Strimzi runs its own certificate authority, issues each KafkaUser a client certificate, and auto-renews both cluster and client certs before expiry, which defuses the classic self-managed outage of a silently expired cert. Two alternatives exist when mTLS is awkward: SCRAM-SHA-512 (salted username/password, simpler for third-party clients that can’t manage certs) and OAuth (authentication.type: oauth), where clients present a bearer token from your identity provider (Okta/Entra) and the broker validates it — attractive when you already run OIDC and want Kafka access governed by the same tokens as everything else. Authorization is orthogonal: simple (Kafka ACLs), opa (Open Policy Agent), or keycloak/OAuth-based, all declared on the Kafka and KafkaUser resources.

High availability: rack, ISR, and min.insync.replicas

Durability is a two-number contract. Replication factor (here 3) is how many copies of each partition exist; min.insync.replicas (here 2) is how many must be caught up for an acks=all write to be accepted. The set of caught-up replicas is the ISR (in-sync replica set). With RF=3, min.insync.replicas=2, and rack-aware placement across three AZs, a full AZ can vanish and every partition still has an in-sync replica elsewhere — a sub-second leader election, zero data loss, no trading halt. The trap is setting min.insync.replicas equal to the replication factor (3): now losing any one replica blocks all writes, turning a durability setting into an availability outage. Watch UnderReplicatedPartitions — a non-zero value means the ISR has shrunk and you are one failure away from a stall.

Upgrades: rolling brokers one at a time

Strimzi upgrades Kafka one broker at a time with controlled shutdown: it drains leadership off a broker, restarts it on the new version, waits for it to rejoin the ISR, and only then moves to the next — so the cluster stays available throughout. A version change spans two fields, the Kafka version and the inter.broker.protocol.version/metadata.version, bumped in sequence so you can roll forward and, if needed, back. Pin both explicitly, promote through a staging cluster, and gate the change through your normal review — the operator makes the mechanics safe, but when to upgrade a regulated platform is still a human decision.

Practice challenges

Work these top to bottom; they escalate from “do you get the model” to “can you operate it under fire.” Try each before opening the solution.

1 — Beginner: name the resource. You need a new stream md.quotes.nasdaq with 48 partitions, kept for 6 hours, on the trading-bus cluster. Which Strimzi custom resource do you write, and which two spec fields set the partitions and the cluster it belongs to?

<details><summary>Show solution</summary>

A KafkaTopic. spec.partitions: 48 sets the partition count; the cluster is chosen by the label strimzi.io/cluster: trading-bus in metadata.labels (not a spec field). Retention is spec.config."retention.ms". Why: topics are declarative objects the Topic Operator reconciles; the strimzi.io/cluster label is how every Strimzi sub-resource says which Kafka it belongs to. </details>

2 — Beginner: pick the acks. The market-data fan-out re-sends every quote on the next tick, so a dropped message is harmless; the order journal is a regulated record. What acks do you set on each producer, and why is acks=0 wrong for the journal?

<details><summary>Show solution</summary>

Fan-out: acks=1 (or even acks=0 if you truly never care) — shave the acknowledgement wait. Journal: acks=all — the write is only acknowledged once min.insync.replicas copies have it. acks=0 means the producer never waits for any confirmation, so a broker crash between send and persist loses the write silently — unacceptable for a record you must retain for the regulator. Why: acks is the durability/latency dial, and you set it per topic against that topic’s actual loss tolerance. </details>

3 — Intermediate: the min.insync.replicas trap. A teammate sets RF=3 and min.insync.replicas=3 “for maximum safety.” During a routine node replacement, all producers on that topic start failing with NOT_ENOUGH_REPLICAS. Explain and fix.

<details><summary>Show solution</summary>

With min.insync.replicas=3 and RF=3, all three replicas must be in-sync for an acks=all write to succeed — so losing even one (a rolling restart, a node replacement) drops the ISR to 2 and blocks every write. Fix: set min.insync.replicas=2. That still requires a second AZ to have the data (durable) but tolerates one replica being briefly down (available). Why: the durable-and-available sweet spot is min.insync.replicas = RF − 1; setting it equal to RF converts a durability knob into an availability outage. </details>

4 — Intermediate: kill the cross-AZ bill. Your consumers are spread across three AZs and the AWS data-transfer line item is the biggest cost on the platform. Two config changes cut it without moving any pods. Name them.

<details><summary>Show solution</summary>

(1) Set broker.rack from topology.kubernetes.io/zone (the rack.topologyKey in the Kafka CR) so replicas are AZ-spread and each consumer has a local replica. (2) Set replica.selector.class to RackAwareReplicaSelector so consumers use follower fetching — reading from the in-zone replica instead of always the (possibly cross-AZ) leader. Together they keep the read path in-zone. Why: inter-AZ data transfer is billed per GB; a chatty cross-zone consumer quietly dominates the bill, and rack-aware follower fetching is the fix. </details>

5 — Advanced: scale out safely. You add a seventh broker (id 6) to absorb open-auction volume, but throughput doesn’t improve and the new broker sits nearly idle. What’s happening, and what do you apply — without hand-writing a reassignment plan or breaking AZ spread?

<details><summary>Show solution</summary>

Adding a broker does not move existing partitions; the new broker only receives newly created partitions, so existing hot topics ignore it. Apply a KafkaRebalance in add-brokers mode targeting broker 6, and include RackAwareGoal so the rebalance never places two replicas of a partition in the same AZ:

spec:
  mode: add-brokers
  brokers: [6]
  goals: [RackAwareGoal, DiskCapacityGoal, ReplicaDistributionGoal]

Inspect status.optimizationResult, then annotate strimzi.io/rebalance: approve. Why: Cruise Control generates a goal-aware plan (Strimzi even auto-triggers it on scale-up), so you never risk a hand-built reassignment that violates rack awareness. </details>

6 — Advanced: diagnose the latency spike. Grafana shows producer p99 on the order journal jumping from 3 ms to 40 ms every ~30 seconds on one broker, while the others are flat. UnderReplicatedPartitions is 0 and the disk is fine. What’s the most likely cause and the fix?

<details><summary>Show solution</summary>

A periodic single-broker spike with healthy replication and disk is the signature of a stop-the-world GC pause — the JVM freezes for tens of milliseconds while it collects, and every partition that broker leads stalls for that window. Fix: shrink and fix the heap (-Xms/-Xmx equal, ~6 GB — Kafka wants RAM in the page cache, not the heap), set a low MaxGCPauseMillis G1 target, and give the broker Guaranteed QoS on a dedicated, CPU-pinned node so it isn’t also fighting a noisy neighbor. Why: at a single-digit-ms SLO, GC pauses and CPU contention — not Kafka itself — are the usual latency villains, which is exactly why the brokers are tuned and isolated. </details>

Common beginner mistakes

These are misconceptions, not symptom-to-fix entries — the wrong mental model, then the right one.

“Kafka is just pods; I’ll schedule the brokers on the shared node pool with everything else.” On a latency SLA, a broker sharing a node with batch jobs, sidecars, and someone’s CI runner will have its CPU stolen and its page cache evicted at the worst moment — the open. Right model: brokers get dedicated, tainted nodes with Guaranteed QoS and CPU pinning, so nothing competes with the hot path. Shared nodes are fine for a throughput-only Kafka; they are disqualifying for a latency one.

“Three replicas means I’m safe — I don’t need rack awareness.” Replication factor 3 with all three replicas in the same AZ means one AZ failure takes all three copies at once — you have three eggs in one basket. Right model: broker.rack from the zone label plus rack-aware placement guarantees the three replicas land in three different AZs, so losing a zone still leaves an in-sync copy. Replication protects against a broker dying; rack awareness protects against a zone dying, and you need both.

“min.insync.replicas should equal the replication factor for maximum durability.” Setting min.insync.replicas=RF means the loss of any single replica — including a routine rolling restart — blocks every write. Right model: min.insync.replicas = RF − 1 (2 with RF=3) is the durable-and-available sweet spot; it still requires a second copy before acknowledging, but tolerates one replica being briefly down. Maximum durability that blocks all writes is just an outage.

“Latency is Kafka’s job; I don’t need to think about the JVM.” Kafka runs on the JVM, and an untuned JVM with a giant heap produces long stop-the-world GC pauses that appear as periodic latency spikes on whichever broker is collecting. Right model: a small fixed heap (Kafka caches in the OS page cache, not the heap), a low G1 pause target, and enough free RAM for the page cache. The heap is for Kafka’s bookkeeping; the data wants RAM the kernel controls.

“Any StorageClass will do — a disk is a disk.” A commit log on a network volume or a low-IOPS gp3 will throttle under the open’s write burst and surface as producer-latency spikes long before it ever says “disk full.” Right model: latency-sensitive brokers use local NVMe (instance store) for the hot log, and where you use PVCs you provision IOPS and throughput, not just capacity. Watch disk latency and IOPS, not just free space.

“acks=0 is fine, it’s faster.” acks=0 means the producer never waits for any acknowledgement, so a broker crash between send and persist loses the data with no error anywhere — the producer thinks it succeeded. Right model: acks=0/acks=1 are acceptable only for genuinely disposable streams (a re-sent market tick); anything you must not lose — above all a regulated order journal — is acks=all with min.insync.replicas=2. Speed you can’t audit isn’t speed, it’s silent data loss.

Glossary

• Kafka — a distributed, durable, replayable event log: producers append messages to topics, many consumers read them independently at their own offset.
• Strimzi — the CNCF Kubernetes operator for Apache Kafka; runs Kafka, its topics, and its users as custom resources and handles the whole lifecycle.
• Operator — software installed into a cluster that encodes an application’s operational knowledge (a CRD plus a controller) and acts as an automated, always-on administrator.
• CRD / custom resource — a CustomResourceDefinition teaches the Kubernetes API a new kind (e.g. KafkaTopic); a custom resource is one instance of it.
• Cluster Operator — Strimzi’s main controller pod; runs the reconcile loop that turns your Kafka/KafkaNodePool resources into a running cluster.
• Entity Operator / Topic Operator / User Operator — the companion Strimzi deploys to reconcile KafkaTopic and KafkaUser objects into real topics, users, ACLs, and client certificates.
• Broker — a Kafka server that stores partition data, serves producers/consumers, and can be the leader for some partitions.
• Controller (KRaft) — a node in the metadata quorum that manages cluster state via Raft consensus; replaces ZooKeeper.
• KRaft — Kafka’s built-in metadata mode (Kafka Raft); the only mode as of Kafka 4.0, with ZooKeeper removed.
• ZooKeeper — the external coordination service Kafka used before KRaft; a second system to run and secure, now gone.
• KafkaNodePool — the Strimzi resource that defines a group of nodes, their roles (broker/controller), replica count, and storage.
• StrimziPodSet — Strimzi’s own pod controller (superseded StatefulSets) that gives brokers stable identity and sticky storage with fine-grained rolling control.
• StatefulSet — the stock Kubernetes controller for identity-stable, disk-sticky pods; the model StrimziPodSet is built on the ideas of.
• Topic — a named stream of messages, split into partitions.
• Partition — the unit of parallelism and ordering within a topic; each partition is an ordered log replicated across brokers.
• Replication factor — how many copies of each partition exist across brokers (here 3).
• Replica / leader / follower — the copies of a partition; one leader handles reads/writes, followers replicate it and can be read via follower fetching.
• ISR (in-sync replicas) — the set of replicas currently caught up to the leader; an acks=all write needs min.insync.replicas of them.
• min.insync.replicas — the minimum in-sync copies required to accept an acks=all write; set to RF − 1 for durable-and-available.
• acks — the producer’s durability dial: 0 (no wait), 1 (leader only), all (a quorum of in-sync replicas).
• linger.ms / batch.size — producer batching knobs; a little linger fills bigger batches (throughput) at the cost of latency.
• Consumer group / consumer lag / offset — a set of cooperating consumers / how far behind the latest message they are / the position each has read to.
• Rack awareness (broker.rack) — labelling brokers by failure domain (AWS AZ) so Kafka spreads a partition’s replicas across zones.
• Follower fetching (RackAwareReplicaSelector) — letting a consumer read from an in-zone replica instead of the leader, avoiding the cross-AZ hop and its cost.
• Listener — a named Kafka port with a type (internal/loadbalancer/…), TLS flag, and auth mode, declared on the Kafka resource.
• mTLS — mutual TLS: both client and broker present certificates; Strimzi issues and auto-renews them per KafkaUser.
• SCRAM-SHA-512 / OAuth — alternative Kafka auth modes: salted username/password / bearer tokens from an identity provider (Okta/Entra).
• ACL / KafkaUser — an access-control rule (who may Read/Write/Describe which topic) / the resource that declares a client’s identity and its ACLs.
• JBOD — “just a bunch of disks”: multiple volumes per broker so Kafka spreads partitions across them.
• Tiered storage — keeping only recent (hot) log segments on local disk and offloading closed segments to object storage (S3) via a pluggable RemoteStorageManager.
• Cruise Control / KafkaRebalance — the component that generates goal-aware partition-balancing plans / the resource that requests one (full/add-brokers/remove-brokers).
• MirrorMaker 2 / KafkaMirrorMaker2 — cross-cluster replication of topics (and offsets/ACLs); here used to replicate durable topics to a DR region.
• Page cache — the OS’s in-RAM file cache; Kafka serves recent reads from it, which is why brokers want lots of free RAM and a small JVM heap.
• NVMe / local instance storage — fast node-local SSD used for the hot commit log; lowest write latency, but not durable across a pod move.
• G1 GC / MaxGCPauseMillis — the JVM garbage collector and its pause-time target; long pauses become latency spikes, so the target is kept low.
• CPU pinning / Guaranteed QoS / hugepages — kubelet features that give a broker exclusive cores (requests==limits) and large memory pages, removing scheduler jitter.
• UnderReplicatedPartitions — the metric that counts partitions whose ISR has shrunk below target; the early warning of a saturated broker or network.
• p99 / tail latency — the 99th-percentile latency; the slow tail the trading desk actually feels, and the number the whole platform is tuned to hold down.
• SLO / RPO / RTO — the latency target you promise / how much data a failure may lose / how long recovery may take.