AWS Enterprise Architecture: Production Microservices on EKS

In a nutshell

Picture an airport, not a single runway. A monolith is one runway: every plane (feature) waits in one line, and if that runway closes, nothing flies. This reference architecture is a whole airport for software — dozens of independent gates (microservices), a control tower deciding what runs (Kubernetes), a ground crew that adds or removes gates as flights surge and empty (Karpenter), passport control on every internal walkway (a service mesh with encryption), and one iron rule: nothing gets built or changed except through an approved blueprint checked into version control (GitOps).

Here is the problem it solves. Once you have more than a handful of services and more than one or two teams, “just deploy to Kubernetes” stops being enough. You suddenly need answers to who is allowed to run what, how services talk to each other safely, how the bill tracks real traffic instead of your worst-ever day, and how you rebuild everything if an AWS region disappears. This design bolts those answers onto Amazon EKS (managed Kubernetes) using five off-the-shelf pieces, so a small platform team can safely serve many product squads.

The one mental model to hold on to is two loops around one cluster. The request loop is what a user’s click does: edge → mesh → service → service → AWS data store. The change loop is how new code gets there: Git commit → build and sign the image → Argo CD reconciles the cluster to match Git. The cluster in the middle is deliberately disposable — it is just a live copy of whatever Git says should exist. Almost everything advanced in this lesson is really about making one of those two loops safer, cheaper, or faster.

You do not need to memorise every acronym on the first read. Treat this as a map: the “In a nutshell” and “business scenario” tell you why, the diagram and component table tell you what, and “Going deeper” tells you how it actually works underneath.

Level: Advanced (but written to be beginner-readable) · Time: ~38 min · Assumes you know: roughly what a pod, Deployment and Service are, and what IAM and a load balancer do. · After this you can: read the reference diagram and narrate every hop; justify EKS + Karpenter + mesh + IRSA + GitOps to a skeptical team; recognise when the design is overkill; size an IP plan; and critique a proposed EKS platform for the five classic failure modes.

Most “run microservices on Kubernetes” guides stop at kubectl apply against a managed cluster and call it production. The gap between that and an audited, multi-team, multi-AZ platform that an enterprise will actually trust with revenue traffic is enormous: who can pull which image, which pod can assume which IAM role, how 200 microservices talk to each other with mTLS and retries, how the cluster scales from 12 nodes at 3 a.m. to 140 nodes during a sale without paging anyone, and how every change reaches the cluster through a reviewed Git commit rather than someone’s laptop.

This article is a concrete, opinionated reference architecture for that platform on Amazon EKS, built from five load-bearing pieces: EKS as the compute substrate, ECR as the signed image supply chain, IRSA as the pod-to-AWS identity bridge, App Mesh for service-to-service traffic management, Karpenter for just-in-time node provisioning, and GitOps (Argo CD) as the only path into the cluster. It is designed to be cloned by a 30-engineer startup and to keep working when that startup becomes a 2,000-engineer org running 18 product teams on one platform.

The flow that makes this production-grade, end to end:

1. A squad pushes to Git — the only way to change what runs. No kubectl apply from laptops.
2. CI builds and signs the image, pushing it to ECR; only signed images are admitted.
3. Argo CD continuously reconciles the cluster to the Git desired state (drift is auto-corrected).
4. The Kubernetes API schedules workloads; admission control verifies image signatures and IRSA bindings.
5. Karpenter provisions just-in-time EC2 capacity across three AZs and bin-packs to cut idle cost.
6. Pods assume scoped AWS permissions through IRSA — no long-lived keys — while App Mesh enforces mTLS, retries and circuit-breaking on all east-west traffic.

The business scenario

Picture a mid-market commerce and logistics company — call the pattern “the platform team’s dilemma.” You’ve outgrown a monolith on a handful of EC2 instances. Forty-plus services now exist: checkout, pricing, inventory, fulfilment, notifications, a recommendations API, fraud scoring, and a long tail of internal tools. Three to twelve product squads ship them, each wanting to deploy several times a day without filing a ticket with a central ops team.

The pain that forces an architecture decision is rarely “we need Kubernetes.” It’s a cluster of concrete failures:

• Deploys are scary and serial. A bad config push takes down checkout because there’s no canary, no automatic rollback, and no mesh-level traffic shifting. Changes are applied imperatively, so nobody can answer “what is actually running in prod right now?” without SSH-ing in.
• Capacity is either wasteful or fragile. Static Auto Scaling Groups are sized for peak, so you pay for idle c5.4xlarge instances all night; or they’re sized for average and fall over during a flash sale. Bin-packing across instance types is done by hand.
• Identity is a mess of long-lived keys. Services hold AWS access keys in environment variables to reach S3, DynamoDB, SQS and Secrets Manager. Those keys leak, never rotate, and grant far more than any one service needs. (This is exactly the class of incident this org has been burned by before — leaked long-lived credentials in source control — so eliminating static keys is a hard requirement, not a nice-to-have.)
• East-west traffic is a black box. When the pricing service gets slow, it silently takes checkout down with it. There are no per-route timeouts, no retries with budgets, no circuit breaking, and no consistent mTLS between services. Security can’t prove that service-to-service traffic is encrypted.
• Supply chain is unverified. Images are pulled :latest from a public registry, unsigned and unscanned. There’s no answer to “was this image built by our pipeline and not tampered with?”

The business goal is a self-service internal platform: squads own their services and deploy via Git; a small platform team owns the substrate, guardrails, and golden paths. The non-functional targets are typical for this size of company: 99.95% availability for tier-1 services, p99 latency budgets enforced at the mesh, RTO ≈ 30 minutes / RPO ≈ 5 minutes for the cluster control plane and workloads, costs that scale roughly linearly with traffic (not with peak), and an audit trail that satisfies SOC 2 and PCI-DSS scope for the checkout path. EKS plus the five focus components is precisely the combination that hits those targets without a 40-person platform org.

Architecture overview

The end-to-end request path and the end-to-end change path are two different flows, and a good EKS platform treats them as equally first-class. Picture the diagram in two layers stacked on the same VPC.

The runtime (request) path — north-south then east-west. A client request hits Amazon Route 53, which resolves to an AWS WAF-protected Application Load Balancer (or an NLB for gRPC/TLS passthrough). The ALB is provisioned and reconciled by the AWS Load Balancer Controller running in-cluster, driven by Ingress/Gateway objects — so the edge is declared in Git, not clicked in the console. Traffic lands on the App Mesh ingress (Envoy) gateway, the single managed entry point into the mesh. From there every hop is east-west traffic inside the mesh: the ingress gateway routes to a VirtualService (say checkout), which resolves through a VirtualRouter to one or more VirtualNodes backed by Kubernetes Services and pods. Each pod runs an Envoy sidecar injected by the App Mesh controller; sidecars carry mTLS (certs from AWS Private CA via the cert-manager integration), per-route timeouts, retries with budgets, and outlier-detection circuit breaking. When checkout calls pricing, that call is another mesh hop with the same guarantees. Pods reach AWS data services — DynamoDB, Aurora, S3, SQS, Secrets Manager — using IRSA (IAM Roles for Service Accounts): the pod’s ServiceAccount is annotated with an IAM role ARN, the pod gets a projected OIDC token, and the AWS SDK exchanges it via STS for short-lived, scoped credentials. No static AWS keys exist anywhere in the cluster.

The compute substrate. All of this runs on an EKS cluster (Kubernetes control plane managed by AWS, spread across three AZs) with worker capacity supplied two ways. A tiny managed node group (2–3 on-demand nodes) hosts the things that must always be up and must not be churned by the autoscaler: Argo CD, the Karpenter controller itself, CoreDNS, and the App Mesh/LB/EBS controllers. Everything else — all application pods — runs on nodes that Karpenter provisions just-in-time. When pods go Pending, Karpenter reads their CPU/memory/affinity/topology requirements, picks the cheapest viable instance types (mixing Spot and On-Demand, Graviton and x86), launches the node in seconds, and consolidates (terminates and repacks) underused nodes minutes later. This is what makes capacity track traffic instead of peak.

The change (GitOps) path. Developers never run kubectl apply against prod. A squad merges a PR to an app repo; CI builds a container image, runs SCA/SBOM generation, pushes to ECR, where the image is scanned (Amazon Inspector / ECR enhanced scanning) and signed. CI then bumps an image digest in a config repo (Helm values or Kustomize overlay). Argo CD, watching that config repo, detects drift and reconciles the desired state into the cluster — applying Deployments, VirtualService/VirtualRouter mesh routes, HPAs, and NetworkPolicies. Progressive delivery (Argo Rollouts) shifts traffic 5% → 25% → 100% by reweighting the mesh VirtualRouter, watching golden-signal metrics, and auto-rolling-back on SLO breach. Git is the single source of truth; the cluster is a cache of Git.

So the picture is: Route 53 → WAF → ALB (LB Controller) → App Mesh ingress → Envoy-sidecar mesh of services on Karpenter-provisioned EKS nodes → AWS data services via IRSA, with ECR + Argo CD forming the supply chain and reconciliation loop that feeds the whole thing from Git.

Component breakdown

A few of these choices deserve a sentence of “why.” Private API endpoint + OIDC is the foundation that makes IRSA and a no-public-control-plane posture possible at once. Karpenter over Cluster Autoscaler matters because Karpenter chooses instance types per-pod and consolidates aggressively, which is where the real cost savings live. Mesh ingress gateway as the single entry keeps all routing/resilience policy in one mental model instead of split between the ALB and the mesh. And separate config repo is what lets Argo CD’s auto-heal be safe: prod state is whatever is in config-repo@main, full stop.

Implementation guidance

Provision in layers with Terraform, then hand the cluster to GitOps. A clean separation is: Terraform owns everything up to and including a bootstrapped cluster with its core add-ons; Argo CD owns everything inside the cluster from then on. Mixing the two (Terraform managing app Deployments) is the classic anti-pattern that causes drift fights.

Layer 1 — network & cluster (Terraform). Use the community modules as a backbone: terraform-aws-modules/vpc/aws for a 3-AZ VPC with private/public/intra subnets and the required kubernetes.io/role/internal-elb + karpenter.sh/discovery subnet tags; terraform-aws-modules/eks/aws for the cluster, the small managed node group, the OIDC provider, EKS access entries, and managed add-ons. Pin the Kubernetes version and turn on the audit/authenticator control-plane logs here.

Layer 2 — IAM & supply chain (Terraform). Create the Karpenter controller IRSA role and node IAM role/instance profile, the AWS Load Balancer Controller and EBS CSI IRSA roles, the ECR repositories with scan-on-push + immutability + lifecycle + replication, and AWS Private CA for mesh mTLS. Generate per-service IRSA roles with a small module that takes (namespace, serviceaccount, policy_json) and emits a role whose trust policy hard-codes the OIDC sub — this is the security crux:

# Trust policy condition that makes IRSA least-privilege:
# only THIS serviceaccount in THIS namespace can assume the role.
StringEquals = {
  "${oidc}:sub" = "system:serviceaccount:checkout:checkout-api"
  "${oidc}:aud" = "sts.amazonaws.com"
}

Layer 3 — in-cluster platform (GitOps, bootstrapped once). Terraform installs only the Argo CD Helm release and a single “root” Application; from there Argo CD installs Karpenter, the App Mesh controller + CRDs, the LB controller, cert-manager, ADOT, Kyverno, and the per-team ApplicationSets. Use sync waves so CRDs and the mesh control plane land before any workload that references a VirtualService. Karpenter NodePool/EC2NodeClass and App Mesh objects are plain manifests in the config repo.

Networking & identity wiring — the load-bearing details.

• VPC CNI in prefix-delegation mode to raise pod density per node (so Karpenter’s bin-packing isn’t capped by ENI limits); enable Network Policy support (VPC CNI or Cilium) and ship NetworkPolicy objects via Argo for default-deny east-west, layered under the mesh.
• VPC endpoints (Interface for ECR API/DKR, STS, Secrets Manager, CloudWatch, App Mesh/Envoy management; Gateway for S3/DynamoDB) so pulls, STS AssumeRoleWithWebIdentity, and AWS API calls never traverse a NAT gateway — this cuts NAT data-processing cost and shrinks the egress attack surface.
• IRSA end-to-end: ServiceAccount → role-arn annotation → projected token → SDK exchanges via STS. Validate with a throwaway pod that runs aws sts get-caller-identity and confirms it returns the service’s role, not the node role. For new clusters consider EKS Pod Identity (no per-cluster OIDC trust plumbing, easier cross-account); the manifest changes but the “no static keys” principle is identical.
• App Mesh sidecar injection is namespace-labelled; the init container programs iptables to redirect traffic through Envoy. Wire AWS Private CA + cert-manager so every VirtualNode gets a workload cert and mTLS is STRICT.

CI/CD shape. App repo CI: build → unit/integration → SBOM (Syft) → push to ECR → Inspector scan gate → cosign sign. A separate job (or Argo CD Image Updater) writes the new digest into the config repo behind a PR. Argo CD syncs; Argo Rollouts runs the canary by reweighting the mesh VirtualRouter and watching AMP metrics, auto-aborting on SLO breach. Kyverno at admission refuses any image that isn’t from your ECR and signed — closing the loop so even a manual kubectl can’t run an unverified image.

(On other IaC: Bicep and Deployment Manager are Azure/GCP-native and don’t target AWS — for this stack Terraform or AWS CDK/CloudFormation are the right tools; the layering above maps cleanly onto CDK constructs if you prefer typed IaC.)

Enterprise considerations

Security & Zero Trust. The architecture is built to assume breach at every layer. Identity: IRSA/Pod Identity means zero long-lived AWS keys; each pod gets least-privilege, short-lived STS credentials scoped to one ServiceAccount — leaked-credentials-in-git simply cannot happen for AWS access. Network: private API endpoint, default-deny NetworkPolicy, and mesh mTLS (STRICT) so all east-west traffic is mutually authenticated and encrypted — security can prove it. Supply chain: ECR enhanced scanning + cosign signatures + a Kyverno admission gate enforce “only our pipeline’s signed, unexpired images run here.” Edge: WAF on the ALB, TLS via ACM, and the mesh ingress gateway as the only door. Secrets: pulled at runtime from Secrets Manager via IRSA (or the Secrets Store CSI driver), never baked into images or env in Git. Map these to PCI-DSS scope by isolating the checkout namespace (dedicated NodePool taint, tighter NetworkPolicy, separate IRSA boundary) so the cardholder path is a small, auditable blast radius.

Cost optimization. This is where Karpenter earns its place. Right-sizing per pod: Karpenter picks the cheapest instance that fits, mixing types instead of one ASG shape. Spot for stateless: run the bulk of stateless services on Spot with On-Demand fallback via capacity-type requirements; Spot at ~70% off is the single biggest lever. Graviton: allow arm64 in NodePools — typically ~20% better price/performance for services with multi-arch images. Consolidation: WhenEmptyOrUnderutilized continuously repacks and terminates waste, so the cluster shrinks at night automatically. No-NAT AWS traffic: VPC endpoints remove NAT data-processing charges for ECR/STS/S3/DynamoDB. ECR lifecycle policies stop image storage from creeping. Track it all with Kubecost/OpenCost + cost-allocation tags per namespace so each squad sees its bill. Net effect: spend tracks traffic, and the 3-a.m. cluster is a fraction of the peak cluster.

Scalability. Three independent axes scale cleanly: pods (HPA on CPU/custom AMP metrics, or KEDA on SQS depth/event lag), nodes (Karpenter, seconds to provision), and traffic shaping (mesh VirtualRouter weights + retries absorb partial failures). The control plane is AWS-managed and scales itself. The ceiling you actually hit first is usually pod IP density (solved by prefix delegation) or a noisy-neighbour data service (solved by per-service throttling at the mesh).

Reliability & DR (RTO ≈ 30 min / RPO ≈ 5 min). Multi-AZ is table stakes: 3-AZ node spread via topology-spread constraints, PodDisruptionBudgets, and Karpenter disruption budgets so consolidation/Spot reclaims never take down a quorum. For regional DR, the cluster is rebuildable from Git in minutes — Terraform recreates the cluster, Argo CD reconciles every workload from the config repo — which is the whole point of GitOps. ECR cross-region replication ensures images exist in the DR region; data RPO comes from the data services (Aurora cross-region replicas / DynamoDB global tables / S3 CRR), not the cluster. So RTO is dominated by Terraform cluster-create + Argo sync (≈30 min, faster with a warm standby cluster), and RPO is whatever the replicated data tier gives you (≈5 min or better with global tables).

Observability. Golden signals from ADOT/OpenTelemetry → Amazon Managed Prometheus, dashboards in Amazon Managed Grafana, container/node metrics from CloudWatch Container Insights. Envoy emits per-route latency/error/retry/circuit-breaker stats and X-Ray spans, so a slow pricing call is visible before it cascades. Logs via FireLens → CloudWatch/OpenSearch. SLOs (availability, p99) are defined as Prometheus recording rules and are the same signals Argo Rollouts uses to gate canaries — one source of truth for “healthy.”

Governance. Argo CD RBAC + SSO scopes each team to its namespaces; Kyverno policies enforce required labels, resource limits, signed images, no-:latest, no-privileged. EKS audit logs + CloudTrail give a full who-did-what trail. Because every change is a reviewed Git commit, change management and the audit story are the same artifact.

Reference enterprise example

Meridian Freight is a (fictional) logistics-and-parcel marketplace: shippers post loads, carriers bid, the platform handles pricing, matching, tracking and settlement. They run 52 microservices across 9 squads, ~1.8 million API calls/hour at baseline with 5–7x peaks during weekday morning dispatch and quarter-end settlement runs. They came from a Rails monolith plus a sprawl of EC2 ASGs, and two incidents pushed the migration: a flash-sale-style dispatch spike that exhausted statically-sized capacity and dropped 8% of bookings, and a near-miss where an AWS access key for the tracking service was found committed in a config repo (matching this org’s prior leaked-credentials scar tissue — hence the hard “no static keys” mandate).

What they built. One EKS cluster per environment (dev/stage/prod), 3-AZ, private API endpoint. A 3-node On-Demand managed group hosts Argo CD, Karpenter, and the controllers; everything else is Karpenter-provisioned. NodePools allow c/m/r families on both Graviton and x86, Spot-first with On-Demand fallback, except a tainted payments NodePool that’s On-Demand-only and PCI-isolated for settlement. App Mesh fronts all 52 services with STRICT mTLS (AWS Private CA), per-route timeouts, retry budgets, and outlier detection; the matching service — historically the cascade culprit — now has a 250 ms timeout and a 3-attempt retry budget so a slow matcher fails fast instead of taking dispatch down. ECR holds all images with Inspector scanning, immutable tags, cosign signatures, and a Kyverno gate. Every service has a least-privilege IRSA role: tracking can read/write exactly two DynamoDB tables and one S3 prefix — nothing else — so the previously leaked key pattern is structurally impossible. Nine ApplicationSets in Argo CD onboard the squads; canaries shift 5/25/100% via mesh weights with auto-rollback on a p99 or error-rate breach.

The numbers and the outcome.

The headline outcome wasn’t just the ~44% compute saving or the move from 5 to 140 deploys a week — it was that the platform team shrank its toil: capacity, rollbacks, and credential rotation became properties of the system rather than human chores, and the settlement path became a small, provable PCI island instead of an audit nightmare.

When to use it

Use this architecture when you have a genuine multi-team, multi-service platform (roughly 10+ services and/or 3+ squads) that needs self-service deploys, fine-grained pod identity, real east-west resilience, and traffic-tracking cost. It shines precisely where static ASGs and imperative kubectl break down: spiky load, many small teams, strict identity/audit requirements, and a need to rebuild the world from Git.

Trade-offs and anti-patterns to avoid.

• Don’t reach for it too early. For 3–5 services and one team, this is over-engineered. AWS App Runner, ECS on Fargate, or EKS Auto Mode (which bundles Karpenter-style provisioning and core add-ons with far less to operate) will get you to production faster with a fraction of the cognitive load. Adopt the mesh and GitOps machinery when team count — not service count — demands it.
• Don’t let Terraform and Argo CD both own workloads. Terraform up to the bootstrapped cluster, GitOps inside. Overlapping ownership = perpetual drift.
• Don’t run a mesh you don’t need. A sidecar per pod has CPU/memory/latency overhead and real operational weight. If you only need mTLS and basic policy, VPC CNI NetworkPolicy + ALB may be enough; add the mesh when per-route retries/timeouts/circuit-breaking and traffic-shifting canaries are genuinely required.
• Mind the App Mesh lifecycle. AWS has placed App Mesh on a deprecation track. For new builds, prefer Istio (ambient mode trims sidecar overhead) or Cilium service mesh (eBPF, sidecar-less) — the architecture here is mesh-agnostic: the VirtualService/router/resilience concepts and the GitOps-driven traffic shifting map directly onto Istio VirtualService/DestinationRule or Cilium policies. Treat the App Mesh specifics as illustrative of the pattern, and pick the in-support mesh for greenfield.
• Don’t put stateful databases in-cluster to “simplify.” Keep Aurora/DynamoDB/S3 as managed services reached via IRSA; that’s what makes the cluster disposable and DR fast.

Alternatives in brief. Pure ECS/Fargate — simpler, no Kubernetes to operate, but you give up the CNCF ecosystem, fine-grained mesh control, and portability. EKS Auto Mode — the same EKS core with AWS operating Karpenter/add-ons for you; an excellent on-ramp that you can graduate from into this fuller architecture as you need more control. Self-managed Kubernetes on EC2 — maximum control, maximum toil; rarely worth it versus EKS. Knative/serverless containers — great for bursty, scale-to-zero event workloads, but not a fit for always-on tier-1 request services. The sweet spot for this reference architecture is the broad middle: organizations large enough to feel the multi-team pain, not so constrained that a managed, lower-control option is clearly better.

Going deeper

Everything above is the what. This section is the how it actually works, the numbers behind the choices, and the tradeoffs the diagram quietly implies. Read it once for the shape, and again later when you are actually designing.

EKS control plane + node strategy

The control plane — API server, scheduler, controller-manager, and the etcd datastore — is run by AWS, replicated across three AZs, patched and scaled for you. You never SSH to a master; the shared-responsibility line sits at the API. What you do own is worker capacity and everything inside the cluster. Each Kubernetes version gets a support window (standard, then a paid extended window), so the durable habit is: run N-1, and treat an upgrade as a scheduled quarterly event, not a fire drill.

The node strategy is the first real design decision, and it has a chicken-and-egg twist. Karpenter is the thing that creates nodes — but Karpenter’s own controller has to run somewhere, and so do CoreDNS, Argo CD, and the App Mesh / Load Balancer / EBS controllers. If those lived on Karpenter-managed nodes, an aggressive consolidation could evict the very controller doing the consolidating. The fix is the pattern already in the component table: a tiny always-on managed node group (2–3 On-Demand nodes) for the control-plane-adjacent add-ons, and Karpenter for every application pod. In Karpenter v1 the shape is two objects — a NodePool (what capacity is allowed) and an EC2NodeClass (how the node is built):

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      requirements:
        - key: kubernetes.io/arch
          operator: In
          values: ["arm64", "amd64"]          # allow Graviton and x86
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]        # Spot-first, On-Demand fallback
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      expireAfter: 720h                          # recycle nodes every 30 days
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 1m
    budgets:
      - nodes: "10%"                             # never disrupt >10% of nodes at once
  limits:
    cpu: "1000"

The load-bearing lines are the last five. WhenEmptyOrUnderutilized is what makes the 3 a.m. cluster shrink automatically; the disruption budget (nodes: "10%") is what stops that same consolidation — or a wave of Spot reclaims — from yanking a quorum out from under a StatefulSet. Karpenter watches an interruption queue (an SQS queue fed by EC2 Spot-interruption and health events) so it gets the ~2-minute Spot warning and cordons/drains the node gracefully instead of letting pods die. If operating all of this sounds like a lot, EKS Auto Mode is AWS running exactly this Karpenter-plus-core-add-ons layer for you — a strong on-ramp you can graduate out of when you need this much control. For the full node-lifecycle treatment see the sibling lesson on Karpenter consolidation, Spot and disruption budgets.

VPC / CNI + IP planning (the math that bites everyone)

The single most common way an EKS platform hits a wall is IP exhaustion, and it is pure arithmetic you can do on day one. The AWS VPC CNI gives every pod a real routable VPC IP address off the node’s Elastic Network Interfaces (ENIs). Without prefix delegation, the ceiling per node is:

maxPods = (ENIs × (IPv4-per-ENI − 1)) + 2

That −1 is the ENI’s own primary IP; the +2 covers host-networked pods like aws-node and kube-proxy. The numbers are smaller than people expect:

(Representative values from the EC2 ENI limits; the formula is exact, the prefix-delegation ceiling is set by the kubelet maxPods you configure.) The trap: an m5.large that could hold 29 pods worth of CPU/RAM may be capped at 29 pods worth of IPs — starving Karpenter’s bin-packing and inflating your node count and bill. Prefix delegation fixes it by handing each ENI slot a /28 prefix (16 IPs at once instead of 1), lifting density to whatever maxPods you set. The other half of the plan is the subnet CIDRs: pods eat VPC IPs, so a stingy /24 per AZ (251 usable) exhausts almost immediately at scale. Size for peak pod count across three AZs, add secondary VPC CIDRs or custom networking if you are IP-constrained, and layer a default-deny NetworkPolicy under the mesh so east-west is closed by default:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress-egress
  namespace: checkout
spec:
  podSelector: {}                     # every pod in the namespace
  policyTypes: ["Ingress", "Egress"]  # deny both directions until explicitly allowed

The full IP-exhaustion and custom-networking story lives in VPC CNI prefix delegation and IP planning.

Ingress + mesh internals

The edge is deliberately two-tier, and the diagram implies a tradeoff at each tier. The AWS Load Balancer Controller reconciles an ALB (or NLB) from your Ingress/Service objects and, critically, runs in IP target mode — the ALB sends packets straight to pod IPs rather than to a node port that then re-hops via kube-proxy. That removes a network hop and makes health-checking honest, but it means every backend pod must have a routable IP (back to the IP-planning section). The ALB then hands off to a single mesh ingress (Envoy) gateway: one policy brain for all north-south entry, at the cost of a component you must now run highly-available.

Inside the mesh, each pod carries an Envoy sidecar injected by the controller; an init container rewrites the pod’s iptables so all traffic is transparently redirected through Envoy. That sidecar is where mTLS, per-route timeouts, retry budgets, and outlier-detection circuit-breaking actually execute — the control plane (App Mesh) only programs the sidecars; the data plane (Envoy) does the work. The honest cost is the sidecar tax: roughly tens of millicores of CPU and 40–100 MB of RAM per pod, plus one extra proxy hop of latency on every call. At 52 services that is negligible; at 2,000 pods it is a line item. That tax is exactly why the industry is moving sidecarless — Istio ambient mode (a per-node ztunnel plus optional waypoint proxies) and Cilium’s eBPF mesh push mTLS and L4 policy into the kernel/node and drop the per-pod proxy. Because AWS has put App Mesh on an end-of-support path (support ends in 2026), treat the VirtualService → VirtualRouter → VirtualNode objects here as illustrative of the pattern: the concepts map almost one-for-one onto Istio’s VirtualService/DestinationRule or Cilium policies, and onto the emerging Gateway API as the vendor-neutral north-south standard. Pick an in-support mesh for anything greenfield.

IRSA / Pod Identity mechanics

“No static keys” is the security crux, so it is worth seeing the actual handshake. With IRSA, the sequence on every pod is: (1) the pod’s ServiceAccount is annotated with eks.amazonaws.com/role-arn; (2) EKS projects a short-lived, signed OIDC token into the pod at /var/run/secrets/eks.amazonaws.com/serviceaccount/token, audience sts.amazonaws.com; (3) the AWS SDK calls sts:AssumeRoleWithWebIdentity with that token; (4) STS validates it against the cluster’s OIDC provider, checks the role’s trust policy, and returns temporary credentials that expire in ~1 hour and auto-rotate. The security lives entirely in the trust-policy condition already shown in the implementation section — sub = system:serviceaccount:<ns>:<sa> — which means only that one ServiceAccount in that one namespace can assume the role. Classic failure modes are all about that handshake: a wrong or missing aud, clock skew that invalidates the token, or a trust policy scoped to * instead of an exact sub (which quietly re-opens the door you thought you closed).

EKS Pod Identity is the newer path and removes the fiddliest part — the per-cluster OIDC trust plumbing:

The “no long-lived keys” principle is identical either way; only the plumbing differs. The migration path and role-reuse patterns are covered in IRSA to Pod Identity migration.

Data, statefulness, and why the cluster stays disposable

The reason the “rebuild from Git in ~30 minutes” DR story works is that state lives outside the cluster — Aurora, DynamoDB, S3, SQS, Secrets Manager — reached via IRSA. The cluster holds no durable data, so it can be deleted and recreated at will. That is a deliberate tradeoff: you accept a network hop and managed-service pricing in exchange for a stateless, disposable compute tier and a DR plan that is mostly terraform apply. When you do need state in-cluster (a Kafka, a specialised database, a workload with strict data-locality), you reach for StatefulSets with the EBS/EFS CSI drivers and usually an operator — but you have now re-coupled durability to the cluster and must back it up and replicate it yourself (Velero, snapshots), which is exactly the toil the managed-services choice avoids. A middle ground for relational load is RDS Proxy to pool the thousands of connections a fleet of pods would otherwise open against Aurora.

Observability at cardinality

The three signals — metrics (ADOT/OpenTelemetry → Amazon Managed Prometheus), logs (FireLens → CloudWatch/OpenSearch), and traces (X-Ray, stitched through Envoy) — are in the component table. The deep problem is cardinality: attach pod, pod_ip, or a request ID as a Prometheus label and each unique value spawns a new time series; a busy platform can go from thousands to millions of series and blow up AMP cost and query latency. The disciplines that keep it sane are: keep high-cardinality identifiers in traces/logs, not metric labels; pre-aggregate hot queries into recording rules; use exemplars to jump from a metric spike straight to the exact trace; and sample traces (head or tail) rather than storing 100%. The payoff is the architecture’s “one source of truth for healthy”: the very same Prometheus SLO recording rules that power the dashboards are the signals Argo Rollouts reads to gate a canary — so “the graph is red” and “the deploy auto-rolled back” are the same fact.

Multi-account and landing zones

One cluster with many namespaces is right until it isn’t. At enterprise scale the blast-radius and audit arguments push you toward multiple AWS accounts — commonly one account per environment (dev/stage/prod) and sometimes per business unit — governed centrally with AWS Organizations / Control Tower (a landing zone). Shared concerns like the ECR registry and log/observability sinks live in their own platform accounts, and workloads reach them cross-account — which is precisely where Pod Identity earns its keep over IRSA. The tradeoff is familiar: account-per-environment gives you a hard security and quota boundary and a clean bill per env, at the cost of more IAM and networking wiring (Transit Gateway, cross-account roles, centralised policy). This reference architecture is the inside-one-account picture; a landing zone is the how many accounts, wired how picture around it. The progression from single cluster to multi-account/multi-region is its own topic — see the architecting ladder lessons in this course.

Cost + resilience math (the tradeoffs the diagram implies)

Two invisible costs hide inside a clean-looking diagram. The first is cross-AZ data transfer: every arrow that crosses an AZ boundary costs roughly $0.01/GB in each direction (representative intra-region pricing). Spread 52 chatty services naively across 3 AZs and a surprising share of your bill is just services talking sideways. The mitigations are Topology Aware Routing (keep a call in-AZ when a local endpoint exists) and VPC endpoints (so ECR/STS/S3/DynamoDB traffic never touches a NAT gateway’s per-GB charge). The second is the Spot ↔ stability tension: Spot at ~70% off is the biggest single lever, Graviton adds ~20% price/performance, and consolidation repacks waste — but all three increase node churn, which is why disruption budgets, PodDisruptionBudgets, and topology-spread constraints are non-optional, not nice-to-haves.

On resilience, the multi-AZ choice is also arithmetic. Three AZs, not two, because quorum-based systems (etcd, and any in-cluster Raft/Paxos database) need a majority to survive one AZ loss: 3 nodes tolerate 1 failure, 2 nodes tolerate 0. And the DR target decomposes cleanly, which is worth internalising:

The single most important DR insight is on that second row: RPO is a property of your databases, not your Kubernetes. GitOps makes the compute rebuildable in minutes; only cross-region data replication buys you a small RPO.

Practice challenges

Work these top-to-bottom; they escalate from “read the diagram” to “defend a design in a review.” Try each before opening the solution.

1 — Trace the request (beginner). Using the architecture diagram, list every hop a single user request takes, from the browser to a read on a DynamoDB table, naming the component at each hop.

<details><summary>Solution</summary>

Browser → Route 53 (DNS) → WAF → ALB (reconciled by the AWS Load Balancer Controller, IP target mode) → App Mesh ingress gateway (Envoy) → the service’s Envoy sidecar → the application pod → (AWS SDK) IRSA/Pod Identity exchanges the projected token at STS for short-lived credentials → DynamoDB (via a VPC endpoint). Why it matters: every one of those hops is a place to add security or observability — and the mesh hops are where mTLS and retries live. </details>

2 — Size the IP plan (beginner → intermediate). You give each AZ a single /24 subnet (251 usable IPs) and run the VPC CNI without prefix delegation on m5.large nodes. At ~25 application pods per node, roughly how many nodes’ worth of pods fit per AZ before you run out of IPs, and what is the fix?

<details><summary>Solution</summary>

An m5.large caps at 29 pods by IP (3×(10−1)+2), so ~25 app pods/node is already near the node ceiling. Worse, 251 pod IPs per /24 ≈ 10 nodes’ worth before the subnet is exhausted — tiny for a real platform, and it collides with ALB/ENI IPs too. Fixes, in order: enable prefix delegation (density jumps to your maxPods), and add secondary VPC CIDRs / larger subnets sized for peak pod count across 3 AZs. The lesson: do this arithmetic on day one, not during a scaling incident. </details>

3 — Least-privilege pod identity (intermediate). The tracking service (ServiceAccount tracking-api in namespace tracking) must read and write exactly two DynamoDB tables and nothing else. Sketch the trust-policy condition and the permission scope.

<details><summary>Solution</summary>

Trust policy condition (IRSA): StringEquals { "<oidc>:sub" = "system:serviceaccount:tracking:tracking-api", "<oidc>:aud" = "sts.amazonaws.com" } — so only that SA in that namespace can assume the role. Permission policy: Allow on dynamodb:GetItem/PutItem/Query/UpdateItem scoped to the two table ARNs (and their indexes) — no dynamodb:*, no Resource: "*". Why: this is precisely what makes the previously-leaked-key incident structurally impossible — even a stolen token only unlocks two tables. </details>

4 — Critique this design (intermediate → advanced). A team proposes: one cluster, one node group (On-Demand only), all services in the default namespace, an Envoy sidecar on every pod “for consistency,” and a single AZ to “save money.” Name at least five problems and the fix for each.

<details><summary>Solution</summary>

(1) Single AZ — one AZ outage is a full outage; spread across 3 AZs with topology-spread + PDBs. (2) One On-Demand node group — no Spot/Graviton, no per-pod right-sizing; replace with Karpenter NodePools mixing capacity types and architectures. (3) Everything in default — no isolation, quota, or blast-radius boundary; use namespaces per team/tier with quotas and NetworkPolicies. (4) Mesh on every pod — pays the sidecar tax platform-wide before it is needed; add the mesh only where per-route retries/mTLS/canaries are genuinely required, or go sidecarless. (5) “Save money” via single AZ — false economy; the real savings are Spot + Graviton + consolidation + VPC endpoints, which don’t cost you availability. Bonus: no GitOps/least-privilege identity mentioned — add both. </details>

5 — Canary that defends itself (advanced). Explain how a 5% → 25% → 100% rollout, shifting traffic by reweighting the mesh VirtualRouter, decides to abort — and what the single source of truth for “healthy” is.

<details><summary>Solution</summary>

Argo Rollouts runs an analysis at each step, querying Amazon Managed Prometheus for the golden-signal SLOs (error rate, p99 latency) defined as recording rules. If a metric breaches its threshold during any step, the rollout auto-aborts and shifts weight back to the stable version; if all steps pass, it promotes to 100%. The source of truth is those same AMP SLO rules that power the dashboards — one definition of “healthy” drives both the graph and the gate, so there is no drift between what humans see and what the pipeline enforces. </details>

6 — DR tabletop (advanced). us-east-1 is gone. Write the recovery order and state what determines RTO and RPO.

<details><summary>Solution</summary>

Order: (1) terraform apply to recreate the cluster + core add-ons in the DR region (or promote a warm standby cluster); (2) Argo CD reconciles every workload from the config repo — images already present via ECR cross-region replication; (3) fail over the data tier (promote the Aurora global-DB secondary / rely on DynamoDB global tables / point at S3 CRR). RTO is dominated by cluster-create + Argo sync (~30 min, less with a warm standby). RPO is set entirely by the data tier’s replication lag (~seconds with global tables), not by Kubernetes. The whole point of GitOps here is that compute recovery is boring and repeatable. </details>

Common beginner mistakes

• Adopting the whole stack too early. The misconception: “real companies run a mesh + GitOps + Karpenter, so we should from day one.” Why it’s wrong: for 3–5 services and one team, the mesh, ApplicationSets, and multi-NodePool machinery are pure cognitive overhead you will pay for daily. Right model: adopt by team count, not service count. Start on EKS Auto Mode, ECS on Fargate, or App Runner, and graduate into this architecture when multiple squads genuinely contend for one platform.

• Single-AZ (or two-AZ) to save money. The misconception: “AZs are just data centres; one is fine, and cross-AZ transfer costs money.” Why it’s wrong: a single AZ failure is a total outage, and quorum systems can’t survive it. Right model: three AZs are table stakes — spread with topology-spread constraints and PodDisruptionBudgets; recover the transfer cost with Topology Aware Routing, not by collapsing to one AZ.

• No IP plan. The misconception: “Kubernetes handles networking; I’ll pick a /24 and move on.” Why it’s wrong: the VPC CNI assigns real VPC IPs to every pod, so a small CIDR or ENI-limited node silently caps pod density and stalls scaling mid-incident. Right model: do the maxPods and subnet arithmetic on day one, enable prefix delegation, and size CIDRs for peak pods across 3 AZs.

• Mesh by default, everywhere. The misconception: “a service mesh is best practice, so inject a sidecar into every pod.” Why it’s wrong: the sidecar tax (CPU/RAM/latency per pod) and the operational weight are real, and most services need only mTLS + basic policy. Right model: NetworkPolicy + ALB until per-route retries/timeouts/circuit-breaking/canaries are genuinely required — and when you do adopt a mesh, prefer a sidecarless one (Istio ambient, Cilium) on greenfield.

• Ignoring cost until the bill arrives. The misconception: “get it working on On-Demand, optimise later.” Why it’s wrong: an all-On-Demand, single-instance-shape, no-consolidation cluster with chatty cross-AZ traffic can cost ~2× the necessary, and “later” rarely comes. Right model: Spot + Graviton + consolidation + VPC endpoints + cost-allocation tags from the start, so spend tracks traffic instead of peak.

• Letting Terraform and Argo CD both own workloads. The misconception: “Terraform manages infra, so it can manage my Deployments too.” Why it’s wrong: two controllers reconciling the same objects means perpetual drift fights — each reverts the other. Right model: Terraform up to the bootstrapped cluster + core add-ons; GitOps for everything inside. One owner per object.

Glossary

• EKS (Elastic Kubernetes Service) — AWS’s managed Kubernetes: AWS runs the control plane; you run workloads and (optionally) worker nodes.
• Control plane — the brain of a cluster (API server, scheduler, controller-manager, etcd). On EKS, fully managed by AWS across 3 AZs.
• Worker node — an EC2 instance that actually runs your pods.
• Managed node group — a small AWS-managed set of always-on nodes; here it hosts the controllers that must never be evicted.
• Karpenter — a just-in-time node autoscaler that launches the cheapest instance that fits pending pods and repacks (consolidates) waste.
• Consolidation — Karpenter terminating and repacking underused nodes to cut cost; WhenEmptyOrUnderutilized is the aggressive policy.
• Spot / On-Demand — Spot is spare EC2 capacity at ~70% off but reclaimable with ~2 min notice; On-Demand is full-price and guaranteed.
• Graviton — AWS’s Arm-based (arm64) CPUs, typically ~20% better price/performance for multi-arch workloads.
• Disruption budget — a Karpenter limit on how many nodes it may disrupt at once, protecting quorum during consolidation/Spot churn.
• VPC CNI — the AWS networking plugin that gives each pod a real, routable VPC IP address from the node’s ENIs.
• ENI (Elastic Network Interface) — a virtual NIC on an EC2 instance; its IP capacity caps pod density unless prefix delegation is on.
• Prefix delegation — assigning each ENI slot a /28 (16 IPs) instead of one, raising pod-per-node density to the kubelet maxPods cap.
• NetworkPolicy — Kubernetes firewall rules between pods; a default-deny one closes all east-west traffic until explicitly allowed.
• App Mesh / service mesh — an overlay that adds mTLS, routing, retries, and circuit-breaking to service-to-service (east-west) calls.
• Envoy — the high-performance proxy that runs as a sidecar (or per-node) and enforces all mesh policy; the mesh’s data plane.
• Sidecar — a helper container injected next to your app container in the same pod; here it’s the Envoy proxy.
• mTLS (mutual TLS) — both sides of a connection present certificates, so traffic is encrypted and both identities are verified.
• VirtualService / VirtualRouter / VirtualNode — App Mesh objects that describe a logical service, its routing rules, and its concrete pod backends.
• Ingress gateway — the single managed Envoy entry point into the mesh for north-south (client → cluster) traffic.
• ALB / NLB — AWS Application (L7) / Network (L4) Load Balancers; the north-south edge, declared as code via the Load Balancer Controller.
• AWS Load Balancer Controller — an in-cluster controller that provisions ALBs/NLBs from Ingress/Service objects (IP target mode sends packets straight to pods).
• IRSA (IAM Roles for Service Accounts) — the mechanism that gives each pod its own short-lived AWS credentials via an OIDC-token → STS exchange, replacing static keys.
• EKS Pod Identity — the newer alternative to IRSA using an agent + association API; no per-cluster OIDC trust, first-class cross-account.
• OIDC / STS — OpenID Connect provides the signed token; AWS STS exchanges it (AssumeRoleWithWebIdentity) for temporary, scoped credentials.
• ServiceAccount — a Kubernetes identity for pods; annotated with a role ARN, it becomes the anchor for IRSA/Pod Identity.
• ECR (Elastic Container Registry) — AWS’s private image registry; the start of the verified supply chain (scan + sign + immutable tags).
• cosign / SBOM — cosign cryptographically signs container images; an SBOM (Software Bill of Materials) lists every component inside an image.
• Kyverno — a policy engine that, at admission time, can refuse any image that isn’t from your ECR and signed.
• Admission control — the Kubernetes gate that validates or mutates objects before they’re created; where the image-signature check lives.
• GitOps — the practice of making Git the single source of truth, with a controller continuously reconciling the cluster to match it.
• Argo CD — the GitOps controller here; watches the config repo and applies/heals the cluster to the declared desired state.
• Argo Rollouts — progressive-delivery controller that runs canaries/blue-green by shifting mesh weights and auto-rolling-back on SLO breach.
• Canary — releasing to a small slice of traffic (e.g. 5%) first, watching metrics, then ramping to 100% only if healthy.
• HPA / KEDA — Horizontal Pod Autoscaler scales pods on CPU/custom metrics; KEDA scales on event sources (SQS depth, lag).
• PodDisruptionBudget (PDB) — a floor on how many replicas must stay available during voluntary disruptions (drains, consolidation).
• Topology-spread / Topology Aware Routing — scheduling pods evenly across AZs; and preferring an in-AZ endpoint to cut cross-AZ transfer cost.
• VPC endpoint — a private door to an AWS service (ECR, STS, S3, DynamoDB) that keeps traffic off the NAT gateway, cutting cost and attack surface.
• RTO / RPO — Recovery Time Objective (how long to restore) and Recovery Point Objective (how much data loss); here RTO ≈ cluster rebuild, RPO ≈ data-tier replication.
• AMP / AMG / ADOT — Amazon Managed Prometheus, Amazon Managed Grafana, and the AWS Distro for OpenTelemetry; the metrics/dashboards/collection stack.
• Landing zone — a multi-account AWS baseline (via Organizations/Control Tower) that this single-cluster architecture sits inside at enterprise scale.