GPU Inference Platform for LLMs on AWS EKS with Karpenter

In a nutshell

Picture a large language model as a very well-read intern who can summarise a 40-page document in seconds — but only if you sit them at the right desk. That desk is a GPU: a chip with thousands of cores built for the massive parallel matrix maths an LLM does for every word it reads or writes. A CPU can run the model too, the way you could bail out a boat with a teaspoon; the GPU is the bucket. “Serving an LLM,” then, means getting your model onto GPUs — and keeping them busy, because idle GPUs are the most expensive thing in the building.

On EKS (Amazon’s managed Kubernetes) you wrap the model and a serving program — here vLLM — in a container; Kubernetes runs it as a pod; the pod asks for a GPU. But Kubernetes owns no GPUs. It only places pods onto machines (nodes) that already have them, so somebody has to make a GPU node appear.

That somebody is Karpenter, a just-in-time node autoprovisioner — think on-demand valet for compute. The instant a pod is stuck Pending because no GPU node exists, Karpenter reads exactly what it needs, rents the cheapest matching GPU machine from AWS in about a minute, and joins it to the cluster. When the work drains, Karpenter hands the machine back so you stop paying. You pay for silicon by the request-storm, not by the calendar — no garage full of GPUs idling overnight.

Everything else here makes that loop cheap, fast, and safe: a token-aware gateway at the front door, KServe to scale model copies up and down (even to zero), vLLM to wring the most tokens from each GPU, and an S3 registry holding the exact weights an auditor signed off on. We start from this picture and go all the way down to the device driver.

Level: Advanced · Time: ~33 min · You should already know Kubernetes pods, deployments and services, plus a first pass at Karpenter and cluster autoscaling. If that is shaky, warm up with Autoscaling — HPA, KEDA & Karpenter and Karpenter on EKS — consolidation, Spot & disruption budgets. The Azure counterpart is GPU inference on AKS with KAITO.

After this lesson you can explain how a GPU becomes a schedulable Kubernetes resource; write a Karpenter GPU NodePool and EC2NodeClass that target the right instance families; choose between vLLM and TGI; split one physical GPU with MIG or time-slicing; and reason about the cold-start, Spot-interruption, and KV-cache tradeoffs that decide whether the platform is fast and cheap or slow and dear.

A national health-insurance payer’s clinical-operations group lands a directive from the chief medical officer: claims adjudicators and nurse case managers are drowning in 40-page prior-authorization packets, and every hour a complex case sits in the queue is an hour a member waits for care and a day the payer carries open liability. The ask is an assistant that summarizes a packet, extracts the diagnosis and procedure codes, and drafts a medical-necessity rationale against policy — at the desk, in seconds. The constraint is the one that kills the easy answer: this is PHI under HIPAA, the data-governance board has a standing prohibition on sending member clinical records to any third-party model API, and the legal team will not accept “your prompts may be retained for abuse monitoring” in a BAA. So the model has to run inside the payer’s own AWS account, on their own GPUs, with no member data ever leaving the VPC. That single non-negotiable — self-hosted inference — is what turns a weekend prototype into the platform this article describes: open-weight LLMs served on Amazon EKS, with GPUs that appear only when there is work and vanish when there is not, behind a gateway that meters and protects every request.

The pressures stack the way they always do when you own the silicon. Cost is the dominant one: an A10G or L40S GPU left running idle overnight burns real money, and an H100 burns a great deal of it, so utilization is not a nice-to-have but the whole economic case. Latency means an adjudicator mid-review will not wait fifteen seconds for a summary, and a cold GPU that takes four minutes to come up and load 30 GB of weights is a latency event, not a scaling event. Scale means a Monday-morning queue surge of hundreds of concurrent packets against a Tuesday-afternoon lull, and the platform has to track that curve without either dropping requests or paying for peak all week. And governance means every token, every model version, and every node has to be auditable for a regulator. Self-hosting on EKS with Karpenter doing GPU lifecycle and vLLM doing high-throughput serving is the pattern that satisfies all four — but only if the pieces are assembled deliberately.

Why not the obvious shortcuts

Three shortcuts will be proposed in the first planning meeting, and naming why each fails saves a quarter of wasted effort.

Call a hosted model API. The cleanest engineering answer and a complete non-starter here: it sends PHI across a tenant boundary the governance board has explicitly forbidden, and no abuse-monitoring retention clause survives the BAA review. For a payer, this option is not “less private,” it is “illegal.”

Run the model on a fixed fleet of always-on GPU EC2 instances. This works and it is what most teams reach for first, but it inverts the cost problem: you size the fleet for Monday’s peak and pay for it through Sunday’s trough, and GPU instances do not get cheaper when they idle. You also inherit manual capacity management — someone watching dashboards and resizing an Auto Scaling Group, badly, at 8 a.m.

Put a single GPU behind a Flask app and a queue. It demos beautifully and collapses under concurrency: one in-flight request blocks the GPU, batching is naive or absent, and throughput per dollar is a fraction of what the hardware can do. The moment a second adjudicator submits, latency doubles.

The platform threads the needle by separating three concerns that the shortcuts conflate. Karpenter owns when GPUs exist — it provisions a right-sized GPU node within a minute of a pending pod and terminates it when the work drains, so you pay for silicon by the request-storm, not by the calendar. vLLM owns how efficiently each GPU serves — continuous batching and PagedAttention push tokens-per-second per GPU to several times what a naive server delivers. And KServe owns how many model replicas exist — it scales serving pods on real demand signals, down to zero between surges. Three independent control loops, each tuned for the thing it controls.

Architecture overview

GPU Inference Platform for LLMs on AWS EKS with Karpenter — architecture

Read the diagram left to right and the platform tells its own story: identity (Okta federated to Entra) and the Akamai edge on the left; the token-aware gateway as the single front door; then KServe routing each request to a vLLM pod running on a GPU node that Karpenter conjured on demand; and off to the side, the S3 model registry that feeds weights in and the Datadog / ServiceNow / security plane that watches it all.

The platform runs two paths that share a cluster but live on different clocks: a synchronous inference path that serves adjudicators in real time, and an asynchronous model-lifecycle path that publishes, validates, and promotes model versions. Keeping them mentally separate is the first step to operating this well — one is measured in milliseconds, the other in builds.

The defining property of the topology is the one the governance board cares about most: the entire serving plane lives in private subnets with no inbound path from the internet, model weights never leave the account, and every egress to AWS services rides a VPC endpoint. Members’ clinical text enters through the payer’s own edge, is processed on GPUs the payer owns, and the completion returns — without a single prompt or token transiting a third party. That is what makes the self-hosting story defensible to a HIPAA auditor.

Inference path, following the control flow:

An adjudicator opens the assistant inside the payer’s claims workbench. Identity federates through Okta as the workforce IdP (the payer’s standard), brokered to Microsoft Entra ID for the Microsoft-side resources, so every call carries a first-class, group-stamped token. Traffic hits Akamai at the edge for TLS termination, global anycast, and WAF/bot protection before it reaches AWS.
The request lands on the token-aware inference gateway — an Envoy-based gateway (the AI Gateway pattern) running in the cluster. It validates the OIDC JWT, attaches the caller’s team and cost-center claims, enforces a per-team token-per-minute budget, rate-limits by tenant, and — critically for self-hosted LLMs — meters prompt and completion tokens rather than just request count, because a 30-page packet and a one-line question cost wildly different amounts of GPU. This is the single front door: one place to authenticate, throttle, meter, and audit every model call.
The gateway routes to a KServe InferenceService fronting the target model. KServe’s router handles model-name routing and, where configured, canary traffic splitting between model versions.
The request reaches a vLLM serving pod scheduled on a GPU node. vLLM has already loaded the weights into GPU memory at startup; it adds the incoming request to its continuous batch, runs decoding with PagedAttention for memory-efficient KV-cache handling, and streams tokens back. If no GPU node currently has capacity, the pod sits Pending — which is the signal that drives the next loop.
A Pending GPU pod is seen by Karpenter, which evaluates its resource and nvidia.com/gpu requirements, selects the cheapest instance type that fits from its NodePool (an g6.xlarge/L4 for a 7-8B model, a g6e/L40S or p4d/A100 for a 70B model), launches it — often as Spot for batch-tolerant traffic with On-Demand fallback — bootstraps it with the GPU device plugin via a Bottlerocket GPU AMI, and the pod schedules within roughly a minute.
The cited, structured answer streams back through the gateway to the adjudicator; the request, token counts, model version, and latency are emitted to Datadog.

Model-lifecycle path, independent and build-driven: a new or fine-tuned open-weight model (say a Llama- or Mistral-family checkpoint, or an internally fine-tuned variant) is published to an S3 model registry — an immutable, versioned prefix layout (s3://payer-models/clinical-summary/v7/) with weights, tokenizer, and a signed manifest. A pipeline runs an offline evaluation (groundedness against a golden packet set, code-extraction accuracy, latency on a reference GPU), and only a passing version is tagged for promotion. KServe storage initializers pull weights from S3 at pod startup over a Gateway VPC endpoint, so the multi-gigabyte download stays on the AWS backbone and never touches the public internet.

Component breakdown

Component	Service / tool	Role in the platform	Key configuration choices
Edge	Akamai	TLS, anycast, WAF, bot mitigation at the perimeter	WAF rules for prompt-flood / token-abuse patterns; origin shield to the private gateway
Identity / SSO	Okta + Microsoft Entra ID	Workforce SSO (Okta) federated to Entra; group claims to the gateway	OIDC federation; cost-center claim drives token chargeback
AI gateway	Envoy AI Gateway	JWT validation, token-aware metering, rate limiting, model routing	Per-team token-per-minute limits; OpenAI-compatible `/v1` routes
Model serving	vLLM	High-throughput LLM inference engine	Continuous batching; PagedAttention; `--tensor-parallel-size` per model
Serving control	KServe	Model deployment, request-driven autoscaling, scale-to-zero, canary	KEDA/KPA on concurrency; `minReplicas: 0` for off-peak models
GPU lifecycle	Karpenter	Just-in-time GPU node provisioning and consolidation	GPU NodePool; Spot + On-Demand fallback; `consolidationPolicy`; TTL
Model registry	Amazon S3	Immutable versioned weight + manifest store	Versioned prefixes; Object Lock; bucket-key SSE-KMS; VPC endpoint
Secrets	HashiCorp Vault	Registry creds, gateway signing keys, fine-tune data tokens	IRSA-backed auth; dynamic leases; Vault Agent sidecar injection
CSPM / posture	Wiz + Wiz Code	Cloud posture, attack-path analysis, IaC scanning pre-merge	Agentless EKS scan; alert on public-exposure or open-SG drift
Runtime security	CrowdStrike Falcon	Runtime threat detection on GPU nodes and the cluster	Sensor as DaemonSet; container drift detection; SOC pipeline
Observability	Datadog	GPU utilization, token throughput, latency SLOs, cost telemetry	DCGM GPU metrics; APM trace per request; SLO monitors
ITSM / approvals	ServiceNow	Model-promotion approvals, change requests, incident records	Change gate before a model version goes live; auto-ticket on SLO breach
CI/CD + IaC	GitHub Actions + Argo CD + Terraform	Build/eval pipeline; GitOps deploy; infrastructure as code	OIDC to AWS (no stored creds); eval gate; Argo syncs KServe manifests

A few of these choices deserve the why, because they are the ones teams get wrong.

Why Karpenter, not the Cluster Autoscaler with GPU ASGs. The Cluster Autoscaler scales pre-defined node groups, which forces you to guess GPU instance types up front and maintain a separate ASG per type. Karpenter is instance-type-aware and bin-packs to the workload: it reads the pending pod’s exact GPU and memory request and picks the cheapest instance that fits from a flexible NodePool, mixes Spot and On-Demand, and — the part that drives the cost case — runs consolidation, proactively replacing underused nodes and terminating empty ones so a GPU never idles for long. For a fleet where a single H100 hour is expensive, “provision the exact GPU the pod needs, and reclaim it the moment it drains” is the entire economic argument.

Why vLLM, not a vanilla model server. The naive pattern processes one request per forward pass and leaves the GPU underfed. vLLM’s continuous (in-flight) batching keeps the GPU saturated by adding and retiring requests from the batch every decoding step, and PagedAttention manages the KV cache like virtual memory so you fit far more concurrent sequences in GPU RAM. The result is several times the tokens-per-second per GPU, which directly divides your cost-per-million-tokens. On self-hosted hardware, throughput per GPU is the unit economics.

Why a token-aware gateway, not a plain API gateway. A standard gateway meters requests; an LLM platform must meter tokens, because cost and GPU time scale with prompt and completion length, not request count. The Envoy AI Gateway enforces a per-team token-per-minute budget so the appeals team cannot exhaust the GPU capacity the adjudication team is paying for, routes by model name across KServe backends, and produces one audit log of who sent what to which model version — the chargeback and compliance backbone in one layer.

Implementation guidance

Provision with Terraform, and treat the GPU NodePool and the network as first deliverables. Get the device plugin, the AMI, and the VPC endpoints right before anything else, or pods schedule onto nodes with no usable GPU and weight downloads silently traverse a NAT gateway you are paying per-GB for.

An EKS cluster (private API endpoint) with private subnets for the serving plane and Gateway VPC endpoints for S3 plus Interface endpoints for ECR, STS, and CloudWatch, so weight pulls and image pulls stay on the AWS backbone.
Karpenter installed with an EC2NodeClass using a Bottlerocket NVIDIA GPU AMI (the device plugin is built in) and a GPU NodePool constrained to GPU families, Spot+On-Demand, with consolidation and a node TTL.
KServe (with Knative/KEDA for request-driven scaling) and the Envoy AI Gateway.
The S3 model registry bucket with versioning, Object Lock, and SSE-KMS; IRSA roles granting the storage initializer read-only access to exactly its model prefix.
Argo CD watching the GitOps repo so InferenceService and gateway manifests deploy by merge, not by kubectl.

A minimal Karpenter GPU NodePool communicates the intent — cheapest fitting GPU, Spot-first, reclaim aggressively:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu-inference
spec:
  template:
    spec:
      requirements:
        - key: karpenter.k8s.aws/instance-family
          operator: In
          values: ["g6", "g6e", "p4d"]      # L4 / L40S / A100
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]      # Spot first, On-Demand fallback
      taints:
        - key: nvidia.com/gpu
          effect: NoSchedule                 # only GPU pods land here
      expireAfter: 720h                      # recycle nodes ~monthly (drift + patching)
      nodeClassRef:
        group: karpenter.k8s.aws             # v1 requires group + kind + name
        kind: EC2NodeClass
        name: gpu-bottlerocket
  limits:
    nvidia.com/gpu: 64                        # hard ceiling on fleet GPUs
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 2m                      # reclaim idle GPUs fast

And the KServe InferenceService that serves a model from the S3 registry with scale-to-zero between surges:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: clinical-summary
spec:
  predictor:
    minReplicas: 0                            # scale to zero off-peak
    maxReplicas: 8
    scaleTarget: 12                           # target concurrent requests/replica
    scaleMetric: concurrency
    model:
      modelFormat: { name: vLLM }
      storageUri: s3://payer-models/clinical-summary/v7/
      args: ["--max-model-len=16384", "--tensor-parallel-size=1"]
      resources:
        limits: { nvidia.com/gpu: "1" }

The pipeline that applies this runs in GitHub Actions, authenticating to AWS via OIDC federation so there is no stored access key to leak. It builds the serving image, runs the offline eval gate, pushes the model version to S3, and updates the GitOps manifest; Argo CD then reconciles the change onto the cluster. Terraform owns the cluster, NodePool, S3, IAM/IRSA, and endpoints; Ansible handles any node-level or golden-AMI hardening config that lives outside the managed AMI.

Identity: kill the static keys, federate the humans. No node and no pod holds a long-lived AWS key. The serving pods, the storage initializer, and the gateway authenticate to AWS with IRSA (IAM Roles for Service Accounts), each scoped to the minimum — the summarizer’s role can read only its own S3 model prefix and decrypt with only its KMS key. Human SSO flows Okta → Entra: adjudicators log in once with the payer’s Okta credentials and conditional-access policies, Okta federates to Entra over OIDC, and the resulting token carries the team and cost-center claims the gateway consumes for routing and chargeback. The residual secrets that are not IAM roles — third-party registry credentials, the gateway’s JWT signing key, tokens for the fine-tuning data lake — live in HashiCorp Vault, leased dynamically and injected by the Vault Agent sidecar, so nothing sensitive sits in a Kubernetes Secret.

How a GPU becomes a schedulable resource

Everything above treats “a GPU node” as a given. This section opens the box: how a physical GPU turns into something the Kubernetes scheduler can hand to a pod, how Karpenter is told which GPUs to buy, which serving engine to run on them, and how to slice one card between several small models.

GPUs are an extended resource, advertised by a device plugin

Kubernetes understands cpu and memory natively. It does not understand GPUs. A GPU reaches the scheduler as an extended resource — an opaque, node-level countable named nvidia.com/gpu — and something has to put that number on the node. That something is the NVIDIA device plugin: a DaemonSet (bundled into the Bottlerocket NVIDIA variant, or deployed via the NVIDIA GPU Operator on other AMIs) that runs on every GPU node, discovers the cards, and reports to the kubelet “this node has 1 (or 4, or 8) nvidia.com/gpu.” The kubelet advertises that capacity, the scheduler filters for nodes that have it, and at container start the NVIDIA container runtime injects the GPU device and driver into the container.

A pod asks for a GPU the same way it would ask for any extended resource:

resources:
  limits:
    nvidia.com/gpu: "1"     # whole GPUs only; you cannot request 0.5 here

Three rules trip up newcomers. Extended resources are limits-only — you set limits and Kubernetes copies the value to requests for you; you cannot request more than you limit. They are integers — nvidia.com/gpu: 1 or 2, never 0.5 (fractional GPUs come from a different trick, below). And they are non-overcommittable — two pods cannot each be handed the same whole GPU, so if a node has one card, exactly one one-GPU pod runs there. When no node advertises a free nvidia.com/gpu, the pod stays Pending — and that pending pod is the doorbell Karpenter answers.

Telling Karpenter which GPUs to buy: NodePool + EC2NodeClass

Karpenter reads two objects. The NodePool (shown earlier) is the policy: which instance families are allowed, Spot vs On-Demand, taints, limits, and how aggressively to reclaim idle nodes. The EC2NodeClass is the hardware template: which AMI, IAM role, subnets, security groups, and disks a node is built from. The NodePool references it by nodeClassRef.

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: gpu-bottlerocket
spec:
  amiSelectorTerms:
    - alias: bottlerocket@latest     # Karpenter picks the NVIDIA variant for GPU types
  role: "KarpenterNodeRole-payer-eks"
  subnetSelectorTerms:
    - tags: { karpenter.sh/discovery: "payer-eks" }
  securityGroupSelectorTerms:
    - tags: { karpenter.sh/discovery: "payer-eks" }
  blockDeviceMappings:
    - deviceName: /dev/xvdb          # Bottlerocket data volume holds images + weights
      ebs:
        volumeSize: 200Gi            # a 70B model in fp16 is ~140 GB — size disk for it
        volumeType: gp3
        encrypted: true

Two details matter. First, alias: bottlerocket@latest lets Karpenter resolve the accelerated (NVIDIA) Bottlerocket AMI automatically whenever it launches a GPU instance type — you never hard-code an AMI ID that goes stale. Second, the disk is not incidental: the KServe storage initializer downloads the model weights onto the node before vLLM starts, and a large model is tens to hundreds of gigabytes, so an undersized data volume is a silent cause of failed pulls and evictions mid-surge.

The NodePool’s requirements are where you target GPU families — g6 (L4) for 7-8B models, g6e (L40S) for mid-size, p4d/p5 (A100/H100) for 70B and up — and let Karpenter pick the cheapest instance that fits the pending pod’s request from that set. That single line, “give me the cheapest thing that fits,” is the whole right-sizing story.

The serving engine: vLLM, TGI, and why it is not a plain web server

A GPU alone does nothing; a model server turns it into an inference endpoint. The naive choice — load the model in a Flask handler, one request per forward pass — wastes most of the card. Production servers win by continuous batching: they add and retire requests from an in-flight batch at every decoding step, so the GPU never waits for the slowest sequence to finish. The two mainstream open engines both do this:

Concern	vLLM	HuggingFace TGI
Throughput core	Continuous batching + PagedAttention KV cache	Continuous batching + paged/Flash attention
Multi-GPU sharding	`--tensor-parallel-size N`	`--num-shard N` (`--sharded true`)
API surface	OpenAI-compatible `/v1/*` (+ native)	`/generate` + OpenAI-compatible `/v1/*`
Quantization	AWQ, GPTQ, FP8, bitsandbytes	AWQ, GPTQ, EETQ, FP8, bitsandbytes
Sweet spot	Max throughput, widest model + feature coverage	Tight HuggingFace-ecosystem integration

For this platform, vLLM is the default for its throughput and its OpenAI-compatible API (the gateway speaks /v1 to every backend uniformly), with --tensor-parallel-size sharding a 70B model across the four GPUs of a p4d. TGI is a fine substitute where a team already lives in the HuggingFace toolchain. Either way the KServe modelFormat selects the runtime and the args pass engine flags straight through.

Fractional GPUs: when a whole card is too much

A 7B model quantized to 4-bit fits in a few gigabytes; giving it an entire L4 (or worse, an A100) wastes the card. Three mechanisms share one physical GPU across pods:

Mode	Isolation	Best for	The catch
Time-slicing	None — time-multiplexed, shared VRAM	Dev, low-QPS, bursty internal tools	A noisy neighbour can OOM the whole card; no fault isolation
MPS (Multi-Process Service)	Spatial SM partitioning, shared VRAM	Steady, cooperative multi-tenant	Still one memory pool; a crash can hit peers
MIG (A100/H100)	Hardware-isolated compute and memory	Production multi-tenant, guaranteed slices	Only on select GPUs; fixed profiles; reconfig drains the node

Time-slicing is a device-plugin setting: advertise each physical GPU as several schedulable replicas.

# NVIDIA device-plugin config: present each GPU as 4 shareable units
version: v1
sharing:
  timeSlicing:
    resources:
      - name: nvidia.com/gpu
        replicas: 4        # node now advertises 4x nvidia.com/gpu per card

After this, a single-GPU node reports nvidia.com/gpu: 4 and four small pods can land on it — but they share one memory space and take turns on the compute, so this is a cost lever for low-traffic models, not a way to hit latency SLOs under load. MIG is the opposite tradeoff: on an A100 or H100 it carves the card into up to seven hardware-isolated instances (profiles like 1g.10gb), each surfaced as its own resource such as nvidia.com/mig-1g.10gb, so a slice behaves like a small, private GPU with guaranteed memory. Use time-slicing to pack cheap dev workloads, MIG to give production models a firm, isolated fraction.

Scale-to-zero, mechanically

The failure-mode note later warns about cold starts; here is the machinery behind them. With minReplicas: 0, KServe (in its Knative/serverless mode) lets a model with no traffic drain to zero pods; Karpenter then sees the now-empty GPU node and reclaims it. The next request does not 503 — Knative’s activator buffers it, forces a scale-from-zero, and holds the connection until a pod is ready. “Ready,” though, is a chain: Karpenter launches a node (~1 min) → the image and the multi-gigabyte weights download → vLLM loads them into VRAM and warms up. That is the cold-start budget you trade for zero idle cost, which is why latency-critical models keep minReplicas: 1 (a warm pool) and only batch-tolerant ones scale to zero. The scaleMetric: concurrency / scaleTarget pair decides when to add replicas: at scaleTarget: 12, KServe aims for ~12 in-flight requests per pod and adds a replica when the average climbs past it.

Going deeper

Karpenter’s control loop: provisioning, drift, consolidation

Karpenter is not a batch autoscaler that resizes fixed groups; it is a continuous controller with three distinct jobs. Provisioning is the fast path: it watches for unschedulable pods, batches the ones that arrive together, computes the cheapest instance type (from every type its NodePool allows) whose resources satisfy the batch, and calls EC2 CreateFleet to launch it — a GPU node is usually Ready in about a minute. Consolidation is the cost engine: on the disruption policy, Karpenter continuously asks “could this workload run on cheaper or fewer nodes?” and acts — deleting empty nodes, replacing an underutilized node with a smaller one, or bin-packing pods off a node so it can be removed. Drift is the correctness engine: when a node’s real configuration no longer matches its NodePool/EC2NodeClass (someone bumped the AMI alias, changed requirements, or expireAfter elapsed), Karpenter marks it drifted and rolls it, launching the replacement before draining the old one. All three respect PodDisruptionBudgets, the karpenter.sh/do-not-disrupt pod annotation, and disruption budgets on the NodePool that cap how many nodes churn at once — essential when each node is an expensive GPU you do not want mass-terminated.

Spot GPUs and interruption handling

Spot GPUs are 60-70% cheaper and can be reclaimed with two minutes’ notice — a bad trade for an in-flight 90-second summary unless you handle it. Karpenter watches an SQS interruption queue fed by EventBridge (Spot interruption warnings, rebalance recommendations, scheduled maintenance, instance state changes); on a warning it cordons and drains the doomed node and provisions a replacement before the two minutes expire, so pods reschedule rather than vanish. Layer on the standard mitigations: keep interactive traffic on On-Demand and reserve Spot for batch (the split the lesson uses); diversify across instance families so one Spot pool drying up does not take the fleet with it (Karpenter’s price-capacity-optimized allocation already biases toward deep, cheap pools); set a terminationGracePeriod and PDBs so drains are graceful; and let the gateway retry idempotent requests on a surviving replica.

Topology spread and zonal reality

GPU capacity is not uniform across Availability Zones — a Spot p4d pool can be deep in us-east-1a and empty in us-east-1c on the same afternoon. For availability you still want replicas spread, via topologySpreadConstraints:

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway     # prefer spread, don't block on scarce GPU AZs
    labelSelector:
      matchLabels: { app: clinical-summary }

Karpenter honours these when it provisions — it will launch nodes in the zones needed to satisfy the spread. The deliberate choice is whenUnsatisfiable: DoNotSchedule gives you strict HA but can wedge a pod Pending when the only free GPUs are in the “wrong” zone; ScheduleAnyway prefers spread yet still places the pod when capacity is one-sided — usually the right call for scarce accelerators. (For the general mechanics of spread, affinity, and preemption, see the scheduling term in the glossary; here the point is that GPU scarcity makes the soft constraint the pragmatic default.)

Right-sizing and the cost arithmetic

The cost case is one number: GPU utilization. A fleet averaging 25% is one you are overpaying for by 3-4x, and three levers push it up. Right-sizing — Karpenter picking an L4 for a 7B model instead of an idling A100 — stops you paying H100 prices for a small model. Consolidation keeps the live nodes near-saturated instead of half-empty. Batching (below) multiplies the tokens each live GPU produces. The trap is measuring the wrong thing: a node at 90% memory can still be at 10% compute if the KV cache is full of near-idle sequences, so watch DCGM’s SM-activity and tokens-per-second, not just the memory figure from nvidia-smi. This platform’s EKS-scale networking, IRSA, and node-provisioning patterns are covered in depth in EKS at scale — Pod Identity, Karpenter & networking.

Batching and the KV cache, from the throughput side

Why does vLLM beat a plain server several-fold? Two ideas. Continuous (iteration-level) batching: an LLM generates one token per forward pass for every sequence in flight, so instead of running fixed batches that wait for the slowest request, vLLM re-forms the batch every step — a finished sequence leaves, a newly arrived one joins — keeping the GPU’s matrix units fed. The KV cache: generating token N needs the attention keys and values of tokens 1…N-1; recomputing them each step would be quadratic, so they are cached in VRAM. That cache is large and grows with sequence length × concurrent sequences, and it competes with the weights for memory — which is exactly why --max-model-len (caps context, bounding per-sequence cache) and --gpu-memory-utilization (default 0.9, the VRAM fraction for weights + cache) are the knobs that decide how many requests fit before an OOM. vLLM’s PagedAttention stores the cache in fixed-size blocks like OS virtual memory, which kills fragmentation, lets sequences share blocks (prefix caching for a shared system prompt), and is what lets one GPU hold far more concurrent sequences than a contiguous-cache server. The tradeoff to remember: bigger batches raise throughput (tokens/sec/GPU, your cost story) but also raise per-request latency (time-to-first-token and inter-token latency, your SLO story) — you tune the batch to the SLO, not to the benchmark.

Enterprise considerations

Security & Zero Trust. The architecture is Zero Trust by construction: private API endpoint, no public ingress to the serving plane, identity-based access only, least-privilege IRSA per workload, and model weights that never leave the account. Layer on top: (a) Wiz running continuous CSPM and attack-path analysis across the EKS cluster and S3, alerting the moment a security group opens or a bucket policy drifts toward public, with Wiz Code scanning the Terraform and KServe manifests in the pull request so a misconfiguration is caught before merge, not after exposure; (b) CrowdStrike Falcon sensors as a DaemonSet on every GPU node for runtime threat detection and container-drift alerts, feeding the payer’s SOC — important because GPU nodes run third-party model and CUDA images that warrant runtime scrutiny; © prompt-injection and PHI-leak guardrails at the gateway, since a malicious instruction hidden in an uploaded packet is the LLM-specific attack here; (d) an SLO breach or a blocked-injection event auto-raises a ServiceNow incident so security and operations have a ticket, not just a metric. KMS encrypts weights at rest, and the S3 Object Lock on the registry means a published, validated model version is immutable — you cannot silently swap the weights an auditor signed off on.

Cost optimization. GPU spend dominates and idle GPUs are the leak, so engineer utilization from day one.

Lever	Mechanism	Typical effect
Scale-to-zero	KServe `minReplicas: 0` drains a model with no traffic; Karpenter reclaims the node	Eliminates off-peak GPU cost entirely
Spot GPUs	Karpenter NodePool Spot-first with On-Demand fallback for batch-tolerant traffic	~60-70% off the GPU hour on the Spot share
Node consolidation	Karpenter bin-packs and replaces underutilized nodes	Keeps live GPUs near-saturated, not half-idle
Right-sized instance	Karpenter picks the cheapest GPU that fits the model (L4 vs L40S vs A100)	Stops paying H100 prices for a 7B model
Continuous batching	vLLM keeps the GPU fed, raising tokens/sec per GPU	Lowers cost-per-million-tokens several-fold
Token chargeback	Gateway meters tokens per team; piped to Datadog	Makes each desk own its GPU spend

Meter tokens per team at the gateway and pipe the metric to Datadog, which the platform team uses for the per-cost-center chargeback dashboard the CFO sees. The single highest-leverage number is GPU utilization: a fleet averaging 25% is a fleet you are overpaying for by 3-4x, and consolidation plus batching plus scale-to-zero exist to drive it up.

Scalability. Each loop scales independently and on its own signal. KServe scales replicas on request concurrency (KPA/KEDA), from zero up to its ceiling. Karpenter scales GPU nodes on pending pods, choosing instance types live. vLLM scales throughput within a GPU via batching, and across GPUs via --tensor-parallel-size for a model too large for one card (a 70B model sharded across 4 GPUs on a p4d). The natural ceilings are your GPU service quota in the region and Spot availability for the chosen instance family — which is why a payer planning a Monday surge requests quota and configures On-Demand fallback early, rather than discovering the ceiling during an incident.

Failure modes, and what each one looks like. Name them before they page you.

Cold-start latency on a scaled-to-zero model — the first request after a lull waits for Karpenter to launch a node and for vLLM to download 30+ GB of weights and load them into GPU memory: minutes, not milliseconds. Mitigation: keep minReplicas: 1 (a warm pool) for latency-critical models and reserve scale-to-zero for batch ones; pre-bake weights into the AMI or use a faster registry pull; size the GPU so weights fit without paging.
Spot GPU reclamation mid-request — AWS reclaims a Spot node with two minutes’ notice and in-flight requests on it fail. Mitigation: Karpenter drains on the interruption signal; the gateway retries idempotent requests on another replica; keep interactive traffic on On-Demand and Spot for batch.
GPU quota exhaustion at peak — Karpenter cannot launch the node the pending pod needs because the regional GPU quota or Spot pool is dry, and requests queue. Mitigation: pre-request quota for peak, configure On-Demand fallback and a second instance family, and alert on sustained Pending GPU pods.
OOM / KV-cache exhaustion — concurrency or prompt length exceeds GPU memory and vLLM rejects or degrades. Mitigation: cap --max-model-len, tune gpu-memory-utilization, and let KServe’s concurrency target shed load to a new replica rather than overloading one.
A bad model version promoted — a regression ships and answers degrade. Mitigation: the offline eval gate, KServe canary traffic splitting to validate a new version on a slice of live traffic, and instant rollback by reverting the GitOps manifest.

Reliability & DR (RTO/RPO). Decide the numbers per tier. The model registry in S3 (versioned, Object-Locked, cross-region-replicated) is the durable source of truth with near-zero RPO — a model version, once published, is recoverable indefinitely. The serving plane itself is stateless: there is no conversation state on a GPU node, so DR is “stand the platform back up in a paired region,” which Terraform and Argo CD make a rebuild, not a restore. A pragmatic target for this platform: RTO 30 minutes to bring the serving plane up in a second region (cluster + Karpenter + Argo sync, gated by GPU quota in that region, which you pre-request), RPO effectively zero for models given S3 replication. Akamai health checks drive edge failover for ingress. The honest caveat: GPU capacity in the failover region is itself a dependency — DR for a GPU platform is partly a quota exercise, not only an automation one.

Observability. Instrument the request end to end in Datadog with APM: one trace covering gateway → KServe router → vLLM, with token counts and per-hop timing. Scrape DCGM GPU metrics so the dashboards show GPU utilization, memory used, temperature, and tokens-per-second per GPU alongside the request metrics. Define explicit latency SLOs — p95 time-to-first-token (what an adjudicator feels in a stream) and end-to-end p95 — as Datadog SLO monitors, with the error budget visible. Emit the business metrics that matter: GPU-hours per team, tokens and cost per cost-center, scale-to-zero/cold-start frequency, Spot-reclamation rate, and eval scores per model version. An SLO burn or a sustained GPU-saturation alert auto-raises a ServiceNow incident; a new model version passes a ServiceNow change approval before promotion, giving compliance a documented gate.

Governance. Pin model versions explicitly by immutable S3 prefix (clinical-summary/v7, never a latest alias) so behavior does not drift, and promote new versions only through the eval gate and the ServiceNow change. Keep KServe and gateway manifests in version control under Argo CD, reviewable and instantly revertable. Log every prompt/response pair (with PHI handled under the same HIPAA controls and a retention/right-to-erasure path) for audit, incident review, and future eval data. Wiz Code enforces that no infrastructure change ships with a public-exposure or open-IAM regression, and Wiz independently verifies in production that the controls are actually holding.

Explicit tradeoffs

Accept these or do not build it. Self-hosting LLMs trades a managed API’s simplicity for control and data residency, and the bill is real: you now operate Kubernetes, GPU drivers, a model registry, an autoscaling stack, and a serving engine — none of which a hosted API made you think about. Cold starts are the price of scale-to-zero: you cannot have both zero idle cost and instant first-token latency on the same model, so you choose per model (warm pool for interactive, scale-to-zero for batch). Spot is the price of cheap GPUs: you accept mid-request reclamation in exchange for 60-70% off, and you keep interactive traffic on On-Demand. And open-weight models on your hardware will, for the hardest reasoning, trail the largest frontier hosted models — you trade some capability ceiling for the right to keep PHI in your account, which for this payer is the entire point and not a regret.

The alternatives, and when they win. If your data is not regulated and you can send it out, a hosted model API is dramatically less to operate and usually the right call until governance, scale, or cost says otherwise. If your inference is sporadic and bursty with no latency floor, a serverless GPU offering or batch-only jobs may beat a standing platform. If you need only one model at modest, steady traffic, a managed inference endpoint (e.g. SageMaker) skips the Kubernetes layer entirely. Graduate to this full EKS-plus-Karpenter platform when you must self-host for residency, run multiple models with independent scaling, drive GPU cost down through consolidation and scale-to-zero, and hold the whole thing to auditable SLOs — which is exactly the payer’s situation.

The shape of the win

For the payer’s clinical-operations team, the payoff is not “an internal chatbot.” It is that an adjudicator opens a 40-page prior-auth packet, gets a structured summary with extracted codes and a policy-grounded medical-necessity draft in a few seconds, and — because the model ran on the payer’s own GPUs inside the VPC and no member record ever left the account — the governance board cleared it for PHI workloads, which a hosted API would never have passed. That clearance is what funds the platform. Everything upstream — Karpenter reclaiming idle GPUs, vLLM saturating the ones that remain, KServe scaling to zero overnight, the S3 registry’s immutable versions, the token-aware gateway’s chargeback, the Vault-held keys, the Wiz posture checks, the CrowdStrike runtime sensors, the Datadog GPU-utilization and latency SLOs — exists to make a CISO, a compliance officer, and a CFO each say yes. The architecture here is the destination; start with a single warm model behind the gateway if you must, but a regulated, at-scale, cost-disciplined “run our own LLMs” lands here.

Practice challenges

Work these in order; each Solution gives the command or manifest plus a one-line why. Assume an EKS cluster with Karpenter (v1), KServe, and the NVIDIA device plugin already installed.

1. (Beginner) See the GPU as a resource. Confirm that a node is actually advertising GPUs to Kubernetes, and how many.