Production-Grade AKS: Networking, Ingress, and Observability

A demo az aks create gets you a cluster; it does not get you a production cluster. This guide covers the four decisions that actually matter at scale: networking model, identity, ingress, and observability — with the commands and manifests to implement each.

In a nutshell

Level: Advanced · Time: ~34 min

Here is the whole idea in one breath: a production AKS cluster is decided by two things you cannot easily change later — the shape of its network and the depth of its observability. Compute you can add. Nodes you can scale. But the networking model is baked in at create time, and if you cannot see the cluster when it misbehaves at 3 a.m., every other decision was academic. Get the network model, the front door, and the metrics/logs/traces stack right on day one, and everything else — GitOps, mesh, progressive delivery — layers on cleanly.

Think of the cluster as a new office building. The networking model is the plumbing and the address plan: how many rooms (pods) you can fit before you run out of addresses, whether rooms can be reached directly from the street, and which corridors connect to which. Pick the wrong plan and you discover, halfway through fit-out, that the building is “full” even though most rooms are empty — not because you ran out of space, but because you ran out of addresses. Ingress is the single guarded front door — one reception desk (with a TLS badge check) that routes every visitor to the right floor, instead of a hundred unlocked side doors. And observability is the building’s sensors and CCTV: the metrics (is the AC running hot?), the logs (who opened which door, and what broke), and the traces (follow one visitor’s whole journey through the building). A building with great plumbing and no sensors is one you cannot operate; a building with sensors and bad plumbing is one you cannot grow.

The three observability signals are worth naming up front because people conflate them. Metrics are cheap numbers over time (CPU %, request rate, error rate) — great for dashboards and alerts. Logs are expensive lines of text (what exactly happened) — great for forensics, but they cost real money per gigabyte. Traces stitch one request’s path across many services — great for “where did the latency come from?”. Managed Azure gives you a first-party home for all three: managed Prometheus for metrics, Container Insights / Log Analytics for logs, and Application Insights (via OpenTelemetry) for traces.

If managed Kubernetes as a category is new to you, skim Managed Kubernetes compared: AKS vs EKS vs GKE first, and if pod networking fundamentals are hazy, the CNI pod-networking model is the prerequisite for the IP-planning math below. This lesson assumes you can already read a Deployment, a Service, and an Ingress manifest.

After this lesson you will be able to:

• Choose between kubenet, Azure CNI, and Azure CNI Overlay for a reason, and do the IP-address math before you provision, not after InsufficientSubnetSize.
• Front your services with a managed ingress and automatic TLS, and explain when App Gateway / AGC beats managed NGINX.
• Lock down east-west traffic with a default-deny network policy, and pick the right policy engine (Azure NPM, Calico, or Cilium).
• Make the API server private and route egress through a firewall or NAT gateway without breaking the cluster.
• Wire up managed Prometheus, Managed Grafana, Container Insights, and control-plane logs — and alert on SLOs, not on noise.

Read the diagram left-to-right as the life of one request: (1) a client hits the managed NGINX ingress where TLS is terminated; (2) ingress routes to a Service, which lands on pods running on an autoscaling user node pool; (3) those pods pull images from ACR and fetch Entra tokens via Workload Identity with no stored secrets; (4) managed Prometheus + Grafana scrape metrics and fire alerts on the golden signals; and (5) east-west traffic between pods is refused by default-deny NetworkPolicy unless explicitly allowed, with HA from PodDisruptionBudgets and topology spread. Every section below implements one arrow of that path.

1. Networking: choose Azure CNI Overlay

Three models, one right default for most:

Before the commands, understand why this is the highest-stakes decision. In AKS the CNI (Container Network Interface) plugin decides where every pod’s IP address comes from, and that choice determines how many pods your cluster can ever hold, whether those pods are reachable from outside, and how much of your precious VNet address space you burn. It is set at create time and, with one exception, cannot be changed without disruption. So the full picture, expanded:

The reason Overlay is the default is the IP math, and it is worth doing by hand once so it never surprises you.

IP planning: the math that causes InsufficientSubnetSize

With classic Azure CNI, every node pre-reserves maxPods IP addresses from the node subnet plus one for itself — whether or not those pods exist yet. So the addresses a node subnet must hold is roughly:

subnet IPs needed ≈ (max nodes during scale + upgrade surge) × (1 + maxPods)

Work a concrete case. A /24 subnet has 256 addresses, of which Azure reserves 5, leaving 251 usable. Provision classic CNI with the common maxPods = 110:

251 usable ÷ (1 + 110) = 2.26  →  only 2 nodes fit

Two nodes. On a /24. That is the trap the enterprise scenario below hit — the cluster reports it is “full” while CPU sits idle, because the subnet, not compute, is exhausted. Even at the CLI default maxPods = 30, a /24 tops out at 251 ÷ 31 ≈ 8 nodes. And you must leave headroom for the upgrade surge (AKS adds an extra node while rolling), so the real ceiling is lower still.

Now the same /24 with Azure CNI Overlay: nodes draw from the VNet subnet, but pods draw from a separate /16 overlay CIDR that never touches the VNet.

node subnet: 251 usable ÷ 1 per node ≈ 250 nodes
pod space:   10.244.0.0/16 = 65,536 pod IPs, entirely off-VNet

Roughly 250 nodes on the exact same subnet, versus 2. The overlay CIDR (--pod-cidr) can be any private range that does not overlap the VNet, on-prem, or peered networks, and it is reused per cluster — pod IPs are only meaningful inside the cluster. That is the whole pitch: VNet-scale node counts without spending VNet IPs on pods. For the datapath details of how packets actually move in each model, see the CNI pod-networking model internals.

The one real cost of Overlay: pod IPs are not routable from outside the cluster. If an on-prem system must reach a pod by its IP (rare, and usually a design smell), Overlay breaks that path — put the workload behind an internal load balancer or the ingress instead. For everything else, Overlay is correct.

Provision with Overlay + Cilium data plane (eBPF) via Terraform:

resource "azurerm_kubernetes_cluster" "prod" {
  name                = "aks-prod-eus"
  location            = "eastus"
  resource_group_name = azurerm_resource_group.aks.name
  dns_prefix          = "aksprod"
  oidc_issuer_enabled       = true   # required for workload identity
  workload_identity_enabled = true

  default_node_pool {
    name                 = "system"
    vm_size              = "Standard_D4ds_v5"
    auto_scaling_enabled = true
    min_count            = 3
    max_count            = 6
    only_critical_addons_enabled = true   # taint system pool; run apps elsewhere
  }

  network_profile {
    network_plugin      = "azure"
    network_plugin_mode = "overlay"
    network_policy      = "cilium"
    network_data_plane  = "cilium"
    pod_cidr            = "10.244.0.0/16"
    service_cidr        = "10.0.0.0/16"
    dns_service_ip      = "10.0.0.10"
  }
  identity { type = "SystemAssigned" }
}

Three fields here decide the network and deserve a plain-language gloss. network_plugin_mode = "overlay" is what turns on Overlay. pod_cidr is the off-VNet pool pods draw from. service_cidr is a separate, virtual range for ClusterIP Services — it never appears on the wire, it is just the pool kube-proxy/Cilium hands out for stable service VIPs — and dns_service_ip (here 10.0.0.10) is the address of the in-cluster DNS service, which must sit inside service_cidr. Pod CIDR, service CIDR, and your VNet must not overlap each other or anything peered.

Add a separate user node pool for workloads so system add-ons never compete with apps:

az aks nodepool add -g rg-aks --cluster-name aks-prod-eus \
  --name apps --mode User --node-vm-size Standard_D8ds_v5 \
  --enable-cluster-autoscaler --min-count 3 --max-count 20

The only_critical_addons_enabled = true on the system pool taints it (CriticalAddonsOnly=true:NoSchedule) so CoreDNS, metrics-server, and the CSI drivers get guaranteed room and your app pods land on the apps pool. This separation is the difference between “a node pressure event took out CoreDNS and my app” and “it only touched the app.”

DNS inside the cluster

Every name lookup a pod makes — postgres, payments.svc, api.stripe.com — goes to CoreDNS, which runs as a Deployment in kube-system and answers on the dns_service_ip you set above. Two AKS-specific facts matter in production. First, CoreDNS is a shared, load-bearing dependency: if it is starved for CPU or evicted under node pressure, the whole cluster looks broken with confusing timeouts — which is exactly why the system-pool taint above exists. Second, for high-QPS clusters enable NodeLocal DNSCache, a per-node DNS cache that answers most lookups on the node itself, cutting CoreDNS load and tail latency. Custom stub domains and upstream forwarders (e.g., to a private DNS zone or on-prem resolver) are configured through the coredns-custom ConfigMap, not by editing the managed CoreDNS config directly — AKS reconciles the managed config and would revert your edits.

2. Identity: Workload Identity, not secrets

Stop mounting service-principal secrets. Microsoft Entra Workload Identity federates a Kubernetes service account to a managed identity — pods get Entra tokens with no secrets to rotate.

# 1) create a user-assigned managed identity and give it access (e.g., Key Vault)
az identity create -g rg-aks -n id-payments
az role assignment create --assignee <id-client-id> \
  --role "Key Vault Secrets User" --scope <keyvault-resource-id>

# 2) federate it to the k8s service account
az identity federated-credential create --name payments-fed \
  --identity-name id-payments -g rg-aks \
  --issuer "$(az aks show -g rg-aks -n aks-prod-eus --query oidcIssuerProfile.issuerUrl -o tsv)" \
  --subject system:serviceaccount:payments:payments-sa

apiVersion: v1
kind: ServiceAccount
metadata:
  name: payments-sa
  namespace: payments
  annotations:
    azure.workload.identity/client-id: "<id-client-id>"
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: payments, namespace: payments }
spec:
  template:
    metadata:
      labels: { azure.workload.identity/use: "true" }   # injects the token
    spec:
      serviceAccountName: payments-sa

The mechanism, in one paragraph, because it is the crux of secret-free Azure access. AKS exposes an OIDC issuer — a public endpoint that signs a short-lived token proving “I am the payments-sa service account in the payments namespace.” You register that claim (the --subject system:serviceaccount:payments:payments-sa) as a federated credential on a managed identity. Now when the pod presents its projected service-account token, Entra ID checks the signature and the subject, sees the federation you configured, and hands back an Entra access token for the managed identity — which has the RBAC you granted (here, Key Vault Secrets User). There is no secret in the cluster: nothing to leak in a git commit, nothing to rotate, nothing that outlives the pod. The mutating webhook from the workload-identity add-on injects the token file and the AZURE_* environment variables whenever it sees the azure.workload.identity/use: "true" pod label — which is why that label, not just the service account, is required.

3. Ingress: managed app routing + cert-manager

Enable the AKS-managed NGINX ingress (app routing add-on) so you don’t operate the controller yourself:

az aks approuting enable -g rg-aks -n aks-prod-eus

Then expose a service with TLS (cert-manager issuing Let’s Encrypt certs):

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: payments
  namespace: payments
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  ingressClassName: webapprouting.kubernetes.azure.com
  tls:
    - hosts: ["pay.kloudvin.com"]
      secretName: pay-tls
  rules:
    - host: pay.kloudvin.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend: { service: { name: payments, port: { number: 80 } } }

Choosing an ingress: managed NGINX vs App Gateway vs AGC

“Ingress” is the cluster’s front door, and Azure gives you several. The right pick depends on where you want TLS terminated, whether you need a WAF, and whether you have adopted the Gateway API. The comparison:

The mental split: NGINX-family (add-on or self-managed) terminates TLS and routes inside the cluster — simplest, most portable. App Gateway family (AGIC, and its successor AGC) puts a managed Azure L7 load balancer with a WAF in front of the cluster, offloading TLS and shielding the origin. AGIC uses the older Ingress API; AGC uses the Gateway API and is where Azure is investing — if you are choosing today and want traffic splitting or WAF, AGC is the forward-looking pick.

4. Network policy: default-deny

Lock down east-west traffic. With Cilium/Azure network policy, start every namespace at default-deny and open only what’s needed:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: default-deny, namespace: payments }
spec:
  podSelector: {}
  policyTypes: ["Ingress", "Egress"]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: allow-from-ingress, namespace: payments }
spec:
  podSelector: { matchLabels: { app: payments } }
  ingress:
    - from:
        - namespaceSelector: { matchLabels: { kubernetes.azure.com/managed: "true" } }
      ports: [{ port: 80 }]

The default posture in Kubernetes is default-allow: any pod can reach any other pod, the API server, and the cloud metadata endpoint. The two objects above invert that for the payments namespace — the first flips every pod to deny-all (an empty podSelector: {} selects them all, and listing both policyTypes with no rules denies both directions), the second re-permits exactly one flow: ingress from the AKS-managed ingress namespace to the payments app on port 80. The moment you enable egress deny, remember DNS is an egress call to CoreDNS on port 53 — you must allow it or every name lookup fails. The full default-deny playbook, including the DNS gotcha and Cilium L7/FQDN rules, lives in zero-trust pod networking.

Which policy engine: Azure NPM, Calico, or Cilium

A NetworkPolicy object only does something if the cluster’s network plugin enforces it. On AKS you choose the engine at create time (network_policy above), and the choice caps what you can express:

For a new production cluster, Azure CNI Overlay powered by Cilium (as in the Terraform above) is the strongest default: you get identity-based enforcement that survives pod churn, L7 rules (HTTP method/path and DNS FQDN egress), and Hubble flow visibility. Whichever you pick, always prove enforcement (apply a deny, confirm it drops) rather than assume the YAML did anything.

5. Observability: managed Prometheus + Grafana

Don’t self-host the metrics stack. Enable Azure Monitor managed Prometheus, Azure Managed Grafana, and Container Insights in one shot:

az aks update -g rg-aks -n aks-prod-eus \
  --enable-azure-monitor-metrics \
  --enable-azure-monitor-app-monitoring

# link a managed Grafana instance
az grafana create -g rg-aks -n graf-kloudvin
az aks update -g rg-aks -n aks-prod-eus --enable-azure-monitor-metrics \
  --grafana-resource-id $(az grafana show -g rg-aks -n graf-kloudvin --query id -o tsv)

Scrape your own app metrics by annotating pods and adding a PodMonitor:

apiVersion: azmonitoring.coreos.com/v1
kind: PodMonitor
metadata: { name: payments, namespace: payments }
spec:
  selector: { matchLabels: { app: payments } }
  podMetricsEndpoints: [{ port: metrics, interval: 30s }]

The four signals to alert on first (the “golden signals”): latency, traffic, errors, saturation. Wire alerts in Grafana → Azure Monitor action groups.

The three signals: metrics, logs, and traces

Beginners reach for “logs” for everything and then drown in cost. Production observability is three different tools for three different questions:

The winning pattern: alert on metrics (cheap, real-time), drill into logs only when an alert fires (expensive, so keep them lean), and use traces to localize which service in a call chain is slow. Sending everything to logs and eyeballing dashboards is how observability bills balloon into the tens of thousands.

Managed Prometheus + Managed Grafana

--enable-azure-monitor-metrics deploys the managed Prometheus collector: an ama-metrics replica and ama-metrics-node DaemonSet land in kube-system, scrape targets per a Data Collection Rule (DCR), and remote-write into an Azure Monitor workspace (not a Log Analytics workspace — metrics and logs have separate stores). You do not run Prometheus servers, manage its storage, or page yourself when its PVC fills. Managed Grafana is the query/dashboard front end, pre-wired to that workspace with the standard Kubernetes dashboards. Your app metrics get in by the PodMonitor/ScrapeConfig CRDs above — note the CRD group is azmonitoring.coreos.com/v1, the Azure-managed flavor, not the community monitoring.coreos.com. If you already know the Prometheus/Grafana stack, the concepts are identical to self-hosted Prometheus & Grafana — you are just outsourcing the servers and storage.

Container Insights, and controlling its log cost

Container Insights (enabled via the monitoring add-on) deploys the ama-logs DaemonSet, which tails every container’s stdout/stderr plus node inventory into Log Analytics, billed per GB ingested. On a chatty cluster this is the line item that surprises finance:

# enable Container Insights, pointed at an existing Log Analytics workspace
az aks enable-addons -a monitoring -g rg-aks -n aks-prod-eus \
  --workspace-resource-id $(az monitor log-analytics workspace show \
     -g rg-aks -n la-kloudvin --query id -o tsv)

Three levers keep the bill sane: (1) use a cost-preset / data collection setting to exclude noisy namespaces (kube-system, log spam) and drop stdout you do not need; (2) put high-volume, rarely-queried tables like ContainerLogV2 on the Basic Logs tier, which trades ad-hoc query power for a fraction of the ingestion price; and (3) set retention deliberately — 30 days hot, archive beyond. Turning on Container Insights with defaults and no exclusions is the single most common way to overspend on an AKS cluster.

Control-plane and diagnostic logs

Managed Prometheus and Container Insights cover your nodes and pods. The control plane — the API server, scheduler, controller-manager, and the audit log — is managed by Azure and its logs are off by default. Turn them on with a diagnostic setting so you can answer “who deleted that deployment?” and “why did the scheduler refuse to place this pod?”:

az monitor diagnostic-settings create \
  --name aks-diag --resource $(az aks show -g rg-aks -n aks-prod-eus --query id -o tsv) \
  --workspace $(az monitor log-analytics workspace show -g rg-aks -n la-kloudvin --query id -o tsv) \
  --logs '[
    {"category":"kube-apiserver","enabled":true},
    {"category":"kube-audit-admin","enabled":true},
    {"category":"kube-controller-manager","enabled":true},
    {"category":"kube-scheduler","enabled":true},
    {"category":"cluster-autoscaler","enabled":true},
    {"category":"guard","enabled":true}
  ]'

Prefer kube-audit-admin over the full kube-audit category in production — the latter includes every read (get/list/watch) and is enormous; the admin variant keeps the writes and privileged reads that matter for forensics at a fraction of the volume.

Traces and OpenTelemetry

Metrics and logs tell you a service is slow; traces tell you which. Instrument apps with OpenTelemetry (the vendor-neutral standard), emit OTLP, and export to Application Insights — either directly from the SDK or through an OpenTelemetry Collector DaemonSet that batches and routes. The Collector pattern is worth the extra hop at scale: it decouples your app from the backend, does tail-based sampling, and can fan the same spans to Application Insights for traces and managed Prometheus for the metrics OTel also produces. A single traceparent header propagated across services is what lets Application Insights draw the end-to-end map from ingress to database.

Alerting and SLOs

Dashboards are for humans watching; alerts are for humans not watching. Do not alert on raw CPU — alert on SLO burn. Define a Service Level Objective (e.g., “99.9% of payments requests succeed under 300 ms over 30 days”), then alert when your error budget is burning fast enough to blow the objective. Managed Prometheus supports Prometheus rule groups (recording + alerting rules) defined as Azure resources; wire the alerts to Azure Monitor action groups (PagerDuty, email, webhook) and/or Grafana alerting. Start with the four golden signals per service — latency, traffic, errors, saturation — as recording rules, then layer SLO burn-rate alerts on top. Alerting on symptoms users feel (latency, errors) rather than causes (CPU, memory) is what separates a pager that means something from one everyone mutes.

Hands-on lab

No cluster? Everything below is schema-correct and safe to author now; the kubectl/az commands run against a real AKS cluster when you have one. The goal: provision an Overlay+Cilium cluster, front an app with managed ingress + TLS, lock it down, and prove observability is on. Outputs are representative (from a comparable cluster), labelled as such.

Step 1 — provision (or inspect) the cluster. Confirm the network model is what you think it is:

az aks show -g rg-aks -n aks-prod-eus \
  --query '{plugin:networkProfile.networkPlugin, mode:networkProfile.networkPluginMode, \
            dataplane:networkProfile.networkDataplane, podCidr:networkProfile.podCidr}' -o table

# representative output
Plugin    Mode     Dataplane    PodCidr
--------  -------  -----------  ---------------
azure     overlay  cilium       10.244.0.0/16

Step 2 — deploy a demo app on the user node pool and expose it via the app-routing ingress with cert-manager TLS (reuse the Ingress manifest from section 3). Verify placement and address:

kubectl get pods -n payments -o wide            # land on the 'apps' pool, not 'system'
kubectl get ingress -n payments                 # ADDRESS populated, HOSTS pay.kloudvin.com

# representative output
NAME                        READY   STATUS    NODE
payments-6c9f5d4b7c-n2k8p   1/1     Running   aks-apps-2245...

NAME       CLASS                                    HOSTS               ADDRESS         PORTS
payments   webapprouting.kubernetes.azure.com       pay.kloudvin.com    20.55.x.x       80, 443

Step 3 — apply default-deny + the ingress allow (the manifests from section 4), then prove enforcement:

# same namespace, no allow rule → should be refused
kubectl -n payments run probe --image=nicolaka/netshoot --rm -it --restart=Never \
  -- curl -m 5 http://payments  ; echo "exit=$?"
# expect a timeout / non-zero exit once default-deny is on

Step 4 — confirm observability is live:

kubectl -n kube-system get ds ama-logs ama-metrics-node   # Container Insights + managed Prometheus agents
kubectl get podmonitor -n payments                        # your app scrape target exists
az aks show -g rg-aks -n aks-prod-eus --query azureMonitorProfile  # metrics profile enabled

Step 5 — validate the YAML before it ships (works with no cluster, catches typos):

kubectl apply --dry-run=server -f ingress.yaml -f netpol.yaml -f podmonitor.yaml

If step 3’s curl succeeds after you applied default-deny, either the policy did not select the pod or your engine is not enforcing — go back to section 4. If the ama-* DaemonSets are missing in step 4, the monitoring add-on/metrics profile did not enable; re-run the az aks update from section 5.

Going deeper

The sections above get you a correct cluster. This is for when you must defend the design in an architecture review, reason about failure modes, or keep the bill and the blast radius under control at scale.

The Overlay datapath, and why pods are not routable

In classic Azure CNI, a pod’s veth is bridged so its VNet IP is a first-class address the fabric routes — which is why pods are directly reachable and why they each cost a VNet IP. In Overlay, each node owns a slice of the pod CIDR and the CNI programs the datapath (with Cilium, eBPF at the node) to encapsulate/route pod traffic between nodes while pods draw from the off-VNet CIDR. Outbound to the VNet or internet is SNAT’d behind the node’s IP, so from the VNet’s perspective traffic appears to come from the node, never the pod. That single fact explains both Overlay’s superpower (pods cost nothing in VNet IPs) and its one limitation (nothing outside the cluster can dial a pod IP directly). It also means your subnet sizing shifts from pods to nodes: size the node subnet for max nodes + upgrade surge + a margin, and size the overlay /16 for pods — two independent budgets instead of one contended pool.

Private clusters and controlled egress

Two hardening steps turn a demo cluster into one a security team will sign off on: closing the front door of the control plane, and controlling what the cluster can talk out to.

Private API server. By default the Kubernetes API server has a public FQDN. --enable-private-cluster replaces it with a private endpoint (a private IP in your VNet via Private Link) resolved through a private DNS zone, so kubectl only works from inside the network:

az aks create -g rg-aks -n aks-prod-eus --enable-private-cluster \
  --network-plugin azure --network-plugin-mode overlay --network-dataplane cilium \
  # ...node pool + identity flags...

The trade-off is connectivity: your CI runners, jump host, or laptop must reach that private IP — via VPN, ExpressRoute, a peered management VNet, Azure Bastion + a jumpbox, or the escape hatch az aks command invoke (which runs a command server-side without network line-of-sight). If a private endpoint is too heavy, the middle ground is a public endpoint restricted by --api-server-authorized-ip-ranges to your egress IPs — less airtight than private, but far better than open-to-the-internet.

Controlled egress. By default AKS egresses through a public load balancer doing SNAT, which has two problems at scale: a limited pool of SNAT ports (exhaustion shows up as intermittent outbound connection failures under load) and no control over what the cluster can reach. The --outbound-type you choose fixes both:

The production-hardened pattern is userDefinedRouting sending egress to an Azure Firewall, with application/network rules allowing exactly the FQDNs AKS needs (Microsoft publishes the required set, and the AzureKubernetesService FQDN tag covers the platform ones) plus your own dependencies. Now the cluster can only reach approved destinations — a hard requirement for regulated workloads and a strong control against data exfiltration from a compromised pod.

The managed-Prometheus and Grafana architecture

Worth knowing where your metrics actually live, because the stores are separate and independently billed. ama-metrics scrapes targets (kube-state-metrics, cAdvisor/node metrics, and your PodMonitors) per the Data Collection Rule, then remote-writes into an Azure Monitor workspace — a purpose-built time-series store, distinct from the Log Analytics workspace where Container Insights sends logs. Managed Grafana queries the Azure Monitor workspace over PromQL. So a full production cluster typically has three data stores: an Azure Monitor workspace (metrics), a Log Analytics workspace (Container Insights + control-plane logs), and an Application Insights resource (traces). Keeping them straight is what lets you reason about cost — you tune each independently: retention on Log Analytics, sampling on App Insights, series cardinality on Prometheus.

The real cost of logs, and cardinality

Two bills bite silently. On the logs side, Container Insights charges per GB into Log Analytics; a single over-logging service (debug logs left on, a health-check spamming stdout) can dominate the invoice. Control it with namespace exclusions, the Basic Logs tier for high-volume tables, and disciplined app log levels. On the metrics side, the killer is cardinality: every unique combination of label values is a separate time series, and a well-meaning label like user_id or a raw path with IDs in it explodes one metric into millions of series, driving cost and slowing queries. Keep metric labels bounded and low-cardinality (status code, route template, method — never raw IDs), and prefer logs/traces for high-cardinality detail. “Metrics for aggregates, logs/traces for specifics” is a cost rule as much as a design one.

Version and API caveats

• kubenet is on a deprecation path. New clusters should not use it; plan migrations off existing kubenet clusters to Azure CNI Overlay.
• Overlay migration is one-way and disruptive. az aks update --network-plugin-mode overlay (as in the enterprise scenario) drains and recycles every node; run it in a window with PodDisruptionBudgets set, and know you cannot go back.
• Managed Prometheus CRDs are Azure-flavored (azmonitoring.coreos.com/v1). Community ServiceMonitor/PodMonitor from monitoring.coreos.com are not the same objects — use the Azure group with managed Prometheus.
• kube-audit volume. The full audit category is huge; prefer kube-audit-admin unless you specifically need read auditing for compliance.

Enterprise scenario

A fintech platform team migrated ~40 services onto a shared AKS cluster using classic Azure CNI. It worked in staging, then production node scale-out started failing with InsufficientSubnetSize. Classic CNI assigns every pod a real VNet IP, and with maxPods=110 per node a single /22 node subnet was exhausted well before the autoscaler hit its ceiling — the subnet, not compute, was the bottleneck. Re-IPing a peered hub-and-spoke VNet that other teams depended on was a non-starter.

The fix was migrating the data plane to Azure CNI Overlay, where pods draw from a private overlay CIDR and only nodes consume VNet IPs. This is an in-place cluster update, not a rebuild:

az aks update -g rg-aks -n aks-prod-eus \
  --network-plugin-mode overlay \
  --pod-cidr 10.244.0.0/16

Two gotchas bit them. First, the migration is one-way and requires draining and recycling every node, so they ran it during a window with PodDisruptionBudgets in place. Second, a legacy service depended on directly routable pod IPs from an on-prem caller; overlay pod IPs are not reachable outside the cluster, so that path had to move behind the internal load balancer. After the cutover the same /22 comfortably supported the full node count, and VNet IP consumption dropped from thousands of pod IPs to a few dozen node IPs — buying years of headroom without touching the hub network. Validate the new plane with kubectl get nodes -o wide and confirm pod CIDRs land in the overlay range.

Production readiness checklist

• Azure CNI Overlay + network policy (default-deny per namespace)
• Separate system/user node pools, both autoscaling
• Workload Identity — no service-principal secrets in cluster
• Managed ingress with automated TLS (cert-manager)
• Managed Prometheus + Grafana + Container Insights
• Pod Disruption Budgets + topology spread for HA
• Image scanning (Defender for Containers) + admission control
• Backup (Velero / AKS backup) + tested restore
• Private API server (or authorized IP ranges) + controlled egress (UDR/NAT)
• Control-plane diagnostic logs (kube-audit-admin) + SLO burn-rate alerts

Verify

kubectl get nodes -o wide                          # node pools healthy
kubectl get networkpolicy -A                       # default-deny present
kubectl describe sa payments-sa -n payments        # workload-identity annotation
kubectl get ingress -n payments                    # address assigned, TLS secret created
az aks show -g rg-aks -n aks-prod-eus --query azureMonitorProfile  # metrics enabled

With these five pillars in place you have a cluster that’s routable at VNet scale, secret-free, TLS-terminated, segmented, and observable. Everything else — GitOps with Argo CD, progressive delivery, service mesh, day-2 upgrade and backup discipline — layers cleanly on top.

Troubleshooting

Practice challenges

Author these against the manifests and commands above; each has a graded difficulty and a worked solution — try before you open the toggle.

Challenge 1 (beginner) — size the subnet. You must run up to 60 nodes on classic Azure CNI with maxPods = 30, plus upgrade surge. Which node subnet size do you need, and why is Overlay easier?

<details> <summary>Solution</summary>

Classic CNI needs ~(nodes + surge) × (1 + maxPods) IPs = ~61 × 31 ≈ 1,891 IPs, so a /21 (2,048 addresses, ~2,043 usable) — a /22 (1,019 usable) is too small. With Overlay the node subnet only needs ~61 IPs (a /25 is plenty) because pods live in the separate /16 overlay CIDR. Why: classic CNI charges VNet IPs per pod; Overlay charges per node. </details>

Challenge 2 (beginner) — don’t break DNS. You enable a namespace-wide default-deny including egress and the app starts timing out on every outbound call. Add the minimal rule to restore name resolution.

<details> <summary>Solution</summary>

Allow egress to CoreDNS in kube-system on UDP and TCP 53:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: allow-dns, namespace: payments }
spec:
  podSelector: {}
  policyTypes: ["Egress"]
  egress:
    - to:
        - namespaceSelector:
            matchLabels: { kubernetes.io/metadata.name: kube-system }
          podSelector:
            matchLabels: { k8s-app: kube-dns }
      ports:
        - { protocol: UDP, port: 53 }
        - { protocol: TCP, port: 53 }

Why: every name lookup is an egress call to CoreDNS on 53; both transports are needed because large answers fall back to TCP. </details>

Challenge 3 (intermediate) — pick the ingress. A regulated service must terminate TLS outside the cluster behind a WAF, and you want to adopt the Gateway API for weighted traffic splits. Which ingress option, and why not managed NGINX?

<details> <summary>Solution</summary>

Application Gateway for Containers (AGC) — a managed Azure L7 with WAF that terminates TLS in front of the cluster and is driven by the Gateway API, giving native weighted HTTPRoute splits. Managed NGINX terminates TLS inside the cluster and has no managed WAF, so it fails the “WAF outside the origin” requirement. (AGIC also offers WAF but uses the older Ingress API, not Gateway API.) Why: the requirement is edge WAF + Gateway API — exactly AGC’s remit. </details>

Challenge 4 (intermediate) — secret-free Key Vault access. A pod gets AuthorizationFailed reaching Key Vault despite a federated credential existing. Name the two most likely causes.

<details> <summary>Solution</summary>

(1) The pod is missing the azure.workload.identity/use: "true" label, so the webhook never injected the token/env. (2) The federated credential’s --subject does not exactly match system:serviceaccount:<namespace>:<serviceaccount>, or the RBAC role assignment (Key Vault Secrets User) is missing/scoped wrong. Why: workload identity needs the label and an exact subject match and the target RBAC — all three. </details>

Challenge 5 (advanced) — alert on SLO, not CPU. Your team pages on “node CPU > 80%” and everyone mutes it. Redesign the alert around a user-facing SLO for the payments service.

<details> <summary>Solution</summary>

Define an SLO (e.g., 99.9% of requests succeed under 300 ms over 30 days), record the golden signals as Prometheus rules (request rate, error rate, p99 latency), then alert on error-budget burn rate — page fast-burn (e.g., 2% of the 30-day budget in 1 hour) and ticket slow-burn. Wire to an Azure Monitor action group. Why: CPU is a cause, not a symptom users feel; burn-rate alerts fire only when the objective is genuinely at risk, so they stay trustworthy. </details>

Challenge 6 (advanced) — lock down egress. A compromised pod must not be able to exfiltrate to arbitrary internet hosts, and the cluster must only reach an approved set of FQDNs. Outline the AKS design.

<details> <summary>Solution</summary>

Set the cluster’s --outbound-type userDefinedRouting so a UDR forces all egress through an Azure Firewall; on the firewall, allow the AKS platform FQDNs (the AzureKubernetesService FQDN tag) plus your explicit application FQDN allow-list, and deny the rest. Layer a Cilium egress NetworkPolicy (FQDN/toFQDNs) inside the cluster for defense in depth. Why: the firewall gives a central, inspectable choke point that no single namespace policy can bypass, and egress is now allow-listed rather than open. </details>

Common beginner mistakes

• Using kubenet (or classic CNI) and hitting the IP wall. kubenet is on a deprecation path and classic CNI burns a VNet IP per pod — do the (nodes + surge) × (1 + maxPods) math before provisioning, and default to Azure CNI Overlay so pods cost nothing in VNet IPs. The failure mode (InsufficientSubnetSize with idle CPU) always arrives in production, never in the demo.
• No network policy at all. A default AKS cluster is default-allow: one compromised pod can reach every other pod, the API server, and the metadata endpoint. “We’ll add segmentation later” means you never do. Ship a default-deny per namespace from day one and open only intended flows.
• Turning on Container Insights with defaults and no cost controls. It ingests every namespace’s stdout into Log Analytics at per-GB pricing; a chatty service can dominate the bill. Apply a cost preset, exclude noisy namespaces, use the Basic Logs tier for high-volume tables, and set retention deliberately.
• Leaving the API server public and open. The control plane on a public FQDN with no IP restriction is an unnecessary attack surface. Use --enable-private-cluster, or at minimum --api-server-authorized-ip-ranges scoped to your egress IPs.
• Alerting on causes (CPU/memory) instead of SLOs. Resource-threshold alerts are noisy and get muted, so real incidents hide in the noise. Alert on user-facing symptoms — latency, error rate, SLO burn rate — and keep resource metrics for dashboards and capacity planning.
• Assuming a NetworkPolicy is enforced because it applied. kubectl apply succeeds even if the engine ignores the object. Pick an enforcing engine (Cilium/Calico/Azure NPM) and prove a deny drops traffic before you trust the posture.
• Forgetting DNS when locking egress. The instant egress is default-deny, name resolution to CoreDNS stops and the app fails with confusing timeouts. Always ship the DNS allow (UDP+TCP 53 to kube-dns) in the same change as any default-deny-egress.
• Running apps on the system node pool. Without only_critical_addons_enabled, your app pods and CoreDNS/CSI compete for the same nodes, so a noisy app can starve the add-ons that keep the cluster alive. Keep a tainted system pool and a separate user pool for workloads.

Cheat-sheet

# --- Networking / cluster ---
az aks show -g rg-aks -n aks-prod-eus \
  --query 'networkProfile.{plugin:networkPlugin,mode:networkPluginMode,dataplane:networkDataplane}'
az aks update -g rg-aks -n aks-prod-eus --network-plugin-mode overlay --pod-cidr 10.244.0.0/16  # 1-way!
kubectl get nodes -o wide                                  # node pools + IPs
az aks nodepool list -g rg-aks --cluster-name aks-prod-eus -o table

# --- Identity (workload identity) ---
az aks show -g rg-aks -n aks-prod-eus --query oidcIssuerProfile.issuerUrl -o tsv
az identity federated-credential list --identity-name id-payments -g rg-aks -o table
kubectl describe sa payments-sa -n payments                # check client-id annotation

# --- Ingress ---
az aks approuting enable -g rg-aks -n aks-prod-eus
kubectl get ingress -A                                     # ADDRESS + HOSTS + PORTS
kubectl get ingressclass                                   # webapprouting.kubernetes.azure.com

# --- Network policy ---
kubectl get networkpolicy -A
kubectl get ciliumnetworkpolicy -A                         # if Cilium engine

# --- Observability ---
az aks update -g rg-aks -n aks-prod-eus --enable-azure-monitor-metrics   # managed Prometheus
az aks enable-addons -a monitoring -g rg-aks -n aks-prod-eus \
  --workspace-resource-id <la-id>                          # Container Insights
kubectl -n kube-system get ds ama-logs ama-metrics-node    # agents present
kubectl get podmonitor -A                                  # scrape targets (azmonitoring.coreos.com/v1)

# --- Private cluster / egress ---
az aks create ... --enable-private-cluster                 # private API endpoint
az aks command invoke -g rg-aks -n aks-prod-eus -c "kubectl get nodes"  # reach private API
az aks create ... --outbound-type userDefinedRouting       # egress via firewall/NVA

# --- Always: validate before shipping ---
kubectl apply --dry-run=server -f manifest.yaml

Interview Q&A

Q: Why is Azure CNI Overlay the default networking recommendation over classic Azure CNI and kubenet? A: Overlay draws pod IPs from an off-VNet overlay CIDR, so only nodes consume VNet addresses — the same subnet that holds ~2 nodes under classic CNI (with maxPods=110) holds ~250 under Overlay. kubenet is on a deprecation path and has limited policy support. The only thing Overlay gives up is direct pod-IP routability from outside the cluster, which is rarely needed.

Q: How does Workload Identity remove secrets from the cluster? A: AKS exposes an OIDC issuer; a Kubernetes service account’s projected token proves its identity, and a federated credential on a managed identity trusts that token’s subject. Entra ID exchanges the SA token for an Entra access token — so the pod gets Azure RBAC with no stored secret to leak or rotate. The azure.workload.identity/use: "true" pod label triggers the injection.

Q: What’s the difference between managed NGINX ingress and Application Gateway for Containers? A: Managed NGINX (app-routing add-on) terminates TLS and routes inside the cluster — simple and portable. AGC is a managed Azure L7 with a WAF that sits in front of the cluster, driven by the Gateway API, offloading TLS and enabling weighted traffic splits. Choose AGC when you need an edge WAF and Gateway-API traffic management; NGINX for a straightforward low-ops ingress.

Q: You enabled a default-deny egress policy and the app broke. What happened? A: Egress deny also blocks DNS, which is an egress call to CoreDNS on port 53. Every name lookup fails, and the app surfaces confusing connection timeouts. Fix: allow egress to kube-system kube-dns on UDP and TCP 53 in the same change as the default-deny.

Q: Metrics, logs, or traces — which do you alert on, and why? A: Alert on metrics (cheap, real-time) — specifically user-facing SLO burn rate on the golden signals. Use logs for forensic drill-down once an alert fires (expensive per GB, so keep them lean), and traces to localize which service in a call chain is slow. Alerting on logs or raw CPU is noisy and costly.

Q: How do you keep an AKS observability bill under control? A: Two levers. Logs: Container Insights cost presets, exclude noisy namespaces, Basic Logs tier for high-volume tables, deliberate retention. Metrics: keep label cardinality bounded (no raw IDs/paths as labels) since every label combo is a separate time series. And separate the stores — Azure Monitor workspace (metrics), Log Analytics (logs), App Insights (traces) — so each is tuned independently.

Q: How do you harden the control plane and egress on AKS? A: Make the API server private (--enable-private-cluster) or restrict it with authorized IP ranges, and route egress with --outbound-type userDefinedRouting through an Azure Firewall that allow-lists required FQDNs — plus a NAT Gateway if you hit SNAT port exhaustion. That closes the inbound control-plane surface and turns egress from open into allow-listed.

Glossary

• AKS (Azure Kubernetes Service) — Azure’s managed Kubernetes; you own the nodes and workloads, Azure runs the control plane.
• CNI (Container Network Interface) — the plugin that gives pods their IPs and wires up (and optionally enforces policy on) pod networking.
• kubenet — the legacy AKS network model; pods use a fake, node-routed overlay. On a deprecation path — avoid for new clusters.
• Azure CNI (classic / node-subnet) — pods get a real VNet IP from the node subnet; directly routable but consumes a VNet IP per pod.
• Azure CNI Overlay — pods draw from an off-VNet pod_cidr; only nodes consume VNet IPs. The recommended default; pods are not routable outside the cluster.
• Cilium — an eBPF-based data plane/policy engine; on AKS, “Azure CNI powered by Cilium” adds identity-based, L7-aware NetworkPolicy.
• maxPods — the pods a node may run; with classic CNI it pre-reserves that many VNet IPs per node, which drives subnet sizing.
• Service CIDR — a virtual range for ClusterIP Service VIPs; never appears on the wire. dns_service_ip (CoreDNS) must sit inside it.
• Node pool (system vs user) — groups of nodes; a tainted system pool runs critical add-ons, a user pool runs your apps so they never contend.
• Workload Identity — federates a Kubernetes service account to an Entra managed identity via the OIDC issuer, so pods get Azure tokens with no stored secret.
• OIDC issuer — the AKS endpoint that signs service-account tokens Entra can trust for federation.
• Ingress — the Kubernetes object (and the controller behind it) that routes external HTTP(S) traffic to Services; the cluster’s front door.
• App Routing add-on — AKS-managed ingress-nginx (ingressClassName: webapprouting.kubernetes.azure.com); low-ops managed NGINX.
• AGIC / AGC — App Gateway Ingress Controller (classic Ingress API) and Application Gateway for Containers (Gateway API); managed Azure L7 with WAF in front of the cluster.
• cert-manager — a controller that obtains and renews TLS certificates (e.g., Let’s Encrypt) for Ingress via a ClusterIssuer.
• NetworkPolicy — the Kubernetes API for allowing pod traffic; additive-allow, enforced only if the CNI/engine supports it.
• Default-deny — the posture where a namespace refuses all pod traffic in a direction except explicitly allowed flows.
• Azure NPM / Calico / Cilium — the three NetworkPolicy enforcement engines selectable on AKS; only Cilium adds L7/FQDN rules.
• Managed Prometheus — Azure Monitor’s hosted Prometheus; ama-metrics agents scrape and remote-write into an Azure Monitor workspace.
• Managed Grafana — Azure’s hosted Grafana, pre-wired to the Azure Monitor workspace for dashboards and PromQL.
• Container Insights — the AKS monitoring add-on; ama-logs tails container stdout/stderr into Log Analytics (billed per GB).
• Control-plane / diagnostic logs — API server, scheduler, controller-manager, and audit logs, streamed via a diagnostic setting (kube-audit-admin is the lean audit category).
• OpenTelemetry (OTel) — the vendor-neutral standard for emitting metrics, logs, and traces; on Azure, traces land in Application Insights.
• Golden signals — latency, traffic, errors, saturation; the first metrics to alert on.
• SLO / error budget / burn rate — an objective (e.g., 99.9% success), the allowed failure it implies, and how fast you are spending it; alert on burn rate, not raw resources.
• Private cluster — an AKS cluster whose API server has a private endpoint (no public FQDN), reachable only from inside the network or via az aks command invoke.
• Outbound type / UDR / NAT Gateway — how the cluster egresses: default load balancer, a NAT Gateway (scalable SNAT), or userDefinedRouting sending egress through a firewall for allow-listing.
• SNAT port exhaustion — running out of source ports for outbound connections behind the default LB; shows up as intermittent egress failures, fixed by a NAT Gateway.