Fixing AKS Ingress: 502/504 Backends, Application Routing Add-on Quirks, and Key Vault TLS Failures

The deploy went green, the pods are Running, and yet curl https://shop.contoso.com/ answers 502 Bad Gateway. Five minutes later the same URL returns 504 Gateway Time-out, and the TLS padlock that worked yesterday now throws a certificate error. Welcome to AKS ingress — the layer that takes an HTTP request from the public internet, terminates TLS, and routes it by hostname and path to the right Service, which in turn load-balances to your Pods. On Azure Kubernetes Service this layer is usually an NGINX ingress controller, and it is the single most common place a healthy-looking cluster breaks in production, because the error you see (502, 504) is emitted by the controller, not by your app. The controller is telling you “I could not get a good answer from the backend you pointed me at.” Why it could not is the entire investigation, and at least a dozen distinct root causes hide behind those three digits.

This article is the diagnostic playbook for that layer, scoped to the three things that actually page you on Azure: 502/504 from the backend (the Service has no ready endpoints, the probe is wrong, the port is mismatched, or the upstream is just slow), the Application Routing add-on (Azure’s managed NGINX — convenient, but it owns the controller config and surprises you in ways a self-installed controller does not), and Key Vault TLS (serving certificates from Azure Key Vault through the Secret Store CSI driver, which fails silently when identity, sync, or the Ingress annotation is off by a hair). For each failure you get the exact kubectl or az command to confirm it and the precise fix, with Bicep where the resource is Azure-side. Because this is a reference you reach for mid-incident, the playbook itself, the error strings, the annotations and the limits are laid out as scannable tables — read the prose once, then keep the tables open.

By the end you will stop guessing. When ingress 502s you will know within ninety seconds whether the Service has zero Endpoints, the readiness probe is failing, the servicePort points at a port the Pod never opened, the controller and your Ingress disagree on ingressClassName, the Application Routing add-on overwrote your tweak, or the Key Vault certificate never synced into a Kubernetes Secret. Knowing which is what separates a five-minute fix from a two-hour bridge call.

What problem this solves

Kubernetes hides a lot of machinery so you can kubectl apply a Deployment, a Service and an Ingress and have a public HTTPS endpoint. That abstraction is a gift until it breaks, and then it becomes an opaque wall. The bare 502 Bad Gateway page from NGINX deliberately tells the anonymous caller nothing — leaking backend detail would be a security hole. So the information you need is real and captured, but it lives across four or five surfaces: the controller’s own logs, the Ingress object’s status and events, the Service/Endpoints pair, the Pod’s readiness state, and — for TLS — a SecretProviderClass, a CSI driver Pod, and a synced Kubernetes Secret. If you do not know which surface maps to which failure, you burn an hour kubectl describe-ing the wrong object.

What breaks without this knowledge: an engineer restarts the ingress controller (which sometimes “fixes” it by accident and teaches the wrong lesson), recreates the Ingress (which re-triggers the same broken cert sync), or scales the app (which does nothing for a port mismatch). Meanwhile the actual cause — a Service whose selector matches no Pods so Endpoints is empty, a readiness probe hitting / on a Pod that only serves /healthz, a Secret that never materialised because the cluster’s managed identity lacks Key Vault Secrets User — sits there, perfectly diagnosable, ignored.

Who hits this: essentially every team running web apps or APIs on AKS behind ingress. It bites hardest on teams adopting the Application Routing add-on expecting it to behave exactly like the community ingress-nginx chart (it does not — it is reconciled and partly read-only), on anyone wiring TLS from Key Vault for the first time (the identity and sync chain has five links and any one breaks it), and on anyone who assumes a Running Pod means a ready backend (readiness, not liveness, is what ingress routes to). The fix is almost never “restart the controller” — it is “find the hop that is lying and make it tell the truth.”

To frame the field before the deep dive, here is every symptom class this article covers, the question it forces, and the first place to look.

Learning objectives

By the end of this article you can:

• Map any AKS ingress 502/503/504 to a specific hop — controller, Ingress rule, Service, Endpoints, readiness probe, or Pod — and name the most likely root cause for each.
• Diagnose a 502 Bad Gateway as empty Endpoints, a port mismatch between the Ingress servicePort and the container port, a failing readiness probe, or an upstream that resets the connection — and confirm which with exact kubectl commands.
• Diagnose a 504 Gateway Time-out as an upstream slower than the NGINX proxy-read-timeout, and decide between speeding up the app and tuning the timeout annotation.
• Work with the Application Routing add-on correctly: know which objects it owns and reconciles, how to customise the managed controller through the supported surface (the NginxIngressController CRD), and how to attach a DNS zone and Key Vault — without fighting the reconciler.
• Wire TLS from Azure Key Vault through the Secret Store CSI driver: create the SecretProviderClass, grant the cluster identity Key Vault Secrets User, mount it so the certificate syncs to a Kubernetes Secret, and reference that Secret from the Ingress tls block — and debug each of the five links when the chain breaks.
• Read the canonical annotation, error-string and timeout reference tables and pick the right setting for each failure class.
• Drive the core diagnostic tools fluently: kubectl describe ingress, controller logs, kubectl get endpoints, kubectl get events, and the CSI driver / SecretProviderClassPodStatus surface.

Prerequisites & where this fits

You should already understand AKS basics: a cluster has a managed control plane and one or more node pools; you deploy Pods (grouped by a Deployment), expose them with a Service (a stable virtual IP and DNS name that load-balances across Pods matching its label selector), and you reach kubectl against the cluster via az aks get-credentials. You should know that a Service of type ClusterIP is internal and that an Ingress object plus an ingress controller is what publishes HTTP(S) to the outside. Familiarity with HTTP status codes, TLS termination, and basic Kubernetes objects (Deployment, Service, Ingress, Secret) is assumed.

This sits in the Observability & Troubleshooting track and assumes the cluster fundamentals from AKS Cluster Architecture: Control Plane vs Data Plane Explained and the networking model choices in AKS Networking: Kubenet vs Azure CNI Models Explained, because how Pod IPs are assigned changes how the ingress controller reaches them. Identity is the other dependency: serving certificates from Key Vault relies on the cluster’s identity having access, which builds on AKS Cluster Authentication: Managed Identity vs Service Principal and the Key Vault permission model in Key Vault RBAC vs Access Policies: Which Permission Model. When the failure is image-side rather than ingress-side, AKS Pod Cannot Pull Image: ACR Authorization Failures is the companion playbook.

A quick map of who owns what during an incident, so you call the right person fast.

Core concepts

Five mental models make every later diagnosis obvious.

The status code names the controller’s complaint, not your bug. Every request flows through an ingress controller — on AKS, an NGINX reverse proxy running as Pods, fronted by an Azure Load Balancer public IP. NGINX terminates TLS, matches the request’s host and path against your Ingress rules, and proxies to the matching Service’s endpoints. A 502 Bad Gateway means NGINX selected an upstream but got a broken or no response (connection refused, reset, or no endpoints to try). A 504 Gateway Time-out means it reached an upstream but the upstream did not answer within the proxy read timeout. A 503 Service Temporarily Unavailable means no upstream was available at all. “Bad answer” (502) vs “too slow” (504) vs “nobody home” (503) is the first fork in every decision tree.

Ingress routes to ready Endpoints, not to Pods. A Service does not point at Pods directly; it points at the set of Pod IPs whose labels match its selector and that are passing their readiness probe. That set is the Endpoints object (or EndpointSlice). If the selector matches nothing, or every matching Pod is failing readiness, Endpoints is empty — and NGINX, having no upstream to proxy to, returns 502. A Pod can be Running (liveness passing) and still be excluded from Endpoints because readiness fails. This single distinction — Running is not Ready, and only Ready joins Endpoints — explains the majority of “but my pods are up!” 502s.

The port contract is explicit at two layers. Your Ingress rule names a Service and a port (service.port.number or name). The Service maps that port to a targetPort on the Pod, which must equal the container port your app actually listens on. Two mismatches bite: the Ingress servicePort not matching any port the Service exposes (NGINX cannot build an upstream), and the Service targetPort not matching the real container port (the connection is refused at the Pod). Either yields 502, and both look like the app is broken when it is the wiring.

The Application Routing add-on owns its controller. When you enable the Application Routing add-on, Azure deploys and manages an NGINX controller for you and exposes a public (or internal) ingress with a default ingressClassName of webapprouting.kubernetes.azure.com. The add-on reconciles its objects: if you edit the managed Deployment, ConfigMap or IngressClass directly, the controller’s reconciler reverts your change on its next loop. You customise it only through the supported surface — the NginxIngressController custom resource and the add-on’s DNS-zone / Key Vault attachments via az aks approuting. Treating the managed controller like a self-installed Helm release is the number-one add-on mistake.

Key Vault TLS is a five-link sync chain. Serving a certificate from Azure Key Vault does not happen by magic. The Secret Store CSI driver (plus the Azure Key Vault provider) reads a SecretProviderClass that names the vault and certificate, authenticates with the cluster’s identity (the add-on’s user-assigned identity or workload identity), and — when a Pod mounts that CSI volume — writes the certificate into a Kubernetes Secret of type kubernetes.io/tls. Your Ingress tls block then references that Secret. Break any link — wrong vault/cert name, identity without Key Vault Secrets User, the secretObjects sync block missing so no Secret is created, no Pod mounting the volume, or the Ingress pointing at the wrong secretName — and NGINX serves its default self-signed “Kubernetes Ingress Controller Fake Certificate” instead, which clients reject. The fake-certificate string is your single best tell that the chain is broken.

The vocabulary in one table

Pin down every moving part before the deep sections. The glossary repeats these for lookup; this table is the mental model side by side.

The HTTP status-code and error-string reference

Before the per-symptom anatomy, here is the lookup table you scan first: every status and error string you realistically see from an AKS NGINX ingress, what it means on this platform, the likely cause, how to confirm it, and the first fix. The non-obvious ones are the NGINX upstream errors in the controller log (connect() failed, upstream prematurely closed, no live upstreams) and the difference between NGINX’s 502 and a 502 your app itself returned.

Two reading notes that save the most time.

Anatomy of a 502 Bad Gateway

A 502 means NGINX selected an upstream and got a bad or no answer. Four distinct causes on AKS. Scan the matrix, then read the detail for whichever row matches.

Cause 1 — The Service has no ready endpoints

This is the overwhelming favourite. Your Service selector does not match the Pod labels, or every matching Pod is failing its readiness probe, so the Endpoints object is empty. NGINX has no upstream, logs no live upstreams, and returns 502 — even though kubectl get pods shows Running.

Confirm. The one command that decides it:

# Empty output (no addresses) = the cause. Compare selector to pod labels.
kubectl get endpoints shop-svc -n shop
kubectl get pods -n shop --show-labels
kubectl describe svc shop-svc -n shop | grep -i selector

If Endpoints is empty but Pods are Running, check readiness explicitly — a Pod that is Running but 0/1 READY is excluded:

# READY column 0/1 with Running status = readiness failing
kubectl get pods -n shop -o wide
kubectl describe pod <pod> -n shop | sed -n '/Conditions/,/Events/p'

Fix. If the selector is wrong, align the Service selector to the Pod labels (or vice versa). If readiness is failing, fix the probe — usually it points at a path or port the app does not serve. The two halves:

# Selector mismatch: see what the Service expects vs what Pods carry, then fix one side.
kubectl get svc shop-svc -n shop -o jsonpath='{.spec.selector}'; echo
kubectl get deploy shop -n shop -o jsonpath='{.spec.template.metadata.labels}'; echo

# Readiness probe must hit a path/port the app actually serves.
readinessProbe:
  httpGet:
    path: /healthz      # NOT "/" if the app only serves /healthz
    port: 8080          # the real container port
  initialDelaySeconds: 5
  periodSeconds: 10

The Endpoints-empty causes and the one check that proves each.

Cause 2 — The Ingress points at a Service port that does not exist

Your Ingress rule names port.number: 80 but the Service only exposes 8080, or names a port.name the Service never defined. NGINX cannot build an upstream for a non-existent Service port and 502s on that host/path while other rules work.

Confirm. Diff the Ingress rule against the Service ports:

kubectl describe ingress shop-ing -n shop | sed -n '/Rules/,/Annotations/p'
kubectl get svc shop-svc -n shop -o jsonpath='{.spec.ports}'; echo

Fix. Make the Ingress service.port match a port the Service actually publishes:

spec:
  rules:
    - host: shop.contoso.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: shop-svc
                port:
                  number: 80     # must be a port the Service exposes

Cause 3 — The Service targetPort does not match the container port

The Service forwards to targetPort: 8080 but the container listens on 3000 (or binds 127.0.0.1 instead of 0.0.0.0). The connection to the Pod is refused; the controller log shows connect() failed (111: Connection refused).

Confirm. Check what the Service targets versus what the Pod actually listens on:

kubectl get svc shop-svc -n shop -o jsonpath='{.spec.ports[*].targetPort}'; echo
# Exec in and confirm the listening port/interface
kubectl exec -it deploy/shop -n shop -- sh -c "ss -ltnp 2>/dev/null || netstat -ltn"

A container that binds 127.0.0.1:8080 rejects connections from the Pod network even when the port number is right — bind 0.0.0.0.

Fix. Align targetPort to the real container port (and the app to bind all interfaces):

apiVersion: v1
kind: Service
metadata: { name: shop-svc, namespace: shop }
spec:
  selector: { app: shop }
  ports:
    - name: http
      port: 80           # the Service port the Ingress references
      targetPort: 8080   # the REAL container port the app listens on

Cause 4 — The upstream resets or crashes mid-request

The Pod accepts the connection but resets it or dies before answering — an app crash on a specific request, an upstream timeout shorter than NGINX expects, or an HTTP keepalive mismatch. The log string is upstream prematurely closed connection while reading response header from upstream.

Confirm. Correlate the controller error with the Pod’s own logs at that moment:

kubectl logs -n app-routing-system deploy/nginx -c nginx --since=10m | grep -i "prematurely closed"
kubectl logs deploy/shop -n shop --since=10m | tail -50

Fix. Fix the crashing code path; do not paper over it with retries. If the app legitimately has a long-lived response, raise the proxy read timeout (next section), but a reset is a crash, not a timeout — different fix.

Anatomy of a 504 Gateway Time-out

A 504 means NGINX reached the upstream but it did not answer within the proxy read timeout. On ingress-nginx the default proxy-read-timeout and proxy-send-timeout are 60 seconds. A backend slower than that — a heavy report, a stuck downstream call, a thread-pool-starved app — gets cut and the client sees 504, while the Pod’s own logs may show the request completing a few seconds later.

Confirm. The controller log shows the timeout and which upstream; kubectl top pod shows whether the app is resource-starved (CPU pinned → slow → 504):

kubectl logs -n app-routing-system deploy/nginx -c nginx --since=10m | grep -i "timed out"
kubectl top pod -n shop          # CPU/memory pressure that slows responses

In the Pod, a request log that takes 70–120 s for an endpoint NGINX cut at 60 s confirms “slow backend, not a network drop.”

Fix. The right fix is to speed up the backend (add CPU/replicas, fix the slow query, make the work async). When the operation is legitimately long (a large upload, a long-poll), raise the per-Ingress timeout via annotation — do not globally raise it and hide every slow endpoint:

metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-read-timeout: "180"   # seconds, default 60
    nginx.ingress.kubernetes.io/proxy-send-timeout: "180"
    nginx.ingress.kubernetes.io/proxy-body-size: "50m"       # if 413 on large uploads

The NGINX timeouts and limits that matter, their defaults, and what hitting each looks like.

A reading note: a 504 with the Pod log showing the request completed (slowly) is a timeout-tuning question; a 502 with connect() failed is a wiring question. Do not raise timeouts to fix a 502 — they are unrelated.

The Application Routing add-on: how the managed controller differs

Azure’s Application Routing add-on gives you a managed NGINX ingress controller without installing or upgrading Helm charts yourself. It is the recommended path on AKS, but it behaves differently from a self-installed ingress-nginx, and those differences cause their own incidents. The core rule: the add-on owns its objects and reconciles them. Edit a managed object directly and the reconciler reverts you.

Enable or inspect it with az aks:

# Enable on an existing cluster
az aks approuting enable -g rg-shop -n aks-shop

# What it deployed (controller Pods live in the app-routing-system namespace)
kubectl get pods -n app-routing-system
kubectl get ingressclass        # default class: webapprouting.kubernetes.azure.com

The default ingressClassName it creates is webapprouting.kubernetes.azure.com. Your Ingress must reference that class (or omit ingressClassName only if it is the default class) or the managed controller ignores it and you get a 404 / default backend.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata: { name: shop-ing, namespace: shop }
spec:
  ingressClassName: webapprouting.kubernetes.azure.com   # the add-on's class
  rules:
    - host: shop.contoso.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend: { service: { name: shop-svc, port: { number: 80 } } }

What the add-on owns versus what you own

Knowing the ownership line is the whole game with this add-on. You own your application objects; the add-on owns the controller machinery.

Customising the managed controller the supported way

To change controller-level behaviour (internal vs public load balancer, a custom default IngressClass name, controller annotations), use the NginxIngressController custom resource, which the add-on watches. This is the supported tuning surface — changes here survive reconciliation because the reconciler reads this CRD.

# Supported customisation surface for the managed controller
apiVersion: approuting.kubernetes.azure.com/v1alpha1
kind: NginxIngressController
metadata: { name: nginx-internal }
spec:
  ingressClassName: nginx-internal
  controllerNamePrefix: nginx-internal
  loadBalancerAnnotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"   # internal LB

Attaching a DNS zone (so the add-on can manage records) and a Key Vault (for certs) is done through the add-on, not by hand-editing controller config:

# Attach a public DNS zone the add-on will manage records in
ZONEID=$(az network dns zone show -g rg-dns -n contoso.com --query id -o tsv)
az aks approuting zone add -g rg-shop -n aks-shop --ids $ZONEID --attach-zones

# Attach a Key Vault for certificate integration
KVID=$(az keyvault show -g rg-shop -n kv-shop --query id -o tsv)
az aks approuting update -g rg-shop -n aks-shop --enable-kv --attach-kv $KVID

The add-on quirks that page people, and how to handle each.

TLS from Key Vault: the Secret Store CSI sync chain

Serving a real certificate from Azure Key Vault on AKS ingress runs through the Secret Store CSI driver and its Azure Key Vault provider. The chain has five links, and any broken link makes NGINX fall back to its default “Kubernetes Ingress Controller Fake Certificate,” which every browser rejects. Walk the chain in order; each link has one confirming check.

The five links, what each does, and the check that proves it.

Link 1–2 — the cluster identity and its Key Vault access

The Secret Store CSI provider authenticates to Key Vault using an identity — with the Application Routing add-on this is the add-on’s user-assigned managed identity; in a DIY setup it is workload identity federated to a service account. Either way, that identity needs read access to the vault’s certificates/secrets, which under the RBAC model is the Key Vault Secrets User role (a certificate’s private key is fetched as a secret). Under the older access-policy model it needs get on secrets and certificates.

# Find the add-on identity (objectId) and grant it read on the vault
OBJ=$(az aks show -g rg-shop -n aks-shop \
  --query "ingressProfile.webAppRouting.identity.objectId" -o tsv)
KVID=$(az keyvault show -g rg-shop -n kv-shop --query id -o tsv)
az role assignment create --assignee-object-id "$OBJ" --assignee-principal-type ServicePrincipal \
  --role "Key Vault Secrets User" --scope "$KVID"

// Grant the cluster/add-on identity read on the vault (RBAC model)
resource kvSecretsUser 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
  name: guid(kv.id, addonIdentityObjectId, 'kv-secrets-user')
  scope: kv
  properties: {
    roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions',
      '4633458b-17de-408a-b874-0445c86b69e6') // Key Vault Secrets User
    principalId: addonIdentityObjectId
    principalType: 'ServicePrincipal'
  }
}

If this link is broken the CSI driver Pod logs a 403 Forbidden / does not have secrets get permission when it tries to fetch the cert. That is your confirmation:

kubectl logs -n kube-system -l app=secrets-store-csi-driver -c secrets-store --since=10m | grep -i forbidden

For the deeper Key Vault permission and firewall failures behind that 403, the dedicated playbook is Key Vault 403 Forbidden: Firewall, RBAC, Soft-Delete Recovery.

Link 3 — the SecretProviderClass

The SecretProviderClass is the declaration of which vault and certificate to fetch, and crucially the secretObjects block that tells the driver to also write the fetched cert into a Kubernetes Secret of type kubernetes.io/tls. Without secretObjects, the cert is mounted as a file but no TLS Secret is ever created, and your Ingress has nothing to reference. This omission is the single most common Key Vault-TLS mistake.

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata: { name: shop-tls, namespace: shop }
spec:
  provider: azure
  secretObjects:                       # <-- THIS block is what creates the K8s Secret
    - secretName: shop-tls-secret       # the Secret your Ingress will reference
      type: kubernetes.io/tls
      data:
        - objectName: shop-cert         # must match objectName below
          key: tls.key
        - objectName: shop-cert
          key: tls.crt
  parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "true"
    userAssignedIdentityID: "<add-on identity clientId>"
    keyvaultName: "kv-shop"
    tenantId: "<tenant-guid>"
    objects: |
      array:
        - |
          objectName: shop-cert
          objectType: secret           # a cert's key material is fetched as a secret

Link 4 — a Pod must mount the volume to trigger the sync

The CSI driver only writes the Secret when a Pod mounts the CSI volume backed by this SecretProviderClass. With the Application Routing add-on this mounting is wired for you when you reference the cert via the Ingress annotation; in a DIY setup you must run a Pod (often the controller, or a small “syncer” Deployment) that mounts the volume so the Secret materialises and stays refreshed. If nothing mounts it, kubectl get secret shop-tls-secret returns NotFound forever.

# The proof: does the TLS Secret exist yet?
kubectl get secret shop-tls-secret -n shop
# Per-pod sync status from the driver
kubectl get secretproviderclasspodstatus -n shop
kubectl describe secretproviderclasspodstatus -n shop | sed -n '/Status/,/Events/p'

Link 5 — the Ingress TLS reference (and the add-on shortcut)

Finally the Ingress tls block names the synced Secret. With the Application Routing add-on you can skip the hand-rolled SecretProviderClass and instead annotate the Ingress to pull the cert straight from the attached Key Vault — the add-on creates the SecretProviderClass and sync for you.

# Add-on shortcut: point the Ingress at a Key Vault cert URI; the add-on wires the sync
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: shop-ing
  namespace: shop
  annotations:
    kubernetes.azure.com/tls-cert-keyvault-uri: https://kv-shop.vault.azure.net/certificates/shop-cert
spec:
  ingressClassName: webapprouting.kubernetes.azure.com
  tls:
    - hosts: [ shop.contoso.com ]
      secretName: shop-tls-secret      # the add-on syncs the KV cert into this Secret
  rules:
    - host: shop.contoso.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend: { service: { name: shop-svc, port: { number: 80 } } }

The fake-certificate confirmation. Whenever TLS is wrong, the fastest single check is what certificate NGINX actually presents. If the CN is the fake cert, a link in the chain is broken:

echo | openssl s_client -connect shop.contoso.com:443 -servername shop.contoso.com 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates
# CN "Kubernetes Ingress Controller Fake Certificate" => sync chain broken (links 1-4)
# A real CN with a past notAfter => genuine certificate expiry => rotate in Key Vault

The Key Vault-TLS failure modes mapped to the link that broke.

Architecture at a glance

The diagram traces an HTTPS request exactly as it flows on AKS, and marks the hop where each failure class bites. Read it left to right. A client sends https://shop.contoso.com/ to the Azure Load Balancer public IP, which forwards to the NGINX ingress controller Pods. NGINX terminates TLS using a certificate that originated in Azure Key Vault and was synced — by the Secret Store CSI driver — into a Kubernetes TLS Secret. NGINX then matches the request’s host and path against your Ingress rules and proxies to the named Service, which resolves to the set of ready Endpoints (Pod IPs passing readiness) and load-balances across your Pods.

The numbered badges sit on the exact hops that fail. Badge 1 is TLS at the controller: if the Key Vault cert never synced, NGINX serves the fake certificate and the handshake is rejected. Badge 2 is the Ingress/IngressClass match: a wrong ingressClassName (especially with the Application Routing add-on’s webapprouting… class) means no rule matches and you get a 404/default backend. Badge 3 is the Service → Endpoints hop: an empty Endpoints set (selector or readiness) yields 502 no live upstreams. Badge 4 is the Pod port/readiness: a targetPort that does not match the container port refuses the connection (connect() failed), and a slow Pod trips the 60 s proxy timeout into a 504. The legend narrates each as symptom, the confirming command, and the fix — that is the whole method: localise the symptom to a hop, run the named check, apply the fix.

Real-world scenario

Beacon Tickets runs a ticketing API on AKS: a Deployment of six replicas behind a ClusterIP Service, published via the Application Routing add-on (managed NGINX) with TLS certificates from Key Vault, in Central India. Traffic averages 300 requests/second with a sharp spike when a popular event opens sales. The platform team is three engineers; the cluster runs a 3-node Standard_D4s_v5 user pool.

The incident began the morning a new event went on sale. At 10:00 the API started returning 502 Bad Gateway to roughly 20% of callers, climbing as load rose. The on-call engineer’s reflex was to restart the NGINX controller Pods. That cleared it for about two minutes, then the 502s returned — a classic sign the controller was fine and the upstream was the problem. Second reflex: scale the API Deployment from 6 to 12 replicas. The error rate barely moved, and now the cluster was burning more nodes.

The breakthrough came from asking the right first question: does the Service have ready endpoints? kubectl get endpoints ticket-svc -n tickets showed only two addresses, not twelve — ten Pods were Running but 0/1 READY. kubectl describe pod on a not-ready Pod showed the readiness probe failing: it hit / with a 200-expectation, but a recent deploy had moved the health endpoint to /healthz and / now returned a 302 redirect to the login page. NGINX, with only two ready upstreams under a 300 rps spike, was overwhelming them and 502-ing the overflow. Scaling to twelve had added unready Pods, not capacity — which is why it did nothing.

While confirming, a second problem surfaced. The browser padlock was now throwing ERR_CERT_AUTHORITY_INVALID. openssl s_client -connect tickets.beacon.in:443 returned CN “Kubernetes Ingress Controller Fake Certificate.” The Key Vault certificate had auto-rotated overnight (a 90-day policy), and the synced TLS Secret still held the old cert — except the controller had restarted (from reflex one) and re-read state, and the CSI sync had not refreshed because the add-on identity’s role assignment had been scoped to the wrong vault after a recent resource-group reorg. The CSI driver logs showed 403 Forbidden. So there were two independent failures coinciding: a readiness-probe regression causing 502s, and a Key Vault RBAC scope error causing the fake cert.

The fix landed in two parts. Immediately: correct the readiness probe to httpGet /healthz: 8080, which moved all twelve Pods to READY within thirty seconds and cleared the 502s (they then scaled back to six). Then: re-create the Key Vault Secrets User assignment for the add-on identity scoped to the correct vault, after which the CSI driver re-synced the rotated certificate into the TLS Secret and NGINX served the real cert. The post-incident rule on the wall: “A 502 behind ingress is a question about Endpoints, not the controller — and a fake cert means the sync chain broke, never the cert itself.” They added two alerts: one on Endpoints count dropping below replica count, and one on the served-certificate CN matching the fake-cert string.

The incident as a timeline, because the order of moves is the lesson.

Advantages and disadvantages

The “managed ingress controller fronting label-selected Pods” model both causes this class of problem and makes it diagnosable. Weigh it honestly.

The model is right for standard web apps and APIs where you want declarative routing and managed certificates rather than operating a reverse proxy by hand. It bites hardest on teams that treat the Application Routing add-on like a self-installed chart, that conflate Running with Ready, and that wire Key Vault TLS for the first time without walking the sync chain. Every disadvantage is manageable — but only if you know it exists, which is the point of this article.

Hands-on lab

Reproduce a 502 from an empty-Endpoints readiness bug, watch it, and fix it — all on a small AKS cluster you tear down at the end. Run in Cloud Shell (Bash) with kubectl configured.

Step 1 — Variables and a tiny cluster (or reuse one).

RG=rg-aks-ingress-lab
LOC=centralindia
AKS=aks-ing-lab
az group create -n $RG -l $LOC -o table
az aks create -g $RG -n $AKS --node-count 2 --node-vm-size Standard_B2s \
  --network-plugin azure --generate-ssh-keys -o table
az aks get-credentials -g $RG -n $AKS --overwrite-existing

Step 2 — Enable the Application Routing add-on (managed NGINX).

az aks approuting enable -g $RG -n $AKS
kubectl get ingressclass     # expect: webapprouting.kubernetes.azure.com
kubectl get pods -n app-routing-system

Step 3 — Deploy an app, a Service, and an Ingress — but with a BROKEN readiness probe (reproduce the bug).

kubectl create namespace shop
cat <<'EOF' | kubectl apply -n shop -f -
apiVersion: apps/v1
kind: Deployment
metadata: { name: shop }
spec:
  replicas: 2
  selector: { matchLabels: { app: shop } }
  template:
    metadata: { labels: { app: shop } }
    spec:
      containers:
        - name: web
          image: mcr.microsoft.com/azuredocs/aks-helloworld:v1
          ports: [ { containerPort: 80 } ]
          readinessProbe:
            httpGet: { path: /does-not-exist, port: 80 }   # WRONG path → never Ready
            initialDelaySeconds: 3
            periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata: { name: shop-svc }
spec:
  selector: { app: shop }
  ports: [ { port: 80, targetPort: 80 } ]
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata: { name: shop-ing }
spec:
  ingressClassName: webapprouting.kubernetes.azure.com
  rules:
    - http:
        paths:
          - path: /
            pathType: Prefix
            backend: { service: { name: shop-svc, port: { number: 80 } } }
EOF

Step 4 — Watch it 502. Get the ingress public IP and curl it:

IP=$(kubectl get svc -n app-routing-system -o jsonpath='{.items[?(@.spec.type=="LoadBalancer")].status.loadBalancer.ingress[0].ip}')
echo "Ingress IP: $IP"
curl -s -o /dev/null -w "%{http_code}\n" http://$IP/        # expect 502
kubectl get endpoints shop-svc -n shop                      # expect: <none> (empty)
kubectl get pods -n shop                                    # Running but 0/1 READY

The empty Endpoints with 0/1 READY Pods is the dead giveaway: readiness is failing, so no Pod joins Endpoints, so NGINX has no upstream → 502.

Step 5 — Fix the readiness probe. Point it at a path the app actually serves:

kubectl patch deploy shop -n shop --type=json -p='[
  {"op":"replace","path":"/spec/template/spec/containers/0/readinessProbe/httpGet/path","value":"/"}
]'
kubectl rollout status deploy/shop -n shop
kubectl get endpoints shop-svc -n shop      # now shows Pod IPs
curl -s -o /dev/null -w "%{http_code}\n" http://$IP/   # expect 200

Step 6 — Tear down.

az group delete -n $RG --yes --no-wait

Expected end state: after the probe fix, Endpoints lists two addresses, the Pods read 1/1 READY, and the curl returns 200. You reproduced and fixed the most common AKS ingress 502 end to end.

Common mistakes & troubleshooting

This is the centerpiece. The playbook below spans the basic wiring failures and the advanced add-on/TLS failures. Find your symptom, run the exact confirm command, apply the fix.

Per-symptom detail on the ones that fool people

Empty Endpoints (rows 1–2). This is the 502 you will hit most. kubectl get endpoints is the single decisive check — blank means NGINX has no upstream. The fork is selector (Pods exist but labels do not match) versus readiness (Pods match but fail the probe). Never restart the controller for this; it changes nothing because the controller is healthy.

Port contract (rows 3–4). Two ports, two mismatches. connect() failed (111: Connection refused) in the controller log means the connection reached a Pod whose port was closed — targetPort is wrong or the app binds loopback. A 502 on one host with others working points at that Ingress’s servicePort not matching the Service.

The reconcile surprise (row 9). With the Application Routing add-on, any direct edit to the managed controller Deployment/ConfigMap/IngressClass is reverted on the next reconcile loop. The supported path is the NginxIngressController CRD. Teams migrating from self-installed ingress-nginx hit this within a day.

The fake-certificate tell (rows 10–14). Whenever TLS is wrong, openssl s_client is faster than reading five objects. CN Kubernetes Ingress Controller Fake Certificate proves the sync chain broke (links 1–4). A real CN with a past notAfter is genuine expiry — rotate in Key Vault. The most common break is secretObjects missing from the SecretProviderClass (no Secret is ever created) or the identity lacking Key Vault Secrets User.

Best practices

• Make readiness probes shallow and honest. Point them at a dedicated /healthz that returns 200 when this Pod can serve a request — not / (redirects) and not a path that hits a slow downstream. Readiness, not liveness, is what joins Endpoints.
• Always alert on Endpoints count vs replica count. A drop below the replica count is your earliest, cleanest 502 signal — earlier than the 502s themselves.
• Reference an explicit ingressClassName on every Ingress. Do not rely on default-class behaviour, especially when both a self-installed and a managed controller could exist.
• For the Application Routing add-on, never edit managed objects directly. Customise only through the NginxIngressController CRD and az aks approuting — anything else is reverted.
• Serve TLS from Key Vault, and alert on the served-cert CN. An alert that fires when the presented certificate CN equals the fake-cert string catches a broken sync chain before users do.
• Grant the cluster identity exactly Key Vault Secrets User scoped to the specific vault — no broader, and re-verify the scope after any resource-group reorg.
• Tune proxy-read-timeout per Ingress, not globally. Raise it only on the endpoints that legitimately run long; a global raise hides every slow endpoint.
• Keep the request path stateless. Avoid session affinity unless a legacy app demands it; affinity concentrates load and undermines even spreading across Pods.
• Set sane proxy-body-size for upload endpoints rather than discovering the 1m default via a 413 in production.
• Use a PodDisruptionBudget and conservative maxUnavailable so a rollout never drops every ready Pod and 503s the Service.
• Treat Ingress, Service and SecretProviderClass as reviewed code (Bicep/YAML in git), not hand-edits — most of these incidents are a one-line manifest error.

Security notes

Ingress is the front door, so its security posture is the cluster’s exposed surface.

• Terminate TLS at the controller with a real Key Vault certificate, never the fake self-signed cert, and prefer TLS 1.2+ via the controller’s ssl-protocols config. Centralising certs in Key Vault keeps private keys out of git and enables rotation.
• Use least-privilege identity for cert access. The add-on/workload identity needs only Key Vault Secrets User on the one vault. Avoid Key Vault Administrator or subscription-wide grants. This builds on Managed Identity: System vs User-Assigned Patterns.
• Scope the public surface. Use an internal load balancer (azure-load-balancer-internal: "true" via the NginxIngressController CRD) for private workloads, and front the public ingress with a WAF when you need OWASP protection — the WAF patterns are covered in Application Gateway with WAF and End-to-End TLS.
• Restrict who can edit Ingress and SecretProviderClass via Kubernetes RBAC; an attacker who can rewrite an Ingress host or tls.secretName can hijack routing or downgrade TLS.
• Do not expose Secrets broadly. The synced TLS Secret holds the private key — keep it in the app namespace, restrict get/list on Secrets, and never echo it in logs.
• Validate certificate rotation end to end. Auto-rotation in Key Vault is only safe if the CSI sync refreshes the Secret; test the rotation path so you do not serve an expired cert or fall back to the fake one.

Cost & sizing

AKS ingress cost is dominated by a few line items, and most “sizing” here is right-sizing the controller and the backing load balancer, not the certificates.

The right-sizing rules: one controller, one load balancer, many hosts — do not spin up an LB per app; route by hostname and path on a single ingress. Run the controller with 2–3 replicas for availability and an HPA if request volume is spiky, but its footprint is small relative to your app Pods. Key Vault and certificate costs are negligible; the meaningful spend is the Standard Load Balancer and its data processing, so keep east-west traffic in-cluster and only send genuinely external traffic through ingress. The Application Routing add-on adds no licence cost — you pay for the underlying LB, IP and compute it consumes, which you would pay for a self-installed controller anyway.

Interview & exam questions

1. Why does an AKS Ingress return 502 when all the Pods show Running? Because ingress routes to ready Endpoints, not to Pods. A Pod can be Running (liveness passing) yet excluded from Endpoints if its readiness probe fails. With empty Endpoints, NGINX has no upstream and returns 502. Check kubectl get endpoints <svc>.

2. What is the difference between a 502 and a 504 from the NGINX ingress controller? A 502 Bad Gateway means NGINX got a bad or no answer from the upstream (no endpoints, connection refused/reset). A 504 Gateway Time-out means the upstream was reached but did not respond within the proxy-read-timeout (default 60 s). 502 is a wiring problem; 504 is a slowness problem.

3. How does the Application Routing add-on differ from a self-installed ingress-nginx? It is a managed controller that Azure deploys and reconciles: direct edits to its Deployment/ConfigMap/IngressClass are reverted. You customise it through the NginxIngressController CRD and az aks approuting, and its default class is webapprouting.kubernetes.azure.com.

4. Walk the chain that serves a Key Vault certificate on AKS ingress. Cluster identity authenticates the Secret Store CSI driver to Key Vault; the identity has Key Vault Secrets User; a SecretProviderClass names the vault/cert and (via secretObjects) a target Secret; a Pod mounts the volume so the cert syncs into a kubernetes.io/tls Secret; the Ingress tls block references that Secret. Any broken link serves the fake cert.

5. A browser shows a certificate error on an AKS ingress. What is your first command? openssl s_client -connect host:443 -servername host then read the subject. CN “Kubernetes Ingress Controller Fake Certificate” means the Key Vault sync chain broke; a real CN with a past notAfter means genuine expiry — rotate in Key Vault.

6. Your Ingress returns default backend - 404. Why? No Ingress rule matched the request — usually a wrong ingressClassName (the controller ignores the Ingress), a host that does not match any rule, or a pathType/path that does not match. Confirm with kubectl describe ingress and kubectl get ingressclass.

7. The controller log shows connect() failed (111: Connection refused). What does it indicate? NGINX reached a Pod but its port was closed: the Service targetPort does not match the container’s real listening port, or the app binds 127.0.0.1 instead of 0.0.0.0. Fix the targetPort or the app’s bind address.

8. How do you correctly raise the request timeout for one slow endpoint without affecting others? Apply nginx.ingress.kubernetes.io/proxy-read-timeout (and proxy-send-timeout) as an annotation on that specific Ingress. A global controller change would hide every slow endpoint; a per-Ingress annotation scopes the change.

9. Why does scaling up Pods sometimes not fix a 502? If the 502 is from a readiness or port bug, the new Pods are also not-ready (or also refuse the connection), so you add zero ready Endpoints. Scaling adds capacity only when existing Pods are healthy and the bottleneck is load.

10. What identity does the Application Routing add-on use to read Key Vault, and what role does it need? A user-assigned managed identity created by the add-on; it needs Key Vault Secrets User (RBAC) on the vault — a certificate’s key material is fetched as a secret. Find its objectId via az aks show … ingressProfile.webAppRouting.identity.

11. How do you give a workload identity-based DIY setup access to Key Vault for TLS? Federate a workload identity to the Kubernetes service account used by the syncer/controller, grant that identity Key Vault Secrets User, and reference it in the SecretProviderClass (clientID). This is the non-add-on path.

12. Map to certs: these questions map to AZ-104 (manage AKS, Key Vault, networking), AZ-500 (secure ingress, identities, Key Vault access), and the CKA/CKAD ingress, Service/Endpoints, and probe objectives.

Quick check

1. Your Ingress 502s and kubectl get endpoints shop-svc shows no addresses, yet Pods are Running. What is the single most likely cause?
2. The controller log shows upstream prematurely closed connection. Is this a timeout to tune, or an app problem?
3. With the Application Routing add-on, you edit the managed ConfigMap and your change vanishes. What is the supported way to make it stick?
4. A browser shows a TLS error and openssl s_client reports CN “Kubernetes Ingress Controller Fake Certificate.” What broke?
5. You see default backend - 404 for a host that should work. Name two things to check.

Answers

1. Readiness is failing (Pods are Running but not Ready, so none join Endpoints) — or the Service selector matches no Pods. kubectl describe pod Conditions decides which. Restarting the controller does nothing.
2. An app problem — the upstream reset the connection (crash, timeout in the app, keepalive mismatch). It is not a proxy-read timeout; raising timeouts will not fix a reset. Read the Pod logs at that moment.
3. Use the NginxIngressController custom resource (and az aks approuting for DNS/Key Vault). The add-on reconciles managed objects, so direct edits are reverted.
4. The Key Vault → CSI → Secret sync chain broke (one of: identity lacking Key Vault Secrets User, missing secretObjects, no Pod mounting the volume, or the Ingress referencing the wrong secretName). The fake cert is the tell, not a real expiry.
5. The ingressClassName (does it match the controller’s class?) and the host/pathType in the rule (does any rule actually match the request’s host and path?). kubectl describe ingress and kubectl get ingressclass.

Glossary

• Ingress controller — the reverse proxy (NGINX on AKS) that terminates TLS and routes HTTP(S) by host/path to Services; emits the 502/503/504 you see.
• Ingress object — the Kubernetes resource declaring host/path → Service routing rules and TLS.
• IngressClass — selects which controller handles an Ingress; the Application Routing add-on’s default is webapprouting.kubernetes.azure.com.
• Service — a stable virtual IP/DNS name that load-balances to Pods matching its label selector.
• Endpoints / EndpointSlice — the set of ready Pod IPs behind a Service; empty Endpoints is the classic 502.
• Readiness probe — the check that decides whether a Pod is ready to serve; only ready Pods join Endpoints.
• Application Routing add-on — Azure-managed NGINX ingress controller for AKS, configured via the NginxIngressController CRD and az aks approuting.
• NginxIngressController — the custom resource that customises the managed controller in a reconcile-safe way.
• Secret Store CSI driver — mounts secrets/certs from external stores (Azure Key Vault) into Pods as volumes and, with secretObjects, into Kubernetes Secrets.
• SecretProviderClass — declares which Key Vault, certificate, identity and target Secret the CSI driver should use.
• TLS Secret — a kubernetes.io/tls Secret holding the certificate and private key that the Ingress tls block references.
• Fake certificate — NGINX’s default self-signed “Kubernetes Ingress Controller Fake Certificate,” served when the real cert is missing — the tell that the sync chain broke.
• proxy-read-timeout — the NGINX annotation bounding how long it waits for an upstream response (default 60 s); exceeding it yields 504.
• targetPort — the container port a Service forwards to; a mismatch with the real listening port yields connect() failed / 502.

Next steps

• AKS Cluster Architecture: Control Plane vs Data Plane Explained — the cluster fundamentals beneath ingress.
• AKS Networking: Kubenet vs Azure CNI Models Explained — how Pod IPs are assigned, which changes how the controller reaches backends.
• AKS Pod Cannot Pull Image: ACR Authorization Failures — the companion playbook when the failure is image-pull, not ingress.
• Key Vault 403 Forbidden: Firewall, RBAC, Soft-Delete Recovery — deeper on the Key Vault access failures behind a broken cert sync.
• Application Gateway with WAF and End-to-End TLS — when you front AKS ingress with a WAF and re-encrypt to the backend.