Azure Arc-Enabled Kubernetes: GitOps, Policy, and Fleet Governance for Hybrid Clusters

A platform team running EKS in one account, GKE in another, and three on-prem clusters in a colo does not have a Kubernetes problem. It has a governance problem: there is no single place to assert “every cluster runs this GitOps config, denies privileged pods, ships logs to one workspace, and is reachable for debugging without poking inbound holes in five firewalls.” Every cluster is a snowflake with its own RBAC, its own admission controller (or none), its own log destination, and its own bastion. Azure Arc-enabled Kubernetes projects any conformant cluster into Azure Resource Manager as a Microsoft.Kubernetes/connectedClusters resource, so the same management-group hierarchy, Azure Policy assignments, and RBAC you already use for native Azure resources now reach the cluster — wherever it physically runs.

This walkthrough onboards a non-Azure cluster, then layers the four controls that actually matter at fleet scale: Flux v2 GitOps for desired-state config, Azure Policy (Gatekeeper) for admission guardrails, cluster connect for kubectl without inbound firewall changes, and Container Insights plus workload identity for observability and secretless Key Vault access. Throughout I assume you have cluster-admin on the target cluster and Owner (or sufficient RBAC) on the Azure side. The goal is not a demo of one cluster — it is the machinery that turns forty snowflakes into “one policy, one Git repo, one identity boundary.”

Arc projects the cluster; it does not run it. The control plane, scheduler, and your nodes stay exactly where they are. Arc adds a set of agents that maintain an outbound connection to Azure and reconcile ARM intent into the cluster. If Azure is unreachable, the cluster keeps serving traffic — only the management plane pauses.

What problem this solves

The pain is operational drift across a heterogeneous fleet. Without a projection layer, every governance question becomes N separate answers. “Are privileged pods blocked everywhere?” means SSHing into N clusters or trusting N different OPA setups. “Who can debug the loyalty cluster at 02:00?” means N bastions, N VPNs, and N firewall change tickets. “Where are the logs?” means N workspaces and no fleet-wide query. When an auditor asks “prove no cluster runs hostPath mounts,” you have no single control plane to answer from.

What breaks without it: configuration entropy (each cluster diverges from the golden baseline because changes are applied by hand), inconsistent security posture (one cluster forgot the admission webhook and now runs root containers), blind operations (an outage on an edge cluster is invisible until a human notices), and access sprawl (every team cuts inbound firewall holes for kubectl, each one a new attack surface). Who hits this: platform/SRE teams running multi-cloud or hybrid Kubernetes, regulated shops that must prove uniform controls, and edge fleets (retail, manufacturing, telco) where clusters sit behind carrier-grade NAT with no public ingress.

Learning objectives

By the end of this article you can:

• Explain the Arc agent architecture and the outbound-only connectivity model, and enumerate every required FQDN — including the *.servicebus.windows.net websocket dependency that breaks cluster connect when proxied.
• Onboard an on-prem or EKS/GKE cluster with az connectedk8s connect, including the proxy flags (--proxy-https, --proxy-skip-range, --proxy-cert) that locked-down networks actually need.
• Configure Flux v2 GitOps via the microsoft.flux extension with correctly scoped Kustomizations, prune=true, and dependsOn ordering, identically across Arc and AKS.
• Assign Azure Policy (Gatekeeper) initiatives at management-group scope, roll out safely in audit before deny, and exclude system namespaces so you do not block Arc’s own agents.
• Grant cluster connect access with Azure RBAC and use az connectedk8s proxy for kubectl with zero inbound firewall changes.
• Enable Container Insights with managed-identity auth (no workspace key in the cluster) and scope ingestion to control cost.
• Federate a user-assigned managed identity to a Kubernetes service account so pods read Key Vault secrets with no credential in the cluster.
• Operate the controls at fleet scale using management groups, tags, Azure Resource Graph inventory, and Bicep-as-intent so new clusters inherit the baseline automatically.

Prerequisites & where this fits

You should be comfortable with core Kubernetes (Deployments, namespaces, RBAC, admission webhooks), kubectl and kubeconfig contexts, and Helm at a basic level. On the Azure side you need an understanding of Azure Resource Manager, management groups, Azure RBAC role assignments, and Azure Policy assignments. Familiarity with GitOps as a concept (desired state in Git, a controller reconciles) makes section 3 land faster — if you want a refresher, the Flux CD GitOps: Monorepo, Kustomize, and Multi-Tenancy and Argo CD App-of-Apps Multi-Cluster GitOps deep-dives cover the upstream engines Arc wraps.

Where this fits in the bigger picture: Arc-enabled Kubernetes is the hybrid arm of a wider Azure governance story. Management groups and Policy initiatives are the same primitives you would use in an Azure landing zone management group and Azure Policy at scale. Arc for servers is the sibling for VMs — see Azure Arc-Enabled Servers: Machine Configuration & Extended Security Updates. If your target is actually a managed Azure cluster, much of this carries over to AKS day-two operations covered in AKS Day-Two: Upgrades & Fleet Operations.

Core concepts

Arc-enabled Kubernetes is a thin projection: a Helm release of agents inside the cluster, a resource in ARM, and a set of cluster extensions that deliver capabilities (Flux, Policy, Monitor, Key Vault). Internalize this vocabulary before the deep sections — every later table assumes it.

The two control planes

Arc gives you two distinct planes, and confusing them is the root of most early mistakes. The management plane is ARM: management groups, Policy assignments, role assignments, extension lifecycle. It is eventually consistent — Policy syncs roughly every 15 minutes, Flux on its own interval. The data plane is your cluster’s kube-apiserver, untouched and authoritative for what actually runs. Arc never inserts itself in the request path of your workloads; it only reconciles intent and brokers kubectl.

1. Agent architecture, connectivity, and outbound requirements

az connectedk8s connect installs a Helm release into the azure-arc namespace. The agents are all-outbound by design — there is no inbound listener Azure dials into. Each agent has a single, separable job; knowing which one owns what turns a vague “Arc is broken” into a targeted fix.

Every agent talks outbound over https://:443 and websockets. The non-obvious requirement is *.servicebus.windows.net with websockets enabled on your proxy/firewall — cluster connect rides Azure Relay over that endpoint, and a Layer-7 proxy that blocks websocket upgrades will let onboarding succeed but break kubectl-over-Arc later. This single trap accounts for the majority of “onboarded fine but proxy hangs” tickets.

Required outbound endpoints

The wildcard Service Bus endpoints resolve per-region; never hard-block them on a deny-by-default proxy without first expanding them for your regions. Expand with:

# Region-specific allowlist to replace the *.servicebus.windows.net wildcard
curl -s "https://guestnotificationservice.azure.com/urls/allowlist?api-version=2020-01-01&location=eastus"

There is no “Azure-initiated inbound” connectivity mode for Arc Kubernetes — it is outbound-only, which is precisely why it fits locked-down on-prem and multi-cloud egress postures. Choose your egress posture deliberately:

Connectivity status meanings

2. Onboard an on-prem or EKS/GKE cluster

Point your kubeconfig at the target cluster (kubectl config use-context my-eks), then prep the Azure side. Register the resource providers once per subscription — registration is asynchronous and can take ~10 minutes, so gate on it.

az extension add --name connectedk8s

az provider register --namespace Microsoft.Kubernetes
az provider register --namespace Microsoft.KubernetesConfiguration
az provider register --namespace Microsoft.ExtendedLocation

# Registration can take ~10 min; gate on it before connecting
az provider show -n Microsoft.Kubernetes --query registrationState -o tsv   # -> Registered

Create a resource group to hold the connected-cluster resources, then connect. connect uses the current kubeconfig context to deploy the Arc agents:

export RESOURCE_GROUP=rg-arc-fleet
export LOCATION=eastus
export CLUSTER_NAME=eks-prod-use1

az group create --name $RESOURCE_GROUP --location $LOCATION -o table

# Uses the CURRENT kubeconfig context to deploy the Arc agents
az connectedk8s connect \
  --name $CLUSTER_NAME \
  --resource-group $RESOURCE_GROUP \
  --location $LOCATION

connect installs its own Helm v3 binary under ~/.azure (it never touches a Helm you already have) and deploys the agents. The flags you will reach for most:

If the cluster egresses through a proxy, do not rely on HTTP_PROXY alone — pass it so the in-cluster agents inherit it. Always include the cluster’s service CIDR in --proxy-skip-range, or in-cluster service-to-service calls will be wrongly routed at the proxy:

az connectedk8s connect \
  --name $CLUSTER_NAME \
  --resource-group $RESOURCE_GROUP \
  --proxy-https https://proxy.corp.local:8080 \
  --proxy-http  http://proxy.corp.local:8080 \
  --proxy-skip-range 10.0.0.0/16,kubernetes.default.svc,.svc.cluster.local,.svc \
  --proxy-cert /etc/ssl/certs/corp-root.crt

--proxy-cert is only for injecting a trusted root the proxy presents; it is not required just to use a proxy. The three flags most environments actually need are --proxy-http, --proxy-https, and --proxy-skip-range.

Distribution support and what changes

Arc onboards any CNCF-conformant cluster. The distribution mostly affects telemetry and which extensions are validated, not whether onboarding works.

Onboarding errors you will actually hit

3. Configure Flux v2 GitOps via the Arc extension

Arc’s GitOps is Flux v2 delivered as the microsoft.flux cluster extension (it installs fluxconfig-agent and fluxconfig-controller alongside the upstream source/kustomize/helm controllers). You rarely install the extension by hand — creating your first fluxConfigurations pulls it in automatically. Register the configuration with az k8s-configuration flux create, scoped at the cluster level, with one or more Kustomizations:

# Needs the k8s-configuration CLI extension
az extension add --name k8s-configuration

az k8s-configuration flux create \
  --name fleet-baseline \
  --cluster-name $CLUSTER_NAME \
  --resource-group $RESOURCE_GROUP \
  --cluster-type connectedClusters \
  --namespace cluster-config \
  --scope cluster \
  --url https://github.com/acme-platform/fleet-gitops \
  --branch main \
  --kustomization name=infra path=./infrastructure prune=true \
  --kustomization name=apps  path=./apps/prod prune=true dependsOn=["infra"]

flux create options that change behaviour

The mechanics worth internalising:

• --scope cluster lets the Kustomizations create cluster-scoped objects (CRDs, namespaces, ClusterRoles). Use --scope namespace for tenant-confined configs that may only touch their own namespace.
• prune=true is non-negotiable for real GitOps: delete a manifest from Git and Flux garbage-collects the object from the cluster. Without it, Git stops being the source of truth.
• dependsOn orders reconciliation — apps waits for infra to go Ready, so your ingress controller and CRDs land before the workloads that need them.
• The same command works against AKS by passing --cluster-type managedClusters. That symmetry is the whole point: one Git repo, one CLI, identical config across Arc and AKS.

Source kinds and how each authenticates

For a connected (non-AKS) cluster you do not need a managed identity to read a public Git repo — the source controller pulls directly. For private repos, pass --https-user/--https-key (PAT) or SSH key material; for Azure Blob sources with workload identity, the azblob kind federates to a UAMI (see section 7).

Flux config status and reconciliation states

Force a reconcile without waiting for the interval by annotating the source/Kustomization (flux reconcile ... if the Flux CLI is installed), or simply bump a commit. On Arc, the config-agent will also re-pull on the next ARM sync.

4. Apply Azure Policy (Gatekeeper) at fleet scope

Azure Policy for Kubernetes extends Gatekeeper v3 (the OPA admission webhook) so you can author guardrails once in ARM and enforce them as in-cluster admission decisions across the fleet. Install the extension per cluster, then assign initiatives at a scope that covers many clusters. Register the provider and install the extension (Microsoft.PolicyInsights):

az provider register --namespace Microsoft.PolicyInsights

az k8s-extension create \
  --cluster-type connectedClusters \
  --cluster-name $CLUSTER_NAME \
  --resource-group $RESOURCE_GROUP \
  --extension-type Microsoft.PolicyInsights \
  --name azurepolicy

Now assign a built-in initiative. The Pod Security baseline standards for Linux workloads initiative (a8640138-9b0a-4a28-b8cb-1666c838647d) bundles the deny rules most teams want — no privileged containers, no host namespaces, no hostPath, drop dangerous capabilities. Assign it at a management group so it lands on every connected cluster underneath, and exclude the system namespaces (otherwise you will block Arc’s own agents):

az policy assignment create \
  --name "psp-baseline-fleet" \
  --display-name "Pod Security baseline - Arc fleet" \
  --policy-set-definition "a8640138-9b0a-4a28-b8cb-1666c838647d" \
  --scope "/providers/Microsoft.Management/managementGroups/mg-arc-prod" \
  --params '{
    "effect": { "value": "deny" },
    "excludedNamespaces": { "value": ["kube-system","gatekeeper-system","azure-arc"] }
  }'

Policy effects and what each does in-cluster

The Kubernetes add-on supports audit, deny, and disabled effects. There is no deployIfNotExists inside the cluster — remediation of K8s objects is via GitOps, not Policy mutation.

Built-in initiatives worth knowing

Common single-rule built-ins (assemble custom initiatives)

Two operational realities to respect:

1. Roll out in audit before deny. Set effect to audit, watch the compliance results in Azure Policy for a week, fix the violators, then flip to deny. Flipping straight to deny on a brownfield cluster will reject existing Deployments on their next rollout and page you at 02:00.
2. Constraints are pulled, not instant. The add-on syncs assignments roughly every 15 minutes and writes Gatekeeper Constraint objects whose names start with azurepolicy-. Inspect them in-cluster with kubectl get constrainttemplates and kubectl get constraints.

Policy troubleshooting playbook

For org-specific rules beyond the built-ins (e.g. “all images must come from acme.azurecr.io”), author a custom constraint template + Rego and ship it as a custom policy definition — same assignment model, same fleet scope. If you treat policy definitions as source-controlled artifacts, the Azure Policy as Code pipeline pattern applies unchanged here.

5. Cluster connect: kubectl without inbound firewall changes

This is the feature that wins over on-prem teams. The clusterconnect-agent holds an outbound channel open; az connectedk8s proxy uses your Azure token to open a local proxy and writes a kubeconfig that targets it. No inbound port, no VPN, no bastion. First grant access. With Azure RBAC, assign the user/group a built-in role at the cluster scope — no kubectl ClusterRoleBinding required:

ARM_ID=$(az connectedk8s show -n $CLUSTER_NAME -g $RESOURCE_GROUP --query id -o tsv)
AAD_ID=$(az ad signed-in-user show --query id -o tsv)

# "Cluster User Role" grants the cluster-connect channel; "Viewer/Writer" grants in-cluster RBAC
az role assignment create --role "Azure Arc Enabled Kubernetes Cluster User Role" --assignee $AAD_ID --scope $ARM_ID
az role assignment create --role "Azure Arc Kubernetes Viewer" --assignee $AAD_ID --scope $ARM_ID

Arc Kubernetes built-in roles

The Cluster User Role only opens the channel; it grants no in-cluster permissions. You must also assign a Viewer/Writer/Admin role for the request to do anything once impersonated. Granting one without the other is the classic “I can connect but everything is forbidden” mistake.

Then open the proxy (it blocks the shell) and run kubectl from a second shell:

# Shell 1 - opens the proxy, blocks
az connectedk8s proxy -n $CLUSTER_NAME -g $RESOURCE_GROUP

# Shell 2 - normal kubectl, routed over the Arc channel
kubectl get pods -A

If you prefer native Kubernetes RBAC over Azure RBAC, bind a service account token instead and pass --token $TOKEN to the proxy command. Either way, the request path is: your token → Azure Relay → clusterconnect-agent → kube-aad-proxy (Entra auth + user impersonation) → kube-apiserver. The impersonation step is why a fleet-wide Azure Arc Kubernetes Viewer role gives read-only kubectl on every cluster at once.

Azure RBAC vs native Kubernetes RBAC for connect

Cluster connect failure modes

6. Enable Azure Monitor Container Insights

Ship stdout/stderr logs, inventory, and container metrics from every Arc cluster into one Log Analytics workspace via the Microsoft.AzureMonitor.Containers extension. Use managed identity auth (amalogs.useAADAuth=true) so there is no workspace key sitting in the cluster:

WORKSPACE_ID="/subscriptions/<sub>/resourceGroups/rg-observability/providers/Microsoft.OperationalInsights/workspaces/law-fleet"

az k8s-extension create \
  --name azuremonitor-containers \
  --cluster-name $CLUSTER_NAME \
  --resource-group $RESOURCE_GROUP \
  --cluster-type connectedClusters \
  --extension-type Microsoft.AzureMonitor.Containers \
  --configuration-settings \
      logAnalyticsWorkspaceResourceID=$WORKSPACE_ID \
      amalogs.useAADAuth=true

The extension deploys the ama-logs DaemonSet (every node) and ama-logs-rs ReplicaSet (cluster-level) into kube-system. To control ingestion cost on chatty clusters, scope collection to specific namespaces with dataCollectionSettings at install time:

az k8s-extension create \
  --name azuremonitor-containers \
  --cluster-name $CLUSTER_NAME \
  --resource-group $RESOURCE_GROUP \
  --cluster-type connectedClusters \
  --extension-type Microsoft.AzureMonitor.Containers \
  --configuration-settings amalogs.useAADAuth=true \
      dataCollectionSettings='{"interval":"1m","namespaceFilteringMode":"Include","namespaces":["prod","ingress"],"enableContainerLogV2":true}'

Container Insights configuration settings

Key Container Insights tables (KQL)

Once data lands, query the whole fleet from one workspace. Container logs carry the cluster identity, so a single KQL query slices across every onboarded cluster:

ContainerLogV2
| where TimeGenerated > ago(1h)
| where LogLevel in ("error","critical")
| summarize Errors = count() by Computer, ContainerName, _ResourceId
| sort by Errors desc

Note the migration: the legacy Helm-chart onboarding for the Container Insights agent is retired. On Arc, install via the Microsoft.AzureMonitor.Containers extension — that is the supported path and the one that participates in extension lifecycle/upgrades.

If you want metrics in Prometheus/Grafana rather than (or alongside) Log Analytics, the managed Prometheus/Grafana pattern in Azure Monitor: Managed Prometheus & Managed Grafana for AKS applies to Arc clusters via the metrics extension. For shaping ingestion with data collection rules, see Azure Monitor: Data Collection Rules, Workbooks & Alerting.

7. Workload identity and Key Vault secret access

Static secrets in manifests are the failure mode Arc lets you finally kill. The Azure Key Vault Secrets Provider extension (Microsoft.AzureKeyVaultSecretsProvider) installs the Secrets Store CSI Driver plus the Azure provider, so pods mount Key Vault secrets as files on tmpfs with no credential in the cluster:

az k8s-extension create \
  --cluster-name $CLUSTER_NAME \
  --resource-group $RESOURCE_GROUP \
  --cluster-type connectedClusters \
  --extension-type Microsoft.AzureKeyVaultSecretsProvider \
  --name akvsecretsprovider \
  --configuration-settings \
      secrets-store-csi-driver.enableSecretRotation=true \
      secrets-store-csi-driver.rotationPollInterval=2m \
      secrets-store-csi-driver.syncSecret.enabled=true

Key Vault CSI extension settings

For the auth itself, federate a user-assigned managed identity to a Kubernetes service account (workload identity) so the CSI provider exchanges the pod’s projected token for an Entra token — no client secret anywhere. A SecretProviderClass ties the service account to the vault:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: app-kv
  namespace: prod
spec:
  provider: azure
  parameters:
    clientID: "<USER_ASSIGNED_CLIENT_ID>"   # the federated UAMI
    keyvaultName: "kv-acme-prod"
    tenantId: "<TENANT_ID>"
    objects: |
      array:
        - |
          objectName: db-connection-string
          objectType: secret
  # Optional: project mounted secrets into a native K8s Secret for env vars
  secretObjects:
    - secretName: app-db
      type: Opaque
      data:
        - objectName: db-connection-string
          key: DB_CONN

Grant the UAMI Key Vault Secrets User on the vault via Azure RBAC, federate it to the service account’s OIDC subject, then any pod using that service account and mounting this SecretProviderClass reads the secret. Because the access is scoped per-service-account, you get least privilege and clean per-app audit instead of one node-wide credential.

Secret access patterns compared

Workload identity federation mapping

Rotation caveat: enableSecretRotation=true refreshes the mounted file on the poll interval. Apps that read the file each request pick up new values automatically; apps that load secrets once at boot, or consume the synced Secret as env vars, still need a restart to see a rotated value. Env vars are snapshotted at pod start — the kernel cannot rewrite a running process’s environment.

The same federation model underpins Azure Key Vault Workload Identity for Secrets and the AKS-flavoured Secrets Store CSI with Key Vault sync & rotation; the deep mechanics of federated credentials are in Entra Managed Identities: User-Assigned, FIC & RBAC. For rotation strategy across the vault itself, see Azure Key Vault Secret Rotation with Managed Identity.

8. Scale governance across many clusters

Onboarding one cluster is a demo. Governing forty is the job. Three primitives make Arc fleet-ready.

Management groups carry policy and RBAC. Place subscriptions (and therefore their connected clusters) under a management-group hierarchy and assign Policy initiatives + Arc Kubernetes roles at the MG level. A new cluster onboarded into any child subscription inherits the baseline the moment it appears — you do not touch it cluster-by-cluster.

Tags drive targeting and chargeback. Tag connected clusters with environment, owner, and data-classification, then write policy assignments that key off tags or build Azure Resource Graph queries for fleet inventory:

// Every Arc cluster, its agent version, and connectivity health
resources
| where type == "microsoft.kubernetes/connectedclusters"
| project name, location,
          distribution = properties.distribution,
          k8sVersion   = properties.kubernetesVersion,
          connectivity = properties.connectivityStatus,
          agentVersion = properties.agentVersion,
          env = tags.environment
| order by connectivity asc

Fleet inventory queries worth saving

GitOps is the fleet rollout mechanism. Because the same az k8s-configuration flux create works across every connected cluster, codify it. The Bicep below registers the Flux config as ARM intent, so onboarding a cluster and deploying a Policy assignment that requires this config means new clusters self-bootstrap their baseline:

resource fluxBaseline 'Microsoft.KubernetesConfiguration/fluxConfigurations@2023-05-01' = {
  name: 'fleet-baseline'
  scope: connectedCluster      // the Microsoft.Kubernetes/connectedClusters resource
  properties: {
    scope: 'cluster'
    namespace: 'cluster-config'
    sourceKind: 'GitRepository'
    gitRepository: {
      url: 'https://github.com/acme-platform/fleet-gitops'
      repositoryRef: { branch: 'main' }
    }
    kustomizations: {
      infra: { path: './infrastructure', prune: true }
      apps:  { path: './apps/prod', prune: true, dependsOn: ['infra'] }
    }
  }
}

The end state: a cluster joins the fleet, ARM applies the inherited Policy initiative (admission guardrails), the Flux config (desired state), the Monitor extension (telemetry), and the role assignments (kubectl access) — all without a human SSHing into the cluster.

Architecture at a glance

Read the diagram left to right as the path that intent travels and telemetry returns. On the far left, the platform SRE and the Git repository are the sources of truth — humans issue az commands and assign Policy, while desired configuration lives as YAML on branch: main. That intent lands in the Azure control plane zone: a management group that carries Policy and RBAC down to every child cluster, the Azure Policy engine that compiles initiatives into Gatekeeper constraints, and the Log Analytics workspace that all clusters report into. Critically, nothing in this zone reaches into your network — it publishes intent to ARM and waits.

The Arc agents zone is the bridge, living in the azure-arc namespace inside your cluster and dialling outbound only. The clusterconnect agent (badge 1) holds the Azure Relay channel open over *.servicebus.windows.net:443 so kubectl works with no inbound port; the config + extension manager (badge 3) pulls Flux/Policy/Monitor intent and reconciles it; and kube-aad-proxy (badge 4) authenticates each kubectl caller with Entra and impersonates them against the apiserver. Finally, the hybrid cluster zone — EKS, GKE, or k3s — keeps its kube-apiserver exactly where it was, runs your workloads with prune=true GitOps, and mounts Key Vault secrets via the CSI driver (badge 5). The two return flows (badge 2 marks the Policy admission decision; the amber arrow carries inventory and logs back) close the loop: intent flows right, evidence flows left, and not one inbound firewall rule was opened.

Real-world scenario

A retail platform team ran 28 store-edge clusters (k3s on ruggedised hardware, one per regional distribution center) plus a GKE cluster for their loyalty service. Security mandated two things the existing setup could not deliver: a centrally enforced ban on privileged containers, and break-glass kubectl access for the on-call SRE without opening inbound ports on store networks — the stores sat behind carrier-grade NAT with no public ingress and a websocket-stripping Layer-7 proxy.

The constraint that bit them first was the proxy. Onboarding succeeded, Flux reconciled, Policy enforced — but az connectedk8s proxy hung, because cluster connect rides Azure Relay over *.servicebus.windows.net and the proxy silently dropped the websocket upgrade. The fix was an allow-rule for the resolved, regional Service Bus endpoints with websockets explicitly permitted, expanded from the wildcard via the guest-notification allowlist API:

# Run per store region; feed results into the proxy allowlist with websockets enabled
for region in eastus westus2 centralus; do
  curl -s "https://guestnotificationservice.azure.com/urls/allowlist?api-version=2020-01-01&location=$region"
done

With egress fixed, they assigned the Pod Security baseline initiative at the mg-retail-edge management group — in audit first. The audit results surfaced exactly the violators they expected: a legacy label-printer DaemonSet that ran privileged to access /dev. They refactored it to a specific device plugin, then flipped the initiative to deny. New store clusters now onboard via a pipeline that runs az connectedk8s connect, and inherit the deny policy and the Flux baseline automatically from the management group — zero per-store configuration.

On-call SREs hold Azure Arc Enabled Kubernetes Cluster User Role at the MG scope, giving them az connectedk8s proxy into any store on earth without a single inbound firewall rule. The whole 28-cluster fleet went from “28 snowflakes” to “one policy, one Git repo, one identity boundary” in under a sprint. The lasting win was not any single control — it was that adding store #29 became a pipeline run, not a project.

Advantages and disadvantages

When each matters: the outbound-only model is decisive for edge and regulated on-prem where inbound is simply not allowed. Management-group inheritance is the multiplier once you pass ~5 clusters — below that, the per-cluster effort is small and Arc’s value is mostly uniformity, not labour saved. The eventual-consistency caveat matters most for security expectations: do not assume a freshly assigned deny is enforced the instant you click save; budget ~15 minutes and verify with a negative test. The egress dependency is the thing that bites in practice — almost every painful Arc incident traces back to a firewall or proxy, not to Arc itself.

Hands-on lab

This lab onboards a local kind cluster (free, no cloud cost beyond minimal ARM/Log Analytics) and layers Policy + cluster connect. You need Azure CLI, kubectl, Docker, and an Azure subscription.

# 0) Prereqs
az login
az extension add --name connectedk8s
az extension add --name k8s-configuration
az extension add --name k8s-extension

# 1) A throwaway local cluster
kind create cluster --name arc-lab
kubectl config use-context kind-arc-lab

# 2) Register providers (idempotent; wait for Registered)
for ns in Microsoft.Kubernetes Microsoft.KubernetesConfiguration Microsoft.ExtendedLocation Microsoft.PolicyInsights; do
  az provider register --namespace $ns
done
az provider show -n Microsoft.Kubernetes --query registrationState -o tsv   # -> Registered

# 3) Onboard
export RESOURCE_GROUP=rg-arc-lab LOCATION=eastus CLUSTER_NAME=kind-arc-lab
az group create -n $RESOURCE_GROUP -l $LOCATION -o table
az connectedk8s connect -n $CLUSTER_NAME -g $RESOURCE_GROUP -l $LOCATION

# Expected: connectivityStatus -> Connected; azure-arc pods Running
az connectedk8s show -n $CLUSTER_NAME -g $RESOURCE_GROUP --query connectivityStatus -o tsv
kubectl get pods -n azure-arc

# 4) GitOps against a public repo
az k8s-configuration flux create \
  --name lab-baseline -g $RESOURCE_GROUP \
  --cluster-name $CLUSTER_NAME --cluster-type connectedClusters \
  --namespace cluster-config --scope cluster \
  --url https://github.com/Azure/gitops-flux2-kustomize-helm-mt \
  --branch main \
  --kustomization name=infra path=./infrastructure prune=true

# 5) Policy add-on + a deny baseline at the SUBSCRIPTION scope for the lab
az k8s-extension create --cluster-type connectedClusters \
  --cluster-name $CLUSTER_NAME -g $RESOURCE_GROUP \
  --extension-type Microsoft.PolicyInsights --name azurepolicy

SUB=$(az account show --query id -o tsv)
az policy assignment create \
  --name psp-baseline-lab \
  --policy-set-definition a8640138-9b0a-4a28-b8cb-1666c838647d \
  --scope "/subscriptions/$SUB" \
  --params '{"effect":{"value":"audit"},"excludedNamespaces":{"value":["kube-system","gatekeeper-system","azure-arc"]}}'

# 6) Cluster connect — grant yourself, then proxy
ARM_ID=$(az connectedk8s show -n $CLUSTER_NAME -g $RESOURCE_GROUP --query id -o tsv)
ME=$(az ad signed-in-user show --query id -o tsv)
az role assignment create --role "Azure Arc Enabled Kubernetes Cluster User Role" --assignee $ME --scope $ARM_ID
az role assignment create --role "Azure Arc Kubernetes Cluster Admin" --assignee $ME --scope $ARM_ID
# Shell 1: az connectedk8s proxy -n $CLUSTER_NAME -g $RESOURCE_GROUP
# Shell 2: kubectl get nodes      # routed over the Arc channel

# 7) TEARDOWN (avoid lingering cost)
az policy assignment delete --name psp-baseline-lab --scope "/subscriptions/$SUB"
az connectedk8s delete -n $CLUSTER_NAME -g $RESOURCE_GROUP --yes
az group delete -n $RESOURCE_GROUP --yes --no-wait
kind delete cluster --name arc-lab

Common mistakes & troubleshooting

These are the failure modes that actually generate tickets, in rough order of frequency. The first is responsible for more lost hours than the rest combined.

A fast negative test for Policy: kubectl run pwn --image=nginx --privileged=true -n prod should be denied by the Gatekeeper webhook once the baseline initiative is in deny mode. If it succeeds, your assignment scope or namespace exclusions are wrong, or the constraints have not synced yet.

# Verify each layer landed before you call a cluster "governed"
az connectedk8s show -n $CLUSTER_NAME -g $RESOURCE_GROUP --query connectivityStatus -o tsv   # -> Connected
kubectl get pods -n azure-arc          # all Running
az k8s-configuration flux show --name fleet-baseline -g $RESOURCE_GROUP \
  --cluster-name $CLUSTER_NAME --cluster-type connectedClusters \
  --query "statuses[].complianceState" -o tsv
kubectl get constraints                 # azurepolicy-* present
kubectl get ds ama-logs -n kube-system  # Monitor agent shipping

Best practices

• Treat egress as the dependency that decides everything. Pre-stage the FQDN allowlist (especially *.servicebus.windows.net with websockets) before onboarding, per region. Most Arc incidents are firewall incidents.
• Always roll policy audit → remediate → deny. Never flip a brownfield fleet straight to deny. Watch compliance for a week first.
• Exclude kube-system, gatekeeper-system, and azure-arc from every policy assignment. Forgetting this blocks Arc’s own agents.
• Assign Policy and RBAC at the management-group scope, not per cluster. Inheritance is the whole point; per-cluster assignments do not scale and drift.
• Set prune=true on every Kustomization. Without it, Git is not the source of truth and deletes silently leak.
• Use dependsOn to land CRDs/ingress before workloads that need them, or reconciliation order will bite you on a fresh cluster.
• Grant both the Cluster User Role and an in-cluster role (Viewer/Writer/Admin). One without the other is useless.
• Use managed-identity auth everywhere — amalogs.useAADAuth=true for Monitor, workload identity for Key Vault. No keys in the cluster, ever.
• Scope Container Insights ingestion with dataCollectionSettings on chatty clusters; default “collect everything” is a cost trap at fleet scale.
• Codify the baseline as Bicep fluxConfigurations so new clusters self-bootstrap. Onboarding should be a pipeline run, not a runbook.
• Tag every connected cluster (env, owner, data-classification) and keep saved Resource Graph queries for inventory and drift.
• Pin agent/extension versions in change-controlled fleets (--disable-auto-upgrade) and roll upgrades through a ring, the same way you would for AKS day-two upgrades.

Security notes

Arc’s security model is “outbound-only management plane + least-privilege identity,” and you should keep it that way deliberately.

The identity boundary is the crown jewel: because human access is Entra-mediated and impersonated per request, you get one place (Entra + Activity log) to answer “who touched which cluster, when.” Protect the Arc Cluster Admin role with PIM and just-in-time elevation; a standing Cluster Admin at MG scope is a standing cluster-admin on every cluster in the fleet.

Cost & sizing

Arc-enabled Kubernetes has no per-cluster Arc fee for the core control plane (onboarding, cluster connect, GitOps, Policy). What you pay for is the value-added services that ride on top — chiefly log/metric ingestion and any Arc-enabled data/app services. Sizing is therefore mostly an observability-cost exercise plus a small in-cluster resource footprint for the agents and add-ons.

Right-sizing the in-cluster footprint

Practical guidance: on a 28-cluster edge fleet, the dominant line item is almost always Container Insights ingestion, not anything Arc-specific. Turn on namespaceFilteringMode: Include for just prod/ingress, raise the metric interval to 5m where 1-minute resolution is not needed, and move long-tail logs to a cheaper retention/archive tier. Free-tier-wise, Log Analytics gives a small daily ingestion allowance and 31 days retention at no charge — enough to validate the pipeline in the lab above without a meaningful bill. (INR figures approximate at ~₹84/USD and vary by region and commitment tier; treat them as order-of-magnitude.)

Interview & exam questions

1. What does Arc-enabled Kubernetes actually add to a cluster, and what does it not touch? It installs a Helm release of outbound-only agents in the azure-arc namespace and creates a connectedClusters ARM resource; it adds Policy/GitOps/Monitor/Key Vault via cluster extensions. It does not change your control plane, scheduler, nodes, or the data path of your workloads — Arc reconciles intent and brokers kubectl, nothing more.

2. Why is *.servicebus.windows.net special in the egress allowlist? Cluster connect rides Azure Relay over that endpoint using websockets. A Layer-7 proxy that blocks websocket upgrades will let onboarding succeed but break kubectl-over-Arc, which is a notoriously confusing failure. You must allow the resolved regional FQDNs with websockets enabled.

3. Why roll Azure Policy out in audit before deny? deny causes Gatekeeper to reject non-compliant admissions, so flipping straight to deny on a brownfield cluster rejects existing Deployments on their next rollout. audit surfaces violators without blocking, letting you remediate first, then promote to deny safely.

4. Which namespaces must you exclude from policy assignments, and why? kube-system, gatekeeper-system, and azure-arc. They run system and Arc agent workloads that may legitimately need elevated settings; failing to exclude them can block Arc’s own agents and brick management.

5. Explain the cluster-connect request path. Your Azure token → Azure Relay → clusterconnect-agent → kube-aad-proxy (Entra authN + user impersonation) → kube-apiserver. Impersonation is why a fleet-wide Viewer role grants read-only kubectl on every cluster at once.

6. Cluster User Role is assigned but everything returns forbidden. Why? The Cluster User Role only opens the connect channel; it grants no in-cluster permissions. You must also assign an in-cluster role (Viewer/Writer/Admin) for the impersonated request to do anything.

7. What does prune=true change, and why is it non-negotiable? With prune=true, deleting a manifest from Git causes Flux to garbage-collect the corresponding object from the cluster. Without it, deletions never propagate, so Git stops being the authoritative source of truth.

8. How do you give every new cluster the baseline automatically? Assign Policy initiatives and Arc Kubernetes roles at a management group, and register the Flux config as Bicep fluxConfigurations. A cluster onboarded into any child subscription inherits the policy, GitOps config, and access without per-cluster work.

9. How do you read a Key Vault secret with no credential in the cluster? Install the Key Vault Secrets Provider extension, federate a user-assigned managed identity to a Kubernetes service account (workload identity), and bind a SecretProviderClass. The CSI provider exchanges the pod’s projected token for an Entra token — no client secret anywhere.

10. A secret was rotated but the app still uses the old value. Why, and what fixes it? Rotation refreshes the mounted file on tmpfs, but environment variables are snapshotted at pod start and the synced Secret-as-env path is static. Apps that read the file per request pick up changes; apps that load at boot or use env vars need a pod restart.

11. How does Arc Kubernetes differ from AKS for these controls? The controls are nearly identical — same az k8s-configuration flux, same Policy initiatives, same extensions — but Arc uses --cluster-type connectedClusters and runs on non-Azure clusters with an outbound-only agent set, while AKS uses managedClusters and is already in Azure. The symmetry is intentional.

12. Which certs map to which exam? This material maps to AZ-305 (designing governance/hybrid) and AZ-104 (Arc, Policy, RBAC), with Kubernetes depth overlapping CKA/CKS for the in-cluster admission and RBAC mechanics.

Quick check

1. What single egress FQDN, if proxied without websockets, lets onboarding succeed but breaks kubectl-over-Arc?
2. You assigned the Arc Cluster User Role but every kubectl command is forbidden. What did you forget?
3. Name the three namespaces you must exclude from a fleet policy assignment.
4. What does prune=true do when you delete a manifest from Git?
5. Why does a freshly assigned deny policy sometimes still admit a privileged pod for a few minutes?

Answers

1. *.servicebus.windows.net — cluster connect rides Azure Relay over websockets there. Allow the resolved regional FQDNs with websockets enabled.
2. An in-cluster role. The Cluster User Role only opens the connect channel; you must also assign Viewer/Writer/Admin for the impersonated request to have permissions.
3. kube-system, gatekeeper-system, and azure-arc — excluding them keeps the policy from blocking system and Arc agent workloads.
4. Flux garbage-collects the corresponding object from the cluster, keeping Git as the source of truth.
5. The Policy add-on syncs assignments roughly every 15 minutes and writes the azurepolicy-* Gatekeeper constraints on that cadence; until they land (or if the scope is wrong), admission is not yet enforced.

Glossary

Next steps

• Azure Policy at Scale: Governance with Management Groups & Initiatives — go deeper on the policy engine you assigned here, fleet-wide.
• Flux CD GitOps: Monorepo, Kustomize & Multi-Tenancy — structure the Git repo your Arc clusters reconcile from.
• Azure Key Vault Workload Identity for Secrets — the federation model behind secretless secret access.
• Azure Arc-Enabled Servers: Machine Configuration & Extended Security Updates — the VM sibling that completes your hybrid Arc estate.
• AKS Day-Two: Upgrades & Fleet Operations — apply the same fleet discipline to managed Azure clusters.