GCP Enterprise Architecture: Hybrid & Multi-Cloud

In a nutshell

Level: Advanced · Time: ~52 min

Imagine you run an airline. You do not manage each aircraft as a unique, hand-built machine with its own maintenance manual, its own trained crew, and its own safety checklist — that way lies chaos the moment you own more than a couple of planes. Instead you run a fleet: one set of standard operating procedures, one operations center that sees every aircraft on one screen, one training standard, and reliable routes connecting your hubs. Any plane you buy — or inherit from an airline you acquired — gets folded into the same fleet and follows the same rules from day one. Nobody re-invents “how do we service a 737” per aircraft.

This lesson is about doing exactly that for Kubernetes clusters scattered across your own data center, Amazon’s cloud (AWS), and Google’s cloud (GCP). Left alone, every cluster becomes a snowflake: its own security rules, its own way of proving who is allowed to do what, its own monitoring, its own “wait, is this one even compliant?” Once you have more than a handful, the cluster stops being a useful unit and becomes a liability — every new one multiplies the drift, the patching surface, and the confusion. The Google product that turns that sprawl into a governed fleet is GKE Enterprise (you will still hear its former name, Anthos): you register every cluster — no matter which cloud it lives in — as a member, then govern all of them with one policy source, one identity model, one service mesh, and one compliance dashboard.

There are two halves, and keeping them separate is the whole trick. The management plane (the fleet) is how you govern and observe every cluster as one thing. The data plane (the network) is how packets actually travel privately between your data center, AWS, and GCP — built from Cloud Interconnect (a private high-speed line into your data center), HA VPN (an encrypted backup, and the link across to AWS), and Network Connectivity Center (the hub that ties the links into one routable network). You cannot govern what you cannot reach; reaching without governing is how the sprawl started in the first place. This article builds both halves as a single reference architecture you can shrink to two clusters or grow to hundreds without redrawing the diagram.

Prerequisites — what to know first. You will get the most from this if you already understand what a Kubernetes cluster is (nodes, pods, namespaces, RBAC) and the basics of GKE (GKE deep dive), plus roughly what a VPC, subnet, and VPN are (HA VPN, Cloud Router & BGP, Shared VPC). Helpful but not required: GitOps (declaring desired state in Git), service mesh (Istio) and mTLS, and keyless auth via Workload Identity Federation (keyless CI/CD with WIF). If any of those is new, skim those lessons first — this one assembles them into an estate-wide architecture.

After this lesson you will be able to:

• Explain the difference between the management plane (fleet) and the data plane (network), and why the architecture keeps them apart.
• Describe how a cluster in AWS or on-prem registers into a GCP fleet (the Connect Agent, outbound-only) and inherits governance as a fleet default.
• Choose among Cloud Interconnect, HA VPN, Cross-Cloud Interconnect, and Network Connectivity Center for a given hybrid/multi-cloud link — and justify the choice.
• Explain how Config Sync + Policy Controller turn “the standard” from a wiki page into an enforced control loop.
• Reason about cross-cloud identity federation (Workload Identity → AWS IAM) and cross-cloud analytics with BigQuery Omni without moving keys or data.
• Weigh the honest trade-offs of multi-cloud — avoiding lock-in and honoring data gravity — against its real costs in egress, latency, and operational complexity.

The hardest question in any multi-cloud strategy is not “how do I run a container on three clouds” — Kubernetes solved that years ago — it is “how do I operate, secure, and reason about thirty clusters spread across two data centers, AWS, and GCP as one thing, without thirty separate audit stories, thirty RBAC models, and thirty CI/CD snowflakes.” The moment you have more than a couple of clusters, the cluster stops being the unit of management and becomes a liability: every new one multiplies the policy drift, the patching surface, the “wait, which cluster is that service even on” confusion. The discipline that turns a sprawl of clusters into a governable platform is fleet management — treating a set of clusters as a single object with one identity model, one policy source, one service-mesh control plane, and one place to ask “is everything compliant right now.” On Google Cloud the product that does this is GKE Enterprise (the capability formerly and still widely branded Anthos): a fleet abstraction that spans GKE on Google Cloud, GKE on AWS/Azure, and Google Distributed Cloud (GDC) on-prem (VMware or bare metal), with Config Sync for GitOps policy, Policy Controller for guardrails, Cloud Service Mesh for identity-based traffic, and fleet Workload Identity so every workload on every cluster proves who it is the same way.

That fleet only means something if the clusters can actually reach each other and reach Google’s APIs over a private, reliable network — which is the other half of this article. A multi-cloud control plane riding over flaky public-internet tunnels is a demo, not a platform. So the connectivity spine here is Cloud Interconnect (Dedicated or Partner) for the high-bandwidth, low-jitter link to the data center, HA VPN as the encrypted day-one and permanent-backup path (and the primary link into AWS), Network Connectivity Center as the hub that ties the spokes together, and Cloud DNS for the hybrid name resolution that makes service discovery work across the boundary. The thesis of the article is that these two halves — the fleet (control) and the connectivity (data path) — are a single architecture: you cannot govern what you cannot reach, and reaching without governing is how you end up with the very sprawl you were trying to escape.

The business scenario

Helvetia Risk is a fictional but representative mid-market specialty insurer: roughly 3,400 employees, writing commercial property and marine cover, with around ₹4,100 crore (~USD 490M) in annual gross written premium. Their estate is the textbook “we ended up multi-cloud by accident, and now it’s strategy” shape that lands on a platform team three years into a cloud journey:

• A regulated core that cannot leave the building (yet). The policy-administration and claims systems run on VMware on-premises in a Zurich data center, next to a mainframe-adjacent actuarial system and a Db2 database the regulator’s audit cycle is built around. A full exit is a multi-year program; in the meantime new digital workloads must run next to this data, not 200 ms away across the public internet.
• A born-in-AWS acquisition. Eighteen months ago they bought a digital-broker startup whose entire stack — quoting APIs, a customer portal, a pricing engine — runs on EKS in AWS (eu-west-1). Re-platforming it onto GCP wholesale is not on the table; the team, the tooling, and the data-residency commitments to brokers are all anchored to AWS.
• A new GCP-native bet. The strategic build-out — a real-time pricing and fraud-scoring platform, a data/AI layer on BigQuery and Vertex AI — is going onto GKE on Google Cloud in europe-west6 (Zurich), chosen for the Swiss region and the data/AI services.
• Three Kubernetes worlds, zero shared anything. On-prem runs a hand-rolled kubeadm cluster; AWS runs EKS with IAM-based RBAC and AWS Load Balancer Controller; GCP runs GKE. Three RBAC models, three secret stores, three ingress patterns, three CI/CD pipelines, three monitoring stacks. A single auditor question — “show me that mTLS is enforced for all PII-handling services across the estate” — takes two weeks and three teams to answer, and the answer is “mostly.”
• Policy drift is a standing finding. Each cluster’s network policies, pod-security settings, and allowed registries diverged the moment they were created. The security team’s “standard” exists in a wiki, not in the clusters. Every audit surfaces a cluster that quietly fell out of compliance.
• Latency and integration pain. The new GCP pricing engine needs the policyholder and claims data that lives on-prem in Zurich; the AWS quoting API needs to call the GCP fraud-scoring service. Today both cross the public internet with one-off VPN tunnels, variable latency, and no consistent service identity — so the security model is “trust the IP range,” which the security team rightly hates.

The mandate is therefore not “migrate to GCP.” It is: operate all three Kubernetes environments as one fleet with one policy source, one identity model, and one compliance dashboard; connect Zurich on-prem to GCP over a private, high-bandwidth, redundant link; bring the AWS workloads under the same governance and onto the same private network as the GCP and on-prem ones; give services a consistent, identity-based way to call each other across all three clouds with mTLS; and make “prove this control is enforced everywhere” a dashboard query, not a project. Every one of those maps onto a specific GKE Enterprise or connectivity primitive — that is what makes this a hybrid/multi-cloud architecture and not just “we have clusters in three places.”

Crucially the shape scales both ways. A 15-person team with one on-prem cluster and one GKE cluster deploys the identical pattern — a single fleet, Config Sync pointing at one Git repo, HA VPN to the data center, Cloud Service Mesh across two members — for a modest monthly spend, and grows into the Interconnect-backed, multi-cloud, dozens-of-clusters version without redrawing the diagram or relearning the model. That is what makes it a reference architecture rather than a megacorp special.

Architecture overview

The architecture has two planes that are easy to conflate and must be kept distinct: the management plane (the fleet — how clusters are registered, governed, and observed as one) and the data plane (the network — how packets and service calls actually traverse on-prem, AWS, and GCP). The fleet is “Google Cloud’s view of all your clusters”; the network is “how those clusters and Google’s APIs are privately reachable.” You design them together but you reason about them separately.

Read the diagram in two bands and three columns: the horizontal band across the top is the fleet — governance (Config Sync, policy, mesh trust, identity) flowing down into every cluster; the horizontal band across the bottom is the private network spine — workload traffic flowing across; the three columns are your three environments (on-prem, AWS, GCP). The two bands never cross, and that separation is the architecture.

The management plane — one fleet, many members. Everything hangs off a single fleet (a fleet is scoped to one GCP project, the fleet host project). Into that fleet you register each cluster as a member, regardless of where it runs:

1. GKE on Google Cloud clusters in europe-west6 register natively.
2. EKS clusters in AWS register through the Connect Agent — a small deployment in the cluster that establishes an outbound, mTLS tunnel to Google and makes the cluster manageable from the fleet without opening any inbound port on the AWS side. (This is Anthos clusters attached / multi-cloud registration.)
3. On-prem clusters run as Google Distributed Cloud (software, on VMware), which is GKE’s on-prem distribution, and register the same way through Connect.

Once registered, every member gets the same fleet-wide capabilities, turned on as fleet features rather than per-cluster bolt-ons:

• Config Sync continuously reconciles each cluster against a Git repository (the GitOps source of truth) — namespaces, RBAC, network policies, quotas, the lot. The repo is the desired state; drift is auto-corrected. One repo (with per-cluster and per-scope overlays) governs all thirty clusters.
• Policy Controller (Google’s managed OPA Gatekeeper) enforces guardrails as admission policy — “no public LoadBalancer in prod,” “images only from our Artifact Registry,” “every namespace has a NetworkPolicy” — and continuously audits existing resources, surfacing violations on the fleet dashboard.
• Cloud Service Mesh (Google’s managed Istio) gives every member a service mesh with a shared root of trust, so a service on GKE and a service on EKS get SPIFFE identities from the same CA and can do mTLS to each other by service identity, not IP.
• Fleet Workload Identity establishes a common identity pool so a workload on any member authenticates to Google APIs (and, via federation, to AWS) as itself, with no static keys.

The data plane — a private spine, not the public internet. The fleet’s control traffic is outbound-only mTLS to Google, but the workloads need real network reachability across the three environments. That spine is built in layers:

1. A Shared VPC in GCP (host project owns the network; service projects attach) is the GCP-side backbone. Workload subnets for europe-west6 live here, with Private Google Access so nodes reach Google APIs over internal IPs, and Private Service Connect endpoints for consuming Google and partner services privately.
2. Cloud Interconnect (Dedicated, 2 × 10 Gbps in two edge locations) to Zurich on-prem. This is the high-bandwidth, low-jitter, private link the pricing engine uses to read policyholder/claims data. Two interconnects in two metro availability domains give a 99.99% SLA; Cloud Router runs BGP to exchange routes dynamically between on-prem and the VPC.
3. HA VPN as the always-on backup to on-prem and the primary path to AWS. HA VPN is a two-tunnel, 99.99%-SLA managed VPN. To on-prem it advertises the same prefixes as the Interconnect but at a lower BGP priority, so a total Interconnect failure fails over automatically with no operator action. To AWS, an HA VPN ↔ AWS VPN Gateway pair (two tunnels, BGP via Cloud Router and an AWS Transit Gateway / VGW) is the private path between the GCP VPC and the AWS VPC where EKS runs.
4. Network Connectivity Center (NCC) is the hub that makes this a network, not a set of point links. The on-prem (via VLAN attachments) and the AWS (via the HA VPN) connections become spokes on an NCC hub, so on-prem, AWS, and GCP can route to each other transitively through Google’s backbone — rather than you hand-stitching every pair.
5. Cloud DNS with forwarding and peering resolves names across the boundary: GCP private zones, inbound/outbound DNS forwarding to the on-prem resolver, and Cloud DNS for the mesh / GKE so a service in AWS can resolve and reach a GKE service by name.

How a real cross-cloud request flows. Trace a quote: a broker hits the customer portal on EKS (AWS). The portal needs a price, so it calls the pricing service on GKE (GCP). Because both clusters are members of one fleet with Cloud Service Mesh and a shared CA, the call is an mTLS connection authenticated by SPIFFE identity (spiffe://.../ns/pricing/sa/pricing-svc), not a trust-the-IP hop. The packet leaves the EKS pod, traverses the HA VPN tunnel (or the NCC spine) onto the GCP Shared VPC, and Cloud DNS resolves the pricing service to its mesh endpoint. The pricing service, in turn, needs policyholder history that lives on-prem in Zurich: that call goes out over Cloud Interconnect (BGP-routed via Cloud Router), reaching the Db2-fronting service in the VMware cluster — which is also a fleet member, so even that hop is mesh-secured and policy-governed. The response retraces the path. At no point does the traffic touch the public internet, and at every hop the calling workload proves its identity. Meanwhile, entirely out-of-band, Config Sync has guaranteed every one of those clusters is running the network policies and pod-security settings the Git repo mandates, and Policy Controller is continuously asserting none of them drifted.

The diagram in words. Picture three columns — On-prem (Zurich VMware), AWS (eu-west-1, EKS), GCP (europe-west6, GKE) — each containing one or more Kubernetes clusters. Across the top, spanning all three columns, sits a single horizontal band labeled Fleet (GKE Enterprise): Config Sync ← one Git repo, Policy Controller, Cloud Service Mesh (one CA), Fleet Workload Identity, and the unified Cloud Logging/Monitoring + fleet dashboard. Dashed lines run from every cluster up into that band (the Connect Agent’s outbound mTLS). Across the bottom, spanning all three columns, sits the network spine: GCP Shared VPC in the middle, a Cloud Interconnect (solid, fat) line to the on-prem column, an HA VPN (solid) line to the AWS column and a parallel HA VPN backup line to on-prem, all converging on a Network Connectivity Center hub, with Cloud DNS forwarding threaded across. Workload-to-workload calls are horizontal arrows along the bottom spine (mesh mTLS); governance flows are vertical arrows into the top band. The two bands never cross — that separation is the architecture.

Component breakdown

A handful of these choices carry the design and deserve the why, not just the what.

The fleet is the unit of management — not the cluster, not the project. The single most important mental shift is to stop operating clusters and start operating a fleet. Once a cluster is a member, you turn on Config Sync, Policy Controller, and the mesh as fleet-default configurations, so a newly registered cluster inherits the entire governance posture automatically — it joins the fleet already compliant rather than being configured into compliance afterward. Fleet scopes then let you carve the fleet into bounded sets (e.g. team-pricing, env-prod) so a team’s namespaces and the services they can see in the mesh are governed together across whatever clusters they span. The payoff is that “onboard the thirty-first cluster” and “onboard the third” are the same one-step operation, and the auditor’s “is X enforced everywhere” becomes a single dashboard, because everywhere is now a defined, enumerable thing.

Config Sync + Policy Controller is how “the standard” stops living in a wiki. The reason every cluster drifted is that the standard was documentation, not code — and documentation does not reconcile. Config Sync makes the Git repo the desired state: it continuously applies it and reverts anything that drifts, so a hand-edit on a cluster is undone within minutes. Policy Controller is the complementary half — it both blocks non-compliant resources at admission and continuously audits what already exists, reporting violations to the fleet dashboard. Together they convert governance from a periodic human review into a closed control loop: desired state declared once in Git, enforced and verified everywhere, with the dashboard as the live compliance report. This is the mechanism that makes the security team’s mandate (“prove the control is enforced”) a query rather than a quarter.

Cloud Service Mesh with a shared CA is what makes “trust the IP” go away. The reason cross-cloud calls today are secured by IP range is that there was no common identity across the three Kubernetes worlds. Cloud Service Mesh issues every workload a SPIFFE identity from one shared root of trust spanning all members, so a service on EKS and a service on GKE authenticate to each other by identity (spiffe://trust-domain/ns/.../sa/...) over mTLS, regardless of cloud. Setting STRICT mutual TLS in the PII-handling namespaces means a call without a valid workload certificate is refused — which is the actual, enforceable version of the control the auditor was asking about. The mesh also gives you uniform traffic management (retries, canaries, locality-aware routing) and golden-signal telemetry across the fleet, but the identity story is the load-bearing one for a regulated multi-cloud estate.

Interconnect for bandwidth and jitter; HA VPN for resilience and AWS reach — and they are not interchangeable. The on-prem path carries chatty, latency-sensitive reads of policyholder and claims data, so it wants Dedicated Interconnect: private, predictable single-digit-millisecond latency, and 10–100 Gbps. But Interconnect has weeks of lead time, no inherent encryption, and a single circuit is not redundant — so HA VPN is the day-one bring-up, the permanent encrypted backup (advertising the same prefixes at a lower BGP local-preference so failover is automatic and sub-minute), and the primary private path to AWS, where you do not have a colo cross-connect at all. Letting Cloud Router run BGP on every link is what makes all of this dynamic: routes propagate, failover is a routing decision rather than a ticket, and Network Connectivity Center turns the collection of links into a single transitive hub so on-prem and AWS reach each other through Google’s backbone instead of via a separate, hand-built mesh.

Implementation guidance

Stand up the fleet and its host project first (the management plane before the workloads).

• Choose a dedicated fleet host project (e.g. helvetia-fleet-host) — this is not where workloads run; it owns the fleet, the Config Sync/Policy Controller/mesh features, and the unified observability. Keep it small and tightly controlled.
• Register every cluster as a member. GKE-on-Google clusters register with one command/Terraform resource. EKS clusters register as attached members: deploy the Connect Agent, which dials out to Google over mTLS — confirm no inbound rule is needed on the AWS security groups, which is the property that keeps the AWS network closed. On-prem, install Google Distributed Cloud (software) on the VMware cluster and register the same way.
• Turn on fleet features as defaults, not per cluster: enable Config Sync, Policy Controller, and Cloud Service Mesh at the fleet level with fleet-default member configuration, so any future member inherits them. This is the line between “governance you apply” and “governance clusters are born with.”

GitOps repository layout (Config Sync) — get this shape right or you will fight overlays forever.

• Use one repo with a clear separation between fleet-wide policy (applied to all clusters via RootSync — Policy Controller constraints, baseline RBAC, baseline NetworkPolicies, allowed-registry constraints) and per-scope/per-cluster overlays (Kustomize) for the legitimate differences (the on-prem cluster’s load-balancer class, AWS-specific annotations).
• Give application teams their own RepoSync in their namespaces so platform owns cluster-scoped config and teams own their app config — least privilege in the GitOps layer itself.
• Keep Policy Controller constraints in the repo too, starting every new constraint in dryrun/audit mode, reviewing the violation report on the dashboard, then flipping to enforce — so you never break running workloads by enforcing a policy blind.

Identity wiring — fleet Workload Identity end to end, no static keys (this is the single biggest incident-class you design out).

• Every workload uses fleet Workload Identity: a Kubernetes service account is bound to a Google service account, and the pod gets short-lived, auto-rotated credentials — no exported service-account JSON keys anywhere, on any cluster. (This codebase has prior history with leaked database credentials committed to git; the hard rule here is that secrets live in Secret Manager and are reached via workload identity, never embedded in a manifest, image, or pipeline.)
• For the cross-cloud case, federate the fleet WI pool to AWS IAM via OIDC so a GKE (or on-prem) workload can assume an AWS IAM role keylessly to read an S3 bucket or an SQS queue the AWS side owns — and symmetrically, EKS workloads authenticate to Google APIs through the same pool. No long-lived AWS keys cross the boundary.
• Centralize human access with Cloud Identity / IAM and group-based fleet RBAC (Connect gateway lets operators kubectl into any member through Google’s auth, so on-prem and EKS access flow through the same SSO and audit trail as GKE — one access story, not three).

Networking and DNS wiring (the data plane).

• Build the Shared VPC in a host project; service projects (where GKE clusters live) attach. Enable Private Google Access on workload subnets and use Private Service Connect endpoints so clusters consume Google/partner APIs over internal IPs — nothing reaches Google over the public internet.
• Provision Dedicated Interconnect early (weeks of lead time): order two interconnects in two edge availability domains, create VLAN attachments, and attach a Cloud Router per region running BGP to the on-prem routers. Stand up HA VPN the same week as the day-one path and permanent backup, advertising the same on-prem prefixes at a lower BGP priority so the Interconnect is preferred and failover is automatic.
• For AWS, create an HA VPN gateway peered to an AWS VPN Gateway / Transit Gateway (two tunnels, BGP both sides). Then create a Network Connectivity Center hub and attach the on-prem and AWS connections as spokes, so the three environments form one transitive routing domain.
• Wire Cloud DNS: private zones for GCP services peered to the Shared VPC, outbound forwarding to the on-prem resolver for on-prem names, and inbound forwarding (a Cloud DNS inbound endpoint) so on-prem and AWS can resolve GCP/mesh names. Plan non-overlapping CIDRs across on-prem, AWS, and GCP up front — overlapping RFC 1918 ranges are the single most common, most painful hybrid mistake.

Mesh expansion across clouds. Bring the EKS and VMware members into Cloud Service Mesh with a shared trust domain so identities are comparable everywhere. Set mutual TLS to STRICT in the PII-handling namespaces (policyholder, claims, pricing) so any non-mTLS call is rejected — the enforceable form of the audit control. Use the mesh’s locality-aware routing so a service prefers an in-region/in-cloud backend and only spills cross-cloud on failure, which keeps the expensive cross-cloud hops rare.

A few IaC snippets (Terraform) capturing the wiring people most often get wrong. First, registering an EKS cluster as a fleet member and turning on fleet features as defaults — the part teams do per-cluster by hand and then drift:

# The fleet lives in the host project.
resource "google_gke_hub_fleet" "platform" {
  project      = var.fleet_host_project
  display_name = "helvetia-platform"
}

# Register the existing EKS cluster as an ATTACHED member (Connect Agent, outbound-only).
resource "google_gke_hub_membership" "eks_quoting" {
  membership_id = "eks-quoting-euw1"
  project       = var.fleet_host_project
  endpoint {
    # OIDC issuer of the EKS cluster — registration is keyless via Connect.
    edge_cluster { resource_link = null }
  }
  authority { issuer = "https://oidc.eks.eu-west-1.amazonaws.com/id/${var.eks_oidc_id}" }
}

# Turn Config Sync + Policy Controller ON for the whole fleet as a DEFAULT,
# so any future member inherits the governance posture automatically.
resource "google_gke_hub_feature" "configmanagement" {
  name     = "configmanagement"
  project  = var.fleet_host_project
  location = "global"
  fleet_default_member_config {
    configmanagement {
      config_sync {
        git {
          sync_repo   = "https://github.com/helvetia/fleet-config"
          sync_branch = "main"
          policy_dir  = "fleet"
          secret_type = "token"      # or gcpserviceaccount — never a static key in the manifest
        }
      }
      policy_controller {
        policy_controller_hub_config {
          install_spec = "INSTALL_SPEC_ENABLED"
          audit_interval_seconds = 60
        }
      }
    }
  }
}

Second, the redundant on-prem link — two Interconnect VLAN attachments plus an HA-VPN backup advertising the same routes at a lower priority, which is what makes failover automatic:

resource "google_compute_router" "onprem" {
  name    = "cr-onprem-euw6"
  region  = "europe-west6"
  network = google_compute_network.shared_vpc.id
  bgp { asn = 64512 }
}

# Interconnect VLAN attachment (one of two; the second is in a different edge domain).
resource "google_compute_interconnect_attachment" "vlan_a" {
  name                     = "vlan-zurich-a"
  region                   = "europe-west6"
  router                   = google_compute_router.onprem.id
  interconnect             = google_compute_interconnect.dedicated_a.id
  vlan_tag8021q            = 401
  candidate_subnets        = ["169.254.10.0/29"]
}

# HA VPN tunnel to on-prem: same prefixes, but advertised at a WORSE priority,
# so Interconnect wins and VPN takes over only on failure.
resource "google_compute_router_peer" "vpn_backup_peer" {
  name                      = "onprem-vpn-backup"
  router                    = google_compute_router.onprem.name
  region                    = "europe-west6"
  interface                 = google_compute_router_interface.vpn_if.name
  peer_ip_address           = var.onprem_vpn_peer_ip
  peer_asn                  = 65010
  advertised_route_priority = 200          # higher number = less preferred than Interconnect (100)
}

Third, a Policy Controller constraint (delivered through Config Sync) that bans public LoadBalancers in prod and is rolled out safely in audit mode first:

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sNoPublicServices            # from the Policy Controller library / bundle
metadata:
  name: no-public-lb-in-prod
spec:
  enforcementAction: dryrun          # start in audit; flip to "deny" after reviewing the dashboard
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Service"]
    namespaceSelector:
      matchExpressions:
        - { key: env, operator: In, values: ["prod"] }

And the mesh PeerAuthentication that turns “trust the IP” into “prove your identity” for PII namespaces:

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: strict-mtls
  namespace: pricing                 # PII-handling namespace
spec:
  mtls:
    mode: STRICT                     # any non-mTLS call is refused, on every member

Deployment and rollout. Stand up the fleet and the network spine first, then register clusters one at a time, each time confirming Config Sync reconciles green and the mesh sidecars come up before moving on. Roll out every Policy Controller constraint in dryrun first, read the violation report, remediate, then enforce — never enforce blind. Application workloads deploy through normal CI to Artifact Registry, with the promotion across clusters expressed as Git changes to the Config Sync overlays, so “ship to the next environment” is a reviewed pull request, not a kubectl against a member. The whole platform — fleet, memberships, features, VPC, Interconnect, HA VPN, NCC, DNS, IAM — is one Terraform state per environment, promoted dev → staging → prod with the same code.

Enterprise considerations

Security and Zero Trust. This architecture is a Zero Trust posture for a multi-cloud Kubernetes estate, expressed in two reinforcing layers. At the identity layer, every workload on every cluster has a SPIFFE identity from one shared CA, and STRICT mTLS in sensitive namespaces means service-to-service calls are authenticated and encrypted by identity — there is no implicit trust by network location, which is the whole point. Fleet Workload Identity (federated to AWS IAM) means no static keys anywhere, on any cloud, eliminating the most common breach vector (a leaked key) by construction — and explicitly closing the door that this codebase’s history of leaked DB credentials left open. At the policy layer, Policy Controller enforces guardrails (no public LBs, approved registries only, mandatory NetworkPolicies, pod-security baselines) at admission and continuously audits for drift, while Config Sync guarantees the declared security config is actually present on every member and reverts tampering. At the network layer, traffic rides Interconnect/HA VPN over Google’s private backbone rather than the public internet, Private Google Access / PSC keep API calls off the internet, and the GKE clusters are private (no public node IPs). Identity, policy, and network controls compound rather than substitute.

Cost optimization. The headline lever is that fleet management replaces N operational stacks with one — one monitoring stack, one policy pipeline, one identity model, one access path — so the operational cost of the thirty-first cluster is near-zero marginal, which is the real multi-cloud expense. On the network, be honest about the trade-offs: Cloud Interconnect is a fixed monthly port charge plus low egress, justified only by the bandwidth and jitter the on-prem data path actually needs — right-size the port (start at 10 Gbps, not 100) and remember HA VPN is far cheaper for links whose volume does not justify a circuit (the AWS path often starts as VPN-only). Cross-cloud egress is the silent budget killer: every GKE→AWS or GKE→on-prem call may incur egress, so the mesh’s locality-aware routing (prefer in-cloud backends, spill cross-cloud only on failure) is a cost control as much as a latency one — keep chatty service pairs co-located and only span the boundary when the data demands it. GKE Autopilot for new clusters bills per pod resource request (no paying for idle node headroom), and GKE Enterprise’s per-vCPU fleet pricing means you pay for governed capacity, not per-cluster licenses. Tag/label every cluster and workload by owning team to attribute both compute and egress.

Scalability. A single fleet governs hundreds of clusters and thousands of nodes across clouds; onboarding scales linearly because registration + fleet-default features is one operation regardless of count. Config Sync reconciles every member independently and in parallel, so the governance loop does not slow as the fleet grows. The mesh scales horizontally per cluster, and fleet scopes keep the blast radius and the cognitive load bounded as teams multiply. On the network, Cloud Router and NCC carry large route tables and many spokes, and Interconnect scales to 100 Gbps (or multiple circuits with ECMP) when the data-center data path grows.

Reliability and DR (RTO/RPO). Resilience is layered. The network path is the clearest story: two Interconnects in two edge domains survive an edge failure (99.99% SLA), and HA VPN advertising the same prefixes at lower BGP priority fails the on-prem path over automatically in under a minute with no operator action — an effective network RTO measured in seconds. Regional GKE control planes and multi-zone node pools survive a zone loss. For workload DR, the fleet makes active-active or warm-standby across clouds natural: the same Config Sync repo deploys the pricing service to both the GKE and (if desired) the on-prem cluster, and the mesh’s locality routing fails traffic over to the surviving member — so a regional GCP outage degrades to serving from on-prem/AWS rather than going dark. RTO/RPO for stateful data is then governed by the data tier riding over this fabric (Db2/Cloud SQL replication, BigQuery), not by the platform itself — model the cases explicitly: zone loss, region loss, full Interconnect loss, and a whole-cloud loss, each in a runbook.

Observability. GKE Enterprise gives one pane of glass across every member: the fleet dashboard shows cluster health, Config Sync status, and Policy Controller violations for on-prem, AWS, and GCP clusters together. Cloud Logging and Monitoring aggregate logs/metrics fleet-wide, Managed Service for Prometheus scrapes every member with no per-cluster Prometheus to operate, and Cloud Service Mesh emits golden-signal telemetry (latency, traffic, errors, saturation) and a service topology spanning clouds. The two governance vital signs to alert on are Config Sync drift/sync errors (a member that stopped reconciling is a member that is silently diverging) and Policy Controller violation count (anything in violation is a control gap to close) — plus the network’s BGP session state on every Interconnect/VPN link.

Governance. A clean resource hierarchy (org → folders for environments/business units → the fleet host project + service projects), with Organization Policy constraints enforced top-down (disable service-account key creation, restrict regions, domain-restricted sharing). The fleet is the governance object: scopes bound team access, Config Sync makes the Git repo the single auditable source of cluster state, and Policy Controller’s continuous audit is the live compliance report — so “show me the control across the estate” is a dashboard, and “prove it has been enforced over time” is the Git history plus the audit logs. Audit Logs (Admin Activity always on; Data Access on the sensitive services) flow to a logs bucket / BigQuery sink for retention and SIEM, and the Connect gateway means even on-prem and EKS kubectl access is logged through Google’s audit trail.

Reference enterprise example

Helvetia Risk ran the program over four quarters. Concrete decisions and numbers:

Estate at the start. 3 Kubernetes worlds, 22 clusters total: 6 on-prem (VMware, kubeadm), 9 on EKS (the acquired broker stack, eu-west-1), 7 on GKE (the new pricing/AI build-out, europe-west6). 3 RBAC models, 3 secret stores, 3 CI/CD pipelines, 3 monitoring stacks. The trigger was a regulator audit that asked Helvetia to demonstrate mTLS enforcement and consistent network policy for all PII-handling services across the estate — and the honest answer was “we can’t, in under two weeks, with confidence.”

What they built.

• A single GKE Enterprise fleet in a dedicated helvetia-fleet-host project. All 22 clusters registered as members: GKE natively, EKS as attached members via the Connect Agent (outbound-only, no inbound AWS rule), on-prem migrated from kubeadm to Google Distributed Cloud (software, VMware) and registered.
• Config Sync against one fleet-config Git repo (fleet-wide policy via RootSync, per-cluster overlays for the legitimate on-prem/AWS differences, team RepoSync per namespace).
• Policy Controller with the CIS + pod-security policy bundles, every constraint rolled out in dryrun first, then enforced: no public LoadBalancers in prod, Artifact-Registry-only images, mandatory NetworkPolicy per namespace.
• Cloud Service Mesh across all 22 members with a shared CA; STRICT mTLS in the policyholder, claims, and pricing namespaces.
• Fleet Workload Identity federated to AWS IAM via OIDC — the EKS quoting service assumes a GCP identity to call the GKE pricing service, and a GKE batch job assumes an AWS role to read the broker S3 bucket, both keyless.
• Network spine: Shared VPC in europe-west6; 2 × 10 Gbps Dedicated Interconnect to Zurich in two edge domains (BGP via Cloud Router); HA VPN as the on-prem backup and the primary AWS path; Network Connectivity Center hub with on-prem and AWS as spokes; Cloud DNS inbound/outbound forwarding across all three.

Decisions that paid off.

• The audit question became a dashboard. “Show me mTLS and NetworkPolicy enforced for PII services across the estate” went from a two-week, three-team scramble to a single fleet-dashboard view: Policy Controller showing zero violations on the relevant constraints across all 22 members, and the mesh showing STRICT mTLS in the PII namespaces. The auditor accepted the dashboard plus the Git history as evidence.
• Drift stopped being a standing finding. With Config Sync reconciling continuously, the recurring “a cluster fell out of compliance” finding disappeared — a hand-edit on a member was reverted within ~1 minute. The “standard” now lives in Git and is enforced, not described in a wiki.
• Cross-cloud calls got a real security model. The EKS quoting → GKE pricing → on-prem policyholder path became end-to-end mTLS by identity over the private spine, replacing “trust the IP range.” The security team signed off on the cross-cloud data flow for the first time.
• No keys to leak. Every workload moved to fleet Workload Identity; the 3 separate secret stores collapsed toward Secret Manager + WI. The class of incident that previously bit this organization (a credential committed to a repo) was designed out, not merely patched.
• On-prem failover proved boring. A planned Interconnect-maintenance test dropped the primary circuit; the on-prem data path failed over to HA VPN in ~35 seconds with no operator action and no visible impact to the pricing engine’s policyholder reads — exactly because the VPN advertised the same prefixes at a lower BGP priority.
• Onboarding collapsed. Registering the broker’s next EKS cluster and bringing it fully under policy + mesh took under a day (register → Config Sync goes green → mesh sidecars up), versus the multi-week, bespoke effort each cluster used to require.

A failure they handled gracefully. A platform engineer enforced a new “no :latest image tags” Policy Controller constraint that they had not run in dryrun first; it would have blocked a legitimate deploy on the on-prem cluster. Because the constraint was delivered through Config Sync as a reviewed pull request, the pre-merge check (which lints constraints and runs them against a sample) flagged the impending violations, and the change was held and re-rolled in dryrun mode — the violation report showed the affected workloads, those were fixed, and only then was it enforced. The guardrail process caught the guardrail mistake; nothing in production broke.

Outcome. Steady-state GCP spend ~₹19 lakh/month (fleet/GKE Enterprise + Interconnect + the GCP workloads), with the operational headcount to run the estate flat despite the cluster count, because there is now one stack to operate instead of three. The audit finding closed on the strength of the dashboard. Cross-cloud service calls are identity-secured over a private, redundant spine; on-prem failover is automatic and proven; there are no static keys to leak; and onboarding a new cluster — anywhere — is a day, not a quarter. Helvetia now treats “multi-cloud” as an operating model they govern, rather than an accident they survive.

When to use it

Use this architecture when:

• You have Kubernetes in more than one place — multiple clouds, or cloud plus on-prem — and the operational and governance cost of running them as separate islands is the real pain (this is the clearest signal you need a fleet).
• You face a “prove this control is enforced everywhere” mandate (regulated industry, audit, security baseline) across a heterogeneous estate — fleet-wide Config Sync + Policy Controller is purpose-built for exactly this.
• You have regulated or gravity-bound data on-prem that new cloud workloads must sit next to with private, high-bandwidth, low-jitter connectivity — Interconnect + HA VPN is the spine.
• You acquired or inherited a born-in-another-cloud stack (EKS/AKS) you cannot or should not re-platform, but must bring under one identity, policy, and network model.
• You want consistent, identity-based service-to-service security (mTLS) across clouds, replacing IP-range trust — Cloud Service Mesh with a shared CA delivers it.

Be honest about the trade-offs:

• It is a platform, not a weekend project. You are running a fleet, GitOps reconciliation, a multi-cluster mesh, federated identity, and a hybrid network. Without the GitOps and policy discipline, you have built a more complicated way to drift. Commit to Config Sync, dryrun-first policy rollout, and the observability above, or do not adopt it.
• Cross-cloud is not free, in money or latency. Every boundary-crossing call can incur egress and tens of milliseconds. If your services are chatty and co-located logically but spread physically, the bill and the p99 will tell on you — design for locality and span the boundary only where the data requires it.
• It is more than a small, single-cloud problem needs. One cloud, a few clusters, one team, no on-prem data gravity? Plain GKE with Config Sync on a single project (even without the full multi-cloud fleet) is the right answer; adopt the cross-cloud spine when a second environment that must be governed together actually appears.
• Google Distributed Cloud on-prem is real infrastructure. Running GKE on your own VMware/bare-metal is operationally heavier than consuming managed GKE — staff for it, or keep on-prem minimal and exit-oriented.

Anti-patterns to avoid (each is a real incident waiting to happen):

• Operating clusters instead of the fleet. Configuring Config Sync, policy, and mesh per cluster by hand is how you recreate the very drift you adopted the fleet to kill. Turn them on as fleet defaults so members are born compliant.
• Enforcing policy blind. Flipping a Policy Controller constraint straight to deny without a dryrun pass will eventually block a legitimate deploy and look like an outage. Audit, review the dashboard, then enforce — always.
• Overlapping CIDRs across environments. The most common, most painful hybrid mistake. Plan non-overlapping RFC 1918 across on-prem, AWS, and GCP before the first link, or you will be renumbering production later.
• A single Interconnect called “redundant.” One circuit is a single point of failure with weeks of repair lead time. The minimum is two Interconnects in two edge domains plus HA VPN backup; anything less is not a 99.99% path.
• mTLS without a shared trust domain. Turning on the mesh per cluster with different CAs gives you encryption but not cross-cloud identity — the auditor’s question stays unanswerable. One shared root of trust across all members is the whole point.
• Exported service-account keys (or any static cross-cloud keys). The most common breach vector. Use fleet Workload Identity federated to AWS IAM; set the org policy that disables key creation. No exceptions.
• GitOps drift you don’t watch. Config Sync silently failing to reconcile a member means that member is diverging unseen. Alert on sync errors and drift, or the closed loop is quietly open.

Alternatives worth weighing: for a single cloud, plain GKE + Config Sync + Policy Controller on one project gives you the GitOps/policy benefits without the multi-cloud machinery — graduate to the full fleet when a second governed environment appears. For multi-cloud governance without Google’s fleet, a self-managed Argo CD / Flux + OPA Gatekeeper + Istio (multi-primary) stack across clusters is the DIY equivalent — more control and portability, but you now operate the reconciliation, the policy engine, the mesh CA, and the multi-cluster wiring that GKE Enterprise manages for you (and you lose the single fleet dashboard and Connect gateway). For connectivity specifically, if you have no on-prem at all and only span clouds, you can drop Interconnect and run HA VPN + Network Connectivity Center alone; conversely, very high-bandwidth data-center estates may run Cross-Cloud Interconnect to land a private circuit directly between Google and AWS rather than over VPN. And for the on-prem distribution, Google Distributed Cloud (software) assumes you run the cluster; the GDC connected/air-gapped appliances suit edge or sovereignty cases where Google ships the hardware. The shape — one fleet for management, a private redundant spine for the data path, shared identity and policy across every member — stays the same across all of these. Only the connectivity primitive and the on-prem footprint change.

Going deeper

The core teaching above is the what and the why. This section is the how it actually works underneath — the internals, the surface-area caveats, and the two cross-cloud capabilities (identity federation and BigQuery Omni) that the reference architecture leans on but that deserve their own dissection. Read it when you are past “I understand the diagram” and into “I have to build and defend this.”

Registration internals: three paths, one membership object. Every member — GKE, EKS, VMware — ends up as a gkehub membership resource, but the path there differs, and getting the path right matters:

• GKE on Google Cloud is enrolled directly (fleet membership is created for you when you attach the cluster to the fleet, or explicitly):

  gcloud container fleet memberships register pricing-euw6 \
    --gke-cluster=europe-west6/pricing \
    --enable-workload-identity \
    --project=helvetia-fleet-host
  

• EKS / AKS come in as attached clusters — a distinct, current surface that creates both the membership and an attached-cluster resource and installs the Connect Agent:

  gcloud container attached clusters register eks-quoting-euw1 \
    --location=europe-west6 \
    --platform-version=1.29.0-gke.1 \
    --distribution=eks \
    --issuer-url="https://oidc.eks.eu-west-1.amazonaws.com/id/EKS_OIDC_ID" \
    --project=helvetia-fleet-host
  

  The Connect Agent’s defining property is that it dials outbound to gkeconnect.googleapis.com over mTLS and holds the connection open; the fleet reaches the cluster back down that tunnel. There is no inbound port on the AWS security group — which is exactly why security teams accept it. Confirm the agent is healthy with gcloud container fleet memberships list and look for Last connection time moving.
• On-prem is Google Distributed Cloud (software) on VMware/bare metal; it registers through Connect during the cluster install (admin cluster + user clusters), not after the fact.

Once any of the three is a membership, fleet-default member configuration is what makes governance automatic. Enable Config Sync + Policy Controller once at the fleet level and every future membership inherits it — this is the mechanism behind “born compliant”:

# apply-spec.yaml — the ConfigManagement spec applied per membership (or as a fleet default)
applySpecVersion: 1
spec:
  configSync:
    enabled: true
    sourceFormat: unstructured
    git:
      syncRepo: https://github.com/helvetia/fleet-config
      syncBranch: main
      policyDir: fleet
      secretType: token
  policyController:
    enabled: true
    auditIntervalSeconds: 60
    referentialRulesEnabled: true    # needed for cross-object rules (e.g. "every ns has a NetworkPolicy")

gcloud container fleet config-management enable --project=helvetia-fleet-host
gcloud container fleet config-management apply \
  --membership=eks-quoting-euw1 --config=apply-spec.yaml --project=helvetia-fleet-host

Config Sync internals: RootSync, RepoSync, and how drift actually gets reverted. Config Sync is two controllers and a set of reconciler pods, not a cron job:

• A RootSync (namespace config-management-system, one per cluster, platform-owned) syncs cluster-scoped and any-namespace config — RBAC, NetworkPolicies, Policy Controller constraints, namespaces themselves.
• A RepoSync (in a team’s namespace, team-owned) syncs only that namespace’s objects. This is least privilege inside GitOps: a team cannot use its RepoSync to grant itself cluster-admin, because the reconciler’s own RBAC is namespace-bound.

Each sync object runs a dedicated reconciler Deployment that: (1) pulls the Git commit, (2) renders it (Config Sync hydrates Kustomize/Helm for you when the repo uses them), (3) computes the diff against live state, and (4) applies. Crucially it keeps applying — the reconciler holds the declared set and reverts any live object that drifts from it, which is why a kubectl edit on a managed resource is undone within a sync interval. Objects it manages carry the configsync.gke.io/ annotations; objects you create outside the repo are left alone unless a constraint forbids them. The failure mode to alert on is a reconciler that is erroring or stalled (bad render, unreachable Git, conflicting manager) — a member whose reconciler is red is a member silently diverging.

# A RootSync object (what the fleet-default config materializes on each member)
apiVersion: configsync.gke.io/v1beta1
kind: RootSync
metadata:
  name: root-sync
  namespace: config-management-system
spec:
  sourceFormat: unstructured
  git:
    repo: https://github.com/helvetia/fleet-config
    branch: main
    dir: fleet
    auth: token
    secretRef:
      name: git-creds

Policy Controller internals: admission + audit, and the dryrun ladder. Policy Controller is managed Gatekeeper (OPA). A ConstraintTemplate defines a reusable rule (Rego under the hood); a Constraint instantiates it with parameters and a match selector. Two things run continuously: an admission webhook (blocks non-compliant creates/updates in real time) and an audit controller (every auditIntervalSeconds, re-scans existing objects and writes violations to the constraint’s status and the fleet dashboard). The enforcementAction ladder is the safe-rollout discipline: dryrun (audit only — count violations, break nothing) → warn (admit but return a warning) → deny (block). Never skip to deny on a live cluster. Referential constraints (like “every namespace must have a NetworkPolicy”) need referentialRulesEnabled because they read other objects from OPA’s cache — a subtlety that trips people who copy a constraint and wonder why it never fires.

Workload Identity Federation across clouds — the token exchange, keyless, in both directions. This is the single biggest incident-class the architecture designs out, and it is worth understanding the actual flow rather than treating it as magic.

• GKE (or on-prem) workload → AWS. AWS trusts the fleet/cluster OIDC issuer as an IAM identity provider. You create an IAM role whose trust policy allows sts:AssumeRoleWithWebIdentity from that provider, scoped by the Kubernetes ServiceAccount subject. The pod gets a projected, short-lived OIDC token for its KSA; it calls AWS STS with that token and gets back short-lived AWS credentials. No AWS access key ever exists.

  {
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Allow",
      "Principal": { "Federated": "arn:aws:iam::AWS_ACCOUNT_ID:oidc-provider/PROJECT_ID.svc.id.goog" },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": { "StringEquals": {
        "PROJECT_ID.svc.id.goog:sub": "system:serviceaccount:pricing:s3-reader"
      }}
    }]
  }
  

• EKS workload → GCP. Symmetric, using GCP Workload Identity Federation: a workload identity pool with an AWS/OIDC provider maps the external token to a principal that can impersonate (or, with direct resource access, act as) a Google service account. The EKS pod presents its token, Google’s STS returns a short-lived GCP access token.

The trust-domain string PROJECT_ID.svc.id.goog is the fleet Workload Identity pool; every member shares it, which is what makes “a workload on any cluster authenticates as itself” true across clouds. The org-policy backstop is iam.disableServiceAccountKeyCreation — turn it on so nobody can create the static key you designed out.

Cloud Service Mesh across clouds: multi-primary, east-west gateways, and the shared CA. A mesh spanning clouds is multi-network by definition (GCP VPC, AWS VPC, and on-prem are different networks). The pieces that make it work:

• Shared trust domain / one CA. Every member’s sidecars get SPIFFE certs from the same root of trust (Google’s mesh CA, or Certificate Authority Service). This is non-negotiable — different CAs give you encryption but incomparable identities, and the auditor’s cross-cloud question stays unanswerable.
• Multi-primary control planes: each cluster runs its own managed control plane but they discover each other’s endpoints, so a service resolves to backends on any member. (Primary-remote is the alternative where remote clusters borrow a control plane; multi-primary is the resilient default.)
• East-west gateways. Because the networks are separate, cross-network traffic egresses through an east-west gateway on each side (mTLS passthrough, SNI-routed) rather than assuming flat pod-to-pod reachability. This is the mesh-level counterpart to the NCC/VPN data path.
• Locality-aware routing. The mesh prefers same-zone → same-region → same-cluster backends and only spills cross-cloud on failure, which is simultaneously a latency control and (because cross-cloud egress is billed) a cost control.

gcloud container fleet mesh enable --project=helvetia-fleet-host
gcloud container fleet mesh update --management automatic \
  --memberships pricing-euw6,eks-quoting-euw1 --project=helvetia-fleet-host

BigQuery Omni + BigLake: cross-cloud analytics without moving the data. The architecture’s data answer to multi-cloud is not “copy everything into BigQuery” — that fights data gravity, residency, and egress. BigQuery Omni runs BigQuery’s compute inside AWS and Azure regions (on Google-managed Anthos clusters), so the query executes next to the data in S3/Blob and only results cross the boundary. You reach it through a BigLake table: an external table defined over object storage via a connection that holds an IAM role BigQuery assumes (keyless from your side). Omni locations look like aws-us-east-1 or azure-eastus2.

# A connection BigQuery uses to assume an AWS IAM role and read S3 (no keys handed to BigQuery).
bq mk --connection --connection_type='AWS' \
  --iam_role_id='arn:aws:iam::AWS_ACCOUNT_ID:role/bq-omni-s3-reader' \
  --location=aws-us-east-1 \
  broker_s3_conn

-- A BigLake table over the broker's S3 data, queried IN AWS by Omni compute.
CREATE EXTERNAL TABLE broker_ds.quotes_s3
WITH CONNECTION `aws-us-east-1.broker_s3_conn`
OPTIONS (format = 'PARQUET', uris = ['s3://helvetia-broker/quotes/*']);

-- Analyze in place; only the aggregate crosses the boundary.
SELECT product, COUNT(*) AS n, AVG(premium) AS avg_premium
FROM broker_ds.quotes_s3
WHERE quote_date = CURRENT_DATE()
GROUP BY product;

Three patterns build on this: a cross-cloud transfer (CREATE TABLE gcp_ds.daily AS SELECT ... FROM broker_ds.quotes_s3) lands a summary in GCP BigQuery for the AI layer; a cross-cloud join lets one query join a GCP-region BigQuery table with an Omni (AWS) table, BigQuery moving the smaller side automatically; a cross-cloud materialized view keeps a GCP-side rollup fresh over the remote data. It is the same SQL dialect and the same IAM/governance model on both clouds — which is the analytics expression of the whole thesis: consistency across clouds without a forklift. (Check current GA-vs-preview status per feature and region before you commit a design to it.)

Cross-Cloud Interconnect vs HA VPN — when the AWS spine outgrows the tunnel. HA VPN to AWS is cheap, encrypted, quick to stand up, and caps out around a few Gbps per tunnel pair — perfect for a day-one or modest-volume AWS path. When the cross-cloud data path grows (Omni transfers, replication, chatty service traffic), Cross-Cloud Interconnect lets Google provision a dedicated 10/100 Gbps physical circuit directly between Google’s network and AWS/Azure/Oracle at a colocation facility — the Dedicated-Interconnect experience, but the far end is another cloud instead of your data center, with MACsec available for link-layer encryption. The migration is undramatic because Cloud Router runs BGP on both: you bring up the circuit, advertise the same prefixes, prefer it over the VPN by local-preference, and retire the tunnels — the exact failover discipline used for on-prem, reused for the cloud-to-cloud spine.

Scale, quotas, and limits to design around. A single fleet governs into the hundreds of clusters; the constraints you actually plan for are lower down. Cloud Router learns a bounded number of routes per BGP session (plan prefix aggregation rather than advertising thousands of /32s), NCC hubs cap the number of spokes, Interconnect caps VLAN attachments per interconnect, and Config Sync scales per-cluster but a single monolithic repo with thousands of objects slows rendering — shard with RepoSync per team. None of these bite a 3-cluster deployment; all of them bite a 300-cluster one, so read the current quota pages before the estate is large, not after.

Naming and API caveats (the thing that dates every Anthos tutorial). The rebrand matters when you read docs and Terraform: Anthos → GKE Enterprise (the fleet-centric packaging), Anthos Config Management → Config Sync + Policy Controller, Anthos Service Mesh → Cloud Service Mesh, Anthos clusters on VMware/bare metal → Google Distributed Cloud (software), and gcloud container hub → gcloud container fleet. The API is still gkehub.googleapis.com and Terraform resources are still google_gke_hub_* — so old and new names coexist in the surface. When a tutorial says “Anthos,” read “GKE Enterprise / the fleet”; the concepts are identical, only the labels moved.

Practice challenges

Work these in order — they climb from “register one cluster” to “query another cloud’s data in place.” Each is concrete; commands and manifests use placeholders (PROJECT_ID, AWS_ACCOUNT_ID, region names) you would substitute. There is no live cluster here, so treat the outputs as representative and the manifests as schema-correct starting points.

Challenge 1 (Beginner) — Register a GKE cluster into a fleet and confirm the membership. You have a GKE cluster pricing in europe-west6 and an empty fleet host project. Register it as a member with Workload Identity enabled, then list the fleet’s members.

<details> <summary>Solution</summary>

gcloud container fleet memberships register pricing-euw6 \
  --gke-cluster=europe-west6/pricing \
  --enable-workload-identity \
  --project=PROJECT_ID

gcloud container fleet memberships list --project=PROJECT_ID
# NAME          UNIQUE_ID   LOCATION       (representative)
# pricing-euw6  a1b2c3...   europe-west6

Why: membership is the entry point — nothing (Config Sync, mesh, policy, Connect gateway) applies to a cluster until it is a fleet member, and --enable-workload-identity is what lets its workloads later authenticate keylessly. </details>

Challenge 2 (Beginner → Intermediate) — Turn on Config Sync as a fleet default so new members inherit it. Point every current and future member at one Git repo, so a cluster registered next week is governed the moment it joins.

<details> <summary>Solution</summary>

Enable the feature, then apply the ConfigManagement spec (as a fleet default, or per membership):

# apply-spec.yaml
applySpecVersion: 1
spec:
  configSync:
    enabled: true
    sourceFormat: unstructured
    git:
      syncRepo: https://github.com/helvetia/fleet-config
      syncBranch: main
      policyDir: fleet
      secretType: token

gcloud container fleet config-management enable --project=PROJECT_ID
gcloud container fleet config-management apply \
  --membership=pricing-euw6 --config=apply-spec.yaml --project=PROJECT_ID
gcloud container fleet config-management status --project=PROJECT_ID   # look for SYNCED

Why: enabling it as a fleet default is the difference between “governance you remember to apply” and “governance clusters are born with” — the linchpin of the whole model. </details>

Challenge 3 (Intermediate) — Author a Policy Controller constraint, roll it out in dryrun, then enforce. Require that prod Pods pull images only from your Artifact Registry, without breaking anything on day one.

<details> <summary>Solution</summary>

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sAllowedRepos
metadata:
  name: only-artifact-registry
spec:
  enforcementAction: dryrun          # audit first; flip to "deny" after the dashboard is clean
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
    namespaceSelector:
      matchExpressions:
        - { key: env, operator: In, values: ["prod"] }
  parameters:
    repos:
      - "europe-docker.pkg.dev/PROJECT_ID/"

Commit it to the fleet/ dir, let Config Sync apply it, read the violation count on the fleet dashboard, remediate the offenders, then change enforcementAction to deny in a second commit. Why: dryrun first is the non-negotiable safety rung — enforcing a constraint blind is how a guardrail becomes an outage. K8sAllowedRepos is a real Policy Controller library template. </details>

Challenge 4 (Intermediate → Advanced) — Enforce STRICT mTLS in the pricing namespace and prove a plaintext call is refused. Make the auditor’s control real: any non-mTLS call into pricing must be rejected, on every member.

<details> <summary>Solution</summary>

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: strict-mtls
  namespace: pricing
spec:
  mtls:
    mode: STRICT

Then verify from a pod without a sidecar (or with mTLS disabled) that the call fails, and from a mesh member that it succeeds:

# From a non-mesh pod — expect connection reset / 000, NOT a 200:
kubectl exec -n test curl-nomesh -- curl -sS -o /dev/null -w "%{http_code}" http://pricing.pricing.svc:8080/health
# 000  (representative — plaintext refused)

Why: STRICT turns “we trust the IP range” into “prove your workload identity or be refused” — the enforceable form of the control, and only meaningful because every member shares one trust domain (Challenge 5’s federation and this share the same identity backbone). </details>

Challenge 5 (Advanced) — Let a GKE workload assume an AWS IAM role keylessly to read S3. The GKE batch job s3-reader in namespace pricing must read s3://helvetia-broker/… with no static AWS keys anywhere.

<details> <summary>Solution</summary>

1. In AWS, register the fleet OIDC issuer as an IAM OIDC identity provider.
2. Create an IAM role whose trust policy allows AssumeRoleWithWebIdentity from that provider, scoped to the KSA subject:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "Federated": "arn:aws:iam::AWS_ACCOUNT_ID:oidc-provider/PROJECT_ID.svc.id.goog" },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": { "StringEquals": {
      "PROJECT_ID.svc.id.goog:sub": "system:serviceaccount:pricing:s3-reader"
    }}
  }]
}

1. The pod uses its projected KSA token to call AWS STS and receives short-lived credentials — nothing to store, nothing to leak. Backstop with the org policy iam.disableServiceAccountKeyCreation. Why: this deletes the single most common breach vector (a leaked static key) by construction — the exact door this codebase’s leaked-credential history left open. </details>

Challenge 6 (Advanced) — Query an S3 dataset from BigQuery in place, then bring a daily aggregate home to GCP. The broker’s quote data lives in S3 in AWS and cannot be bulk-copied to GCP (residency). Analyze it where it sits, then land only a daily summary in GCP BigQuery for the AI layer.

<details> <summary>Solution</summary>

# Connection BigQuery uses to assume an AWS role and read S3 (keyless from BigQuery's side).
bq mk --connection --connection_type='AWS' \
  --iam_role_id='arn:aws:iam::AWS_ACCOUNT_ID:role/bq-omni-s3-reader' \
  --location=aws-us-east-1 broker_s3_conn

-- BigLake external table; BigQuery Omni compute runs the query IN AWS, next to the data.
CREATE EXTERNAL TABLE broker_ds.quotes_s3
WITH CONNECTION `aws-us-east-1.broker_s3_conn`
OPTIONS (format = 'PARQUET', uris = ['s3://helvetia-broker/quotes/*']);

-- Cross-cloud transfer: only the aggregate (not the raw rows) crosses to GCP.
CREATE OR REPLACE TABLE gcp_ds.daily_quote_summary AS
SELECT product, COUNT(*) AS n, AVG(premium) AS avg_premium
FROM broker_ds.quotes_s3
WHERE quote_date = CURRENT_DATE()
GROUP BY product;

Why: data gravity and residency mean moving the data is the wrong default; Omni moves the compute to the data and returns only results — the same SQL and governance on both clouds, no forklift, minimal egress. </details>

Common beginner mistakes

These are conceptual traps — the wrong mental model, and the right one to replace it. They are distinct from the operational anti-patterns in “When to use it”; those tell you what not to do, these tell you what to stop believing.

“Multi-cloud means my app automatically runs everywhere / is portable.” No. Kubernetes gives you a common runtime, not free portability — every cloud’s storage classes, load balancers, IAM, and managed services differ, and running the same workload live on three clouds multiplies cost and complexity, it does not divide them. Right model: multi-cloud is a governance and connectivity problem, not a portability button. The fleet gives you consistent management across clouds; it does not make your app cloud-agnostic for free.

“A fleet is just a folder or a group of clusters.” A folder groups projects for IAM; a fleet is a control object that carries shared identity (one WI pool), shared policy (Config Sync + Policy Controller), shared mesh trust (one CA), and shared observability. Registering a cluster does not merely list it — it enrolls it into a shared control loop. Right model: the fleet is the unit of management; the cluster stops being the thing you operate.

“The fleet manages my network — once clusters are members they can talk.” No. Management plane ≠ data plane. The Connect Agent is outbound-only control traffic to Google; it does nothing for pod-to-pod reachability. Two members can be fully governed and still be unable to send a single packet to each other until Interconnect/HA VPN/NCC and DNS exist. Right model: govern with the fleet, reach with the network spine — separate designs, both required.

“Turning on the service mesh gives me cross-cloud mTLS identity.” Only if every cluster shares one trust domain / one CA. Enable the mesh per cluster with different CAs and you get encryption but incomparable identities — a cert from cluster A means nothing to cluster B, and the auditor’s “prove mTLS across the estate” stays unanswerable. Right model: the shared root of trust is the point of the mesh here, not the sidecars.

“Config Sync is a one-time kubectl apply from Git.” No — it is a continuous reconcile loop. It keeps re-asserting the declared state and reverts drift, so a hand-edit on a managed object is undone within a sync interval. You do not fix a cluster by editing it; you fix it by changing Git. Right model: Git is the desired state, the cluster is a projection of it, and drift is a bug the loop closes automatically.

“Workload Identity is just a nicer place to keep a key.” There is no key. The pod presents a short-lived, projected OIDC token and exchanges it for short-lived cloud credentials via STS; nothing long-lived is stored, so there is nothing to leak, rotate, or commit to a repo by accident. Right model: federation replaces “hold a secret” with “prove an identity, briefly” — the static key is gone, not hidden.

“Cross-cloud calls are basically the same as in-cloud calls.” They are not — every boundary crossing can incur egress charges and tens of milliseconds of latency, and a chatty service pair split across clouds will punish both your bill and your p99. Right model: design for locality (mesh locality-aware routing, co-locate chatty pairs) and span the boundary only where the data genuinely requires it; for analytics, move the compute to the data (BigQuery Omni), not the data to the compute.

Glossary

• Fleet — a set of Kubernetes clusters treated as one governable object; the unit of identity, policy, mesh trust, and observability. Scoped to one GCP project (the fleet host project).
• Fleet host project — the dedicated GCP project that owns the fleet and its features (Config Sync, Policy Controller, mesh). Not where workloads run.
• Member / membership — a cluster registered into a fleet; represented by a gkehub membership resource. Registration is the prerequisite for every fleet feature.
• GKE Enterprise (Anthos) — Google’s fleet-based platform for governing Kubernetes across GCP, other clouds, and on-prem. “Anthos” is the former (still-common) name.
• Connect Agent — a small in-cluster deployment that dials outbound over mTLS to Google, making a non-GCP cluster manageable from the fleet with no inbound port opened.
• Connect gateway — lets operators kubectl into any member through Google IAM, so on-prem/EKS access uses the same SSO and audit trail as GKE.
• Attached cluster — the registration path for an existing EKS/AKS cluster; creates the membership + an attached-cluster resource and installs the Connect Agent.
• Google Distributed Cloud (GDC), software — GKE’s on-prem distribution (VMware or bare metal), the successor name to “Anthos clusters on VMware/bare metal.”
• Config Sync — the GitOps engine that continuously reconciles each member against a Git repo and reverts drift. Runs reconciler pods per sync object.
• RootSync / RepoSync — cluster-scoped, platform-owned sync (RootSync) vs namespace-scoped, team-owned sync (RepoSync); least privilege inside GitOps.
• Policy Controller — managed OPA Gatekeeper; enforces guardrails at admission and continuously audits existing resources for violations.
• ConstraintTemplate / Constraint — a reusable policy rule (template, Rego under the hood) and its parameterized instance with a match selector.
• enforcementAction (dryrun / warn / deny) — the rollout ladder for a constraint: audit-only → admit-with-warning → block. Always start at dryrun.
• Referential constraint — a policy that reads other objects (e.g. “every namespace has a NetworkPolicy”); requires referentialRulesEnabled.
• Cloud Service Mesh — Google’s managed Istio; gives every workload an identity and mTLS, plus traffic management and telemetry, across all members.
• mTLS (mutual TLS) — both sides of a connection present certificates; the basis for authenticating service-to-service calls by identity rather than IP.
• SPIFFE / SPIFFE ID — a standard workload identity format (spiffe://trust-domain/ns/.../sa/...) the mesh issues to every workload.
• Trust domain / mesh CA — the shared root of trust that signs all workload certs. One CA across all members is what makes identities comparable across clouds.
• PeerAuthentication / STRICT — the mesh object that sets mTLS mode for a namespace; STRICT refuses any non-mTLS call.
• Fleet Workload Identity — one identity pool (PROJECT_ID.svc.id.goog) shared by all members, so a workload on any cluster authenticates as itself with no static key.
• Workload Identity Federation (WIF) — keyless cross-cloud auth via OIDC token exchange (STS): a workload proves an identity and receives short-lived credentials for the other cloud.
• OIDC (OpenID Connect) — the token/identity-provider standard underneath federation; each cluster/fleet exposes an OIDC issuer other clouds can trust.
• Fleet scope / fleet namespace — a bounded subset of the fleet (members + namespaces) governed and made mesh-visible together; “namespace sameness” means a same-named namespace across clusters is treated as one.
• Management plane vs data plane — governance/observability of clusters (the fleet, outbound-only) vs actual packet reachability between them (the network spine). Kept separate by design.
• Shared VPC — a VPC in a host project that service projects attach to; the GCP-side network backbone all workloads and links join.
• Private Google Access — lets nodes reach Google APIs over internal IPs, no public internet.
• Private Service Connect (PSC) — private endpoints for consuming Google/partner services over internal IPs.
• Cloud Interconnect (Dedicated / Partner) — a private, high-bandwidth, low-jitter physical link to your data center. Two circuits in two edge domains for 99.99%.
• VLAN attachment — the logical attachment that binds an Interconnect to a Cloud Router / VPC.
• HA VPN — a managed, two-tunnel, 99.99%-SLA encrypted VPN; the day-one path, the permanent on-prem backup, and the primary path to AWS.
• Cloud Router / BGP / ASN — the component that exchanges routes dynamically over a link; BGP is the routing protocol, the ASN is each side’s autonomous-system number.
• Advertised route priority / local preference — BGP knobs that make one path preferred; advertising the same prefixes at a worse priority on HA VPN is what makes Interconnect→VPN failover automatic.
• Network Connectivity Center (NCC) — the hub that ties on-prem, AWS, and GCP links (as spokes) into one transitive routing domain over Google’s backbone.
• Cross-Cloud Interconnect — a dedicated 10/100 Gbps physical circuit Google provisions directly between its network and AWS/Azure/Oracle; the high-bandwidth successor to a VPN cloud-to-cloud path (MACsec available).
• Cloud DNS (forwarding / peering) — cross-boundary name resolution: private zones, outbound forwarding to an on-prem resolver, inbound forwarding so on-prem/AWS can resolve GCP names.
• BigQuery Omni — BigQuery compute running inside AWS/Azure regions so queries execute next to S3/Blob data; only results cross the boundary.
• BigLake table — an external table over object storage (S3/Blob/GCS) reached through a connection; how Omni exposes remote data as a queryable table.
• Cross-cloud transfer / join / materialized view — Omni patterns: land an aggregate in GCP BigQuery; join a GCP table with a remote one in a single query; keep a GCP-side rollup fresh over remote data.
• Egress — the charge (and latency) for data leaving a cloud/region; the silent budget killer of chatty cross-cloud designs.
• Data gravity — the tendency of large or regulated datasets to be expensive or impossible to move, so compute should come to the data.
• Autopilot — the GKE mode that bills per pod resource request with no node management, so you do not pay for idle node headroom.
• Artifact Registry — Google’s container/artifact registry; the approved image source constraints point at.
• Secret Manager — the managed secret store; secrets live here and are reached via Workload Identity, never embedded in a manifest, image, or pipeline.
• Organization Policy — org-wide guardrails enforced top-down (e.g. iam.disableServiceAccountKeyCreation, region restrictions, domain-restricted sharing).
• CIDR / RFC 1918 / non-overlapping — private IP ranges must be planned to not overlap across on-prem, AWS, and GCP before the first link, or you renumber production later.
• Locality-aware routing — the mesh’s preference for same-zone/region/cloud backends, spilling cross-cloud only on failure; a latency and cost control.
• Lock-in — dependence on one vendor’s proprietary surface; the thing multi-cloud aims to reduce, at the cost of the complexity this architecture manages.