Migrating a Monolith to Microservices on GKE: A Pragmatic Path

In a nutshell

You almost never rewrite a large, business-critical system from scratch. A “big-bang rewrite” — freeze the old thing, build a new thing beside it, and swap them on one big launch night — is the single most reliable way to burn two years and ship nothing. The system you are replacing has a decade of hard-won edge cases baked into it, and the day you flip the switch is the day you discover the ones nobody wrote down.

The pragmatic alternative has a memorable name: the strangler fig, after the vine that grows around a host tree, branch by branch, until one day the original tree has rotted away inside it and the vine stands on its own. You put a thin routing layer (a facade) in front of the old system, then carve off one capability at a time into its own small service running on Kubernetes — here, Google Kubernetes Engine (GKE) — and flip a route so that slice of traffic goes to the new service while everything else still hits the old system. Each carved-off service ships on its own, delivers value the day it lands, and can be rolled back by flipping the route back. The old system keeps earning money the entire time. You are never more than one route-flip away from where you started.

The mental model in one line: grow the new around the old, one revertible route at a time, until the old is small enough to retire. The three prizes you are buying are independent deployability (ship one service without re-testing the rest), fault isolation (a bug in one service does not take the others down), and independent scale (scale the busy service without scaling the quiet ones).

When is this the wrong call? When the monolith is small, well-factored, and not actually under scaling or fault-isolation pressure — then microservices are a tax you are paying for benefits you do not need, and a tidy modular monolith (clean module boundaries inside one deployable) is a perfectly good destination. And when your org cannot re-form teams to own the new services, the technology split will fight the org structure and lose — microservices are an organizational choice at least as much as a technical one. The destination this lesson builds toward — many independently-owned services on GKE — is laid out end to end in the GKE microservices enterprise architecture lesson; here we focus on the far harder question of how you get there from a running monolith without ever stopping the business.

Level: Intermediate (with a beginner on-ramp) · Time: ~38 min · You’ll learn: the strangler-fig pattern and its anti-corruption layer, how to find seams by domain, database-per-service and the data-consistency traps (sagas, the outbox), traffic shifting across the seam, and — just as important — when not to do any of it.

A national logistics and parcel carrier moves 11 million packages a day through a single Java monolith that has accreted features since 2011 — rating, label generation, manifesting, track-and-trace, customs, billing, and the driver dispatch API all in one 2.3-million-line WAR deployed to a fleet of VMs. The CTO’s ultimatum is concrete and arrives the week after a peak-season outage: a rating-engine bug in November took the entire platform down for 47 minutes during the highest-volume hour of the year, because rating, tracking, and dispatch share one process and one heap. Every team waits on a six-week release train. A change to the customs module requires regression-testing checkout. The business wants three things that the monolith physically cannot give them: independent deployability (ship tracking without re-certifying billing), fault isolation (a rating bug must not take down dispatch), and elastic scale (track-and-trace gets 40× its baseline traffic on the morning after a holiday, while billing is flat). The instinct in the room is “rewrite it on Kubernetes.” That instinct, executed literally, is how you get a two-year project that ships nothing and a big-bang cutover that fails on a Saturday night. This article is the pragmatic alternative: a strangler-fig decomposition onto Google Kubernetes Engine (GKE) that keeps the monolith earning revenue every day while microservices grow up around it and eventually replace it.

Why not a big-bang rewrite

The naive plan fails predictably, and naming the failure modes matters because someone will champion the rewrite in the first meeting.

A big-bang rewrite asks you to freeze the monolith’s features for the duration (the business never agrees), reproduce a decade of undocumented edge cases from scratch (you will miss the ones that matter), and cut over everything on one date (the riskiest possible deployment). The reason it is a rewrite is the reason it fails: there is no incremental value and no incremental risk reduction — you carry maximum risk until the last day.

A lift-and-shift of the WAR into a single container on GKE is the opposite mistake. It feels like progress and delivers almost none of the goals: it is still one process, so a rating bug still shares a heap with dispatch (no fault isolation), it still deploys as one unit (no independent deployability), and it still scales as one unit (you scale billing to scale tracking). You have paid for Kubernetes and bought a more expensive monolith.

The strangler fig — named for the vine that grows around a host tree until the original rots away inside it — threads the needle. You put a facade (a routing layer) in front of the monolith, then extract one capability at a time into a microservice on GKE, flipping a route so the new service serves that slice of traffic while everything else still hits the monolith. Each extraction ships independently, delivers value the day it lands, and reduces risk because the blast radius of the next extraction is one capability, not the platform. The monolith shrinks until what remains is small enough to retire or leave as a legacy core. You are never more than one route-flip from rolling back.

Architecture overview

The architecture has three layers that you should hold separately in your head: the edge and facade (where traffic is routed between old and new), the GKE service mesh (where extracted microservices run), and the data and async fabric (how services own their data and talk without coupling). The defining property of the whole topology is the facade: nothing the client sees changes during the migration. The carrier’s driver apps, the public tracking page, and the EDI integrations all keep calling the same hostnames; the facade decides, per route, whether a request is served by the monolith or by a new GKE service.

Request path, following the control flow:

1. A client — a driver’s handheld, the public track-and-trace web app, or a shipper’s EDI gateway — hits Akamai at the edge for TLS termination, global anycast, WAF, and bot mitigation. The morning-after-a-holiday tracking spike is mostly absorbed here by Akamai’s CDN caching of tracking pages that change only when a scan event lands.
2. Identity is brokered before routing. The carrier’s workforce (drivers, dispatchers, ops) authenticates through Okta as the workforce IdP, federated to Microsoft Entra ID where the org already runs Microsoft 365 — so internal tools see a first-class Entra token, and Okta conditional-access policies (device posture, geo) gate the driver app. External shipper APIs authenticate with OAuth client credentials issued by Apigee. This split — humans through Okta/Entra, machines through the API gateway — is deliberate.
3. Traffic reaches the facade, which is Apigee for the externally-published API surface (shipper-facing rating, label, and tracking APIs that need quotas, monetization, developer portal, and versioning) and the Kubernetes Gateway API (Envoy-based, via GKE Gateway) for internal east-west and the web front end. Apigee owns the product API contract; Gateway API owns routing into the mesh. Both consult the same routing rules: a path like /v2/track/** now points at the new tracking service, while /v2/rate/** still proxies to the monolith.
4. For routes that have been extracted, the request lands in the GKE mesh on the relevant microservice — Tracking, Rating, Label, etc. — each a Deployment with its own HPA, its own release cadence, and its own on-call. Cloud Service Mesh (managed Istio) provides mTLS between services, retries, circuit breaking, and the traffic-splitting primitive that makes canaries and the strangler cutover itself possible.
5. For routes not yet extracted, the facade proxies straight through to the monolith, which still runs — but now inside GKE as a single large Deployment (lifted in early so old and new share one platform, network, and observability plane), talking to the original schema.
6. Each microservice owns its data in its own Cloud SQL instance (or Spanner for the globally-distributed tracking ledger). Synchronous calls between services are the exception; the default boundary is asynchronous via Pub/Sub, so a scan event published by the dispatch path fans out to tracking, billing, and notifications without any of them being on each other’s critical path.

Control plane, independent of the request path: infrastructure is Terraform, node and OS configuration is Ansible for the few VM-based virtual appliances at the network edge, application delivery is Argo CD pulling declarative manifests from Git, and the entire request — across the old/new seam — is traced in Datadog so you can see a request hop from Apigee into a new service, out to Pub/Sub, and watch the monolith consume the event.

The seam: how old and new coexist

Most of the hard engineering in a strangler migration is not in the new services — it is in the seam between old and new, and three problems live there.

Routing and cutover. Each capability extraction follows the same ritual: deploy the new service alongside the monolith, replay or mirror a slice of production traffic to it to validate parity, then shift real traffic with a weighted route — 1%, 10%, 50%, 100% — using Cloud Service Mesh traffic splitting (or Apigee target weighting for the external surface). At every step the rollback is a single weight change. The new tracking service ran in shadow mode for two weeks, receiving mirrored production reads and having its responses diffed against the monolith’s, before it served a single real user.

Data ownership and the dual-write trap. The monolith has one giant schema; the microservices must own their data, or you have not decoupled anything. You cannot move all the data at once, so during the transition the new service and the monolith both need a consistent view. The wrong answer is dual writes (the app writes to both stores) — it has no transactional guarantee and silently drifts. The right answer is Change Data Capture: the monolith keeps writing its schema, Datastream captures row-level changes from the monolith’s Cloud SQL and streams them into the new tracking service’s store via Pub/Sub, so the new service has an eventually-consistent, read-correct copy of the data it is taking over — with no code change in the monolith. When tracking is fully extracted, writes flip to the new service and the CDC feed reverses or retires.

Distributed transactions become sagas. A single monolith transaction — “rate the shipment, reserve the label number, debit the shipper’s account” — spanned three modules and one ACID commit. Split across Rating, Label, and Billing services, that ACID transaction is gone; forcing a two-phase commit across services recreates the coupling you are trying to escape. Replace it with a saga: each service does its local transaction and publishes an event; a failure triggers a compensating action (release the reserved label number, reverse the debit). Pub/Sub carries the saga events; idempotency keys make retries safe.

Component breakdown

A few of these choices deserve the why, because they are the ones teams get wrong.

Why lift the monolith into GKE early, before extracting anything. It is tempting to leave the monolith on its VMs and only put new services on GKE. Don’t — straddling two platforms means two networking models, two security baselines, two observability stacks, and a painful network hop across the seam on every proxied request. Lifting the monolith into GKE as one large Deployment on day one (no decomposition yet, just containerize and run) means old and new share one VPC, one mTLS mesh, one Datadog plane, and one Argo CD. The seam becomes an in-cluster route, not a cross-environment integration. This is the single highest-leverage early move.

Why Pub/Sub by default, sync by exception. Every synchronous call between two services couples their availability — if Billing is down, anything that calls it synchronously is also down, and you have re-created the monolith’s blast radius across the network. The default boundary is therefore an event on Pub/Sub: the dispatch path publishes a package.scanned event and is done; tracking, billing, and notifications each consume it on their own schedule. Synchronous gRPC/REST is reserved for the few cases where the caller genuinely needs the answer now (rating must return a price to the shipper in the same request). Async-first is what actually buys the fault isolation the CTO demanded.

Why one database per service, even though it is more work. A shared database is a shared schema, and a shared schema is a coupling that defeats independent deployability — change a column and you must coordinate every service that reads it. Each service owning its own Cloud SQL instance (with private IP, no public exposure) means a service’s schema is its private implementation detail. The transitional cost is real — CDC pipelines, eventual consistency, sagas instead of joins — and it is the price of the decoupling.

Implementation guidance

Sequence the extractions by value and decoupling, not by what is easy. Pick the first capability with three properties: high business value, relatively clean boundaries in the existing code, and a read-heavy or clearly-bounded data footprint. For this carrier that was track-and-trace — it is the most-hit, most spiky, most independently-scalable surface, it is largely read-mostly (scan events in, status queries out), and its data boundary is clean. Extracting it first delivered the elastic-scale win immediately and proved the seam machinery before touching anything transactional like billing. Rating came second; billing — the hardest, most transactional, most regulated — came last, when the team had earned the experience.

Provision with Terraform; gate the IaC with Wiz Code. The network is the first deliverable: a regional VPC, a regional GKE cluster (Autopilot mode for new services so you are not managing nodes; a Standard node pool for the heavier monolith), Cloud SQL instances on private IP only, and Private Service Connect for Pub/Sub and other Google APIs so no data-plane traffic traverses public IPs. Every Terraform change runs through GitHub Actions authenticating to GCP via OIDC Workload Identity Federation (no stored service-account keys to leak), and Wiz Code scans the plan in the pull request — a Cloud SQL instance proposed with a public IP, or an over-broad IAM binding, fails the check before merge, not after deploy.

A minimal Terraform shape for the GKE Autopilot cluster and a per-service Cloud SQL instance communicates the intent — private, no public surface:

resource "google_container_cluster" "primary" {
  name             = "carrier-prod-gke"
  location         = "asia-south1"
  enable_autopilot = true                 # new services: no node ops
  private_cluster_config {
    enable_private_nodes    = true         # nodes have no public IPs
    enable_private_endpoint = false        # control plane reachable from authorized nets
  }
  workload_identity_config { workload_pool = "carrier-prod.svc.id.goog" }
}

resource "google_sql_database_instance" "tracking" {
  name             = "tracking-db-prod"
  database_version = "POSTGRES_16"
  region           = "asia-south1"
  settings {
    tier = "db-custom-4-16384"
    ip_configuration {
      ipv4_enabled    = false             # NO public IP
      private_network = google_compute_network.vpc.id
    }
    backup_configuration { enabled = true  point_in_time_recovery_enabled = true }
  }
}

Delivery is GitOps with Argo CD. Each microservice is an Argo CD Application that syncs declarative manifests from Git; the desired state of the cluster is the repo, so a deploy is a merged PR and a rollback is a git revert. Argo CD’s drift detection means a hand-edited cluster is automatically reconciled back to Git. The strangler cutover itself is GitOps too: the VirtualService weight that shifts traffic from monolith to new service is a manifest in Git, so a cutover and its rollback are auditable commits — which is exactly what the ServiceNow change gate references when approving each weight increase.

Identity: federate the humans, dynamic-lease the machines. Workforce SSO flows Okta → Entra: drivers and dispatchers authenticate with Okta (conditional access on device posture and geography), Okta federates to Entra over OIDC where the org’s M365 identities live, and tools consume the Entra token. Service-to-service auth inside the mesh is mTLS via Cloud Service Mesh — every pod gets a workload identity and certificates rotate automatically. The secrets that are not identities — Cloud SQL passwords, third-party carrier-integration API keys, JWT signing keys — live in HashiCorp Vault, which issues dynamic, short-lived Cloud SQL credentials per service (so a leaked credential expires in minutes and is scoped to one database) and injects them via the Vault Agent sidecar authenticated by GKE Workload Identity. No long-lived database password is ever written to a Kubernetes Secret.

Enterprise considerations

Security & Zero Trust. The mesh is Zero Trust by construction — mTLS everywhere, identity-based service-to-service authorization, no implicit trust between pods. Layer on top: Wiz runs continuous CSPM and attack-path analysis across GKE, Cloud SQL, and IAM, alerting the moment a resource drifts to public exposure or an IAM binding widens, while Wiz Code shifts that left into the pull request so misconfiguration is caught pre-merge. CrowdStrike Falcon sensors run as a DaemonSet on the GKE node pool for container runtime threat detection — anomalous process execution inside a tracking pod, a reverse shell, lateral movement — feeding the carrier’s SOC. A sustained SLO breach or a Falcon detection auto-raises a ServiceNow incident so security and ops have a ticket, not just a dashboard. Org Policy denies any Cloud SQL instance or load balancer created with a public IP, and Wiz independently verifies the policy is actually holding.

Cost optimization. Decomposition can increase cost if you are careless — more instances, more databases, mesh sidecar overhead — so engineer for it.

The honest framing for the CFO: the platform cost may rise modestly, but it is now attributable per service (each team’s GKE namespace and Cloud SQL instance is a cost line), peak scale is paid only where it is needed, and the cost of an outage — a 47-minute peak-hour platform failure — is the number the migration is really reducing.

Scalability. This is the headline win. Each service scales independently on its own signal: track-and-trace scales pods on request concurrency and CPU via HPA, absorbing its 40× holiday spike without anyone touching billing; Pub/Sub consumers scale on subscription backlog depth; billing stays at a flat baseline because its load is flat. The monolith, while it survives, scales as one unit — which is precisely the constraint each extraction relieves. Spanner under the global tracking ledger scales horizontally for the read-heavy status-query load that dominates traffic.

Failure modes, and what each one looks like. Name them before they page you.

• A cutover that regresses parity — the new service returns subtly different results than the monolith for some edge case. Mitigation: shadow/mirror traffic and response-diffing before the first real weight, and keep every cutover a single revertible weight change.
• Pub/Sub redelivery causing double-processing — at-least-once delivery means a saga step can run twice (a shipper double-debited). Mitigation: idempotency keys on every event handler and dead-letter topics for poison messages.
• A saga that fails mid-flight — rating succeeded, label reserved, billing failed; without compensation the system is inconsistent. Mitigation: explicit compensating actions per step and an orchestrator that drives them, monitored in Datadog.
• CDC lag during transition — the new tracking service reads stale data because Datastream fell behind. Mitigation: alert on replication lag; keep the monolith authoritative until the new service is consistent and writes have flipped.
• A synchronous dependency outage cascading — a sync call to Billing takes everything calling it down. Mitigation: prefer async; where sync is unavoidable, circuit breakers and timeouts in the mesh so a slow dependency sheds rather than cascades.

Reliability & DR (RTO/RPO). Decide the numbers per tier. The GKE cluster is regional (control plane and nodes across zones) so a zonal failure is transparent. Cloud SQL runs with a regional HA configuration and point-in-time recovery; the global tracking ledger on Spanner gives multi-region writes with near-zero RPO. Pub/Sub is regional and durable, buffering events through a consumer outage so nothing is lost. A pragmatic target for the carrier’s interactive surfaces: RTO 15 minutes, RPO 5 minutes, with the tracking ledger effectively zero-RPO on Spanner. Akamai health checks drive edge failover. Critically, during the migration the monolith remains the fallback for any not-yet-cutover route, so the strangler itself is a continuity strategy — you are never betting the business on the new path before it has earned trust.

Observability across the seam. This is non-negotiable for a strangler migration, because a single request can cross old and new. Instrument end-to-end distributed tracing in Datadog with OpenTelemetry: one trace covering Apigee → Gateway API → new GKE service → Pub/Sub publish → monolith consume → Cloud SQL, so when latency regresses you can see which side of the seam owns it. Emit the metrics that actually matter — per-route monolith-vs-service traffic share (so you can watch the fig strangle the tree), per-service p95 latency and error rate, Pub/Sub backlog and redelivery rate, saga completion vs compensation rate, and CDC replication lag. Datadog APM, log correlation, and SLO monitors drive the ServiceNow auto-ticketing on breach. New engineers learn the mesh, sagas, and on-call runbooks through a required Moodle course before they are handed the pager for a service — turning the architecture’s complexity into something teachable rather than tribal.

Governance. Pin everything: container images by digest (never latest), Helm/manifest versions in Git, and the cutover weights as auditable commits. Org Policy denies public IPs and requires private clusters; Wiz is the independent check the controls hold. Each capability extraction passes a ServiceNow change approval before its weight is raised — giving ops and risk a documented gate for each step of a migration that, in aggregate, reshapes a revenue-critical platform.

Explicit tradeoffs

Accept these or do not start. Microservices trade in-process simplicity for distributed-systems complexity, and the bill is real: network calls fail in ways method calls do not, eventual consistency replaces a clean ACID join, sagas replace transactions, and you now operate many services instead of one. Debugging a request means a distributed trace, not a single stack trace — which is exactly why Datadog tracing across the seam is load-bearing, not optional. The data decomposition (CDC pipelines, per-service stores, sagas) is the hardest part and the part teams underestimate. And the strangler approach itself has a cost: you run both the monolith and the new services simultaneously for the duration, paying for two worlds during the transition and carrying the seam’s complexity until the last capability is extracted.

The alternatives, and when they win. If your monolith is small, well-factored, and not under scaling or fault-isolation pressure, leave it alone — microservices are a tax you pay for independent deployability and isolation you do not yet need, and a tidy modular monolith is a perfectly good destination. If you genuinely can freeze features and the system is small, a rewrite is occasionally defensible — but almost never for a 2.3-million-line revenue platform. If you want service boundaries without the operational weight of many deployables, a modular monolith (enforced module boundaries in one process) gets you much of the maintainability with none of the distributed-systems cost, and is a legitimate stopping point. The strangler fig wins precisely the case the carrier is in: a large, business-critical monolith that cannot stop earning, under real and uneven scaling pressure, where the goals are independent deployability and fault isolation and the appetite for big-bang risk is correctly zero.

The shape of the win

For the carrier, the payoff is not “we use Kubernetes now.” It is that the next November, a rating-engine bug degrades rating for the shippers hitting that one service — and track-and-trace, dispatch, and billing keep running, because they are separate processes behind separate pods with circuit breakers between them. It is that the tracking team ships a fix on Tuesday afternoon without re-certifying billing. It is that the holiday-morning tracking spike scales tracking pods 40× while billing sits flat and the bill reflects exactly that. Every piece upstream — the Apigee and Gateway API facade that hid the migration from clients, the Cloud SQL-per-service stores fed by Datastream, the Pub/Sub boundaries carrying sagas, the Argo CD GitOps cutovers, the Datadog traces spanning old and new, the Vault-leased credentials, the Wiz and Falcon guardrails, the ServiceNow gates — exists to turn that 47-minute platform outage into a contained, single-service incident. The architecture here is the destination; the strangler fig is how you get there one revertible route-flip at a time, with the business running every single day in between.

Going deeper

The narrative above is the what. This section is the why it works and where it bites — the patterns, the vocabulary, and the failure shapes an experienced engineer needs to run one of these migrations without walking into the well-known ditches.

The strangler fig, precisely — and the anti-corruption layer

The pattern is often quoted and rarely stated precisely. The strangler fig has exactly two moving parts: a facade that intercepts requests and routes them to old-or-new, and a growing set of new services behind it. The facade is what makes the migration invisible to clients and what makes every cutover reversible. That much the main body covered.

The part teams skip is the anti-corruption layer (ACL) — a term from Domain-Driven Design and the single most important defense against a migration that “finishes” but leaves you with the same mess in new packaging. When the new Tracking service needs data or behavior that still lives in the monolith, the naive move is to have it call the monolith’s API and speak the monolith’s data model directly. Do that and the monolith’s legacy naming, its quirks, its status_code=7-means-delivered enums, and its accidental coupling all leak straight into your clean new service — you have moved the crud, not removed it. The ACL is a translation layer the new service owns: it maps the monolith’s model to the new service’s clean domain model on the way in, and back on the way out, so the legacy concepts are quarantined at the boundary and never contaminate the new code. Concretely, it is a small adapter package (or a dedicated translator service for a big seam) that every call across the boundary passes through. The rule of thumb: any time the new service talks to the old system, an ACL sits between them. When the monolith is finally retired, you delete the ACL and the new service does not notice.

Finding the seams: decompose by domain, not by layer

The highest-leverage decision in the whole migration is where you cut, and the most common way to get it wrong is to cut along technical layers — a “data-access service,” a “business-logic service,” a “presentation service.” That looks tidy on a whiteboard and is a disaster in practice: every feature now spans all three layers, so every change touches every service, and you have built a distributed monolith (more on that below) that is strictly worse than the thing you started with.

Cut along business capabilities instead — vertical slices, each owning its API, its logic, and its data top to bottom. Domain-Driven Design gives this a rigorous frame: a bounded context is a boundary within which a term has exactly one meaning. “Shipment” means something specific to Rating (a thing with weight, dimensions, and a price) and something subtly different to Customs (a thing with a declared value, an HS code, and a destination country’s rules). The place where that meaning changes is the seam. A practical way to surface them is event storming: get the domain experts in a room, write every business event on a sticky note in time order (“shipment rated,” “label reserved,” “parcel scanned,” “invoice issued”), and watch where the nouns and the language cluster — the clusters are your bounded contexts.

Two heuristics keep you honest:

• A service should own a noun and the verbs that mutate it. Rating owns rates and the act of rating; Billing owns invoices and charging. If two “services” both write the same noun, they are one context you split by mistake.
• If two services constantly need each other’s data inside the same transaction, they are probably one bounded context. Chatty synchronous coupling is the symptom of a boundary drawn in the wrong place. Redraw it rather than paper over it with more calls.

Database-per-service in depth: the transition, sagas, and the outbox

Per-service data is where the theory meets the hardest engineering. Restating the goal: each service owns a private database that no other service reads directly, so the schema is an implementation detail and can change without coordinating anyone. The problem is that you cannot teleport a decade of data into per-service stores overnight, so you need a transition strategy that keeps old and new consistent enough to serve traffic.

The dual-write trap, stated exactly. The tempting shortcut is to have the application write every change to both the old store and the new store in the same request handler. It has no atomicity: there is no transaction spanning two databases, so a crash (or a network blip, or a timeout) after the first write and before the second leaves the two stores permanently divergent, and nothing ever reconciles them. Dual writes do not fail loudly — they drift silently, and you discover it weeks later when a reconciliation report does not balance. Never dual-write.

CDC for reads during the transition. While the monolith is still authoritative, the new service needs a read-correct copy of the data it is taking over. Change Data Capture — Datastream reading the monolith’s Cloud SQL write-ahead log and streaming row changes through Pub/Sub into the new service’s store — gives exactly that, with zero code change in the monolith. The new service serves reads from its own eventually-consistent copy; the monolith still owns writes. When the new service is ready to own writes, you flip: the new service becomes authoritative and the CDC feed either reverses (to keep the monolith’s view fresh for not-yet-migrated consumers) or retires.

The outbox pattern — closing the dual-write gap for events. Once the new service owns writes, it usually needs to emit an event as part of a write (“a scan was recorded, tell billing and notifications”). You are back to a two-systems atomicity problem: you cannot atomically “commit the database row and publish to Pub/Sub,” because a crash between them either loses the event or publishes a phantom for a write that rolled back. The outbox pattern solves it by making the event part of the same local transaction:

-- one local ACID transaction in the service's own Postgres
BEGIN;
  INSERT INTO scans (shipment_id, location, scanned_at)
    VALUES ('SHP-42', 'BLR-HUB', now());
  INSERT INTO outbox (id, aggregate, event_type, payload, dispatched)
    VALUES (gen_random_uuid(), 'shipment', 'package.scanned',
            '{"shipmentId":"SHP-42","location":"BLR-HUB"}', false);
COMMIT;

A separate relay — Datastream tailing the outbox table, or a small poller — reads rows where dispatched = false, publishes each to Pub/Sub, and marks it dispatched. Because the business row and the outbox row commit together, the event is published if and only if the write committed: no lost events, no phantoms. The relay is at-least-once (it may publish a row twice if it crashes after publishing but before marking), which is exactly why every consumer needs idempotency keys.

Sagas, orchestration vs choreography, and why not 2PC. A workflow that was one ACID transaction in the monolith — rate, reserve the label number, debit the shipper — becomes a saga: a sequence of local transactions, each in one service, stitched together by events. There are two styles:

• Choreography — no central brain; each service reacts to the previous one’s event. Rating publishes shipment.rated; Label consumes it, reserves a number, publishes label.reserved; Billing consumes that and debits. Simple and loosely coupled for short flows; hard to see the whole flow once it has more than a few steps.
• Orchestration — a central coordinator (a saga orchestrator) explicitly drives each step and each compensation. More moving parts, but the flow and its failure handling live in one place, which is far easier to reason about for complex or regulated flows like billing.

The failure handling is the crux: because there is no distributed rollback, a failed step triggers a compensating action — a new forward transaction that semantically undoes a completed one. You cannot “roll back” a sent email; you send a correction. You cannot roll back Billing’s commit from Label’s service; Label runs a compensation that releases the reserved number. And you deliberately do not reach for two-phase commit (2PC) to recreate the old ACID guarantee across services: 2PC re-couples exactly what you are trying to separate (every participant and the coordinator must be up for anything to commit — the monolith’s availability blast radius, rebuilt over the network), and most managed stores will not enlist in a cross-service XA transaction anyway. The accepted trade is eventual consistency plus sagas plus idempotency, monitored so you can see compensations firing.

The seam in the platform: API gateway (north-south) plus mesh (east-west)

Two jobs get conflated under the word “gateway,” and separating them clarifies the whole seam.

• The API gateway handles north-south traffic — requests from outside clients into the platform — plus the product concerns: authentication, quotas, versioning, the published contract, the developer portal. This is where the client-facing cutover happens. Apigee (or Gateway API for the internal front end) shifts external traffic slice by slice with a weighted route. This is the surface shippers and the public tracking page hit.
• The service mesh handles east-west traffic — service-to-service inside the cluster — plus mTLS, retries, circuit breaking, and the fine-grained traffic splitting that powers internal canaries. Cloud Service Mesh (Istio) shifts internal traffic with a VirtualService weight.

The cutover knob at the edge is a weighted HTTPRoute on the Gateway API — send 10% of the /v2/track path to the new service, 90% still to the monolith:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: tracking-cutover
  namespace: edge
spec:
  parentRefs:
    - name: carrier-gateway
  hostnames:
    - "api.carrier.example"
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /v2/track
      backendRefs:
        - name: monolith          # 90% still hits the monolith
          port: 8080
          weight: 90
        - name: tracking-svc      # 10% to the new service
          port: 8080
          weight: 10

The equivalent knob inside the mesh is an Istio VirtualService with the same weights:

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: tracking
  namespace: prod
spec:
  hosts:
    - tracking.prod.svc.cluster.local
  http:
    - route:
        - destination:
            host: monolith.prod.svc.cluster.local
          weight: 90
        - destination:
            host: tracking-svc.prod.svc.cluster.local
          weight: 10

Both manifests live in Git, so a cutover and its rollback are auditable commits — a git revert sets the weight back. The mechanics of these weighted routes, and how to ramp them safely, are the whole subject of the Gateway API traffic-splitting and migration lesson.

Incremental migration: feature flags, shadow traffic, and traffic shifting

“Cut over gradually” hides three distinct levers that people conflate — and using the right one for the right job is what makes a cutover safe.

• Traffic shifting (weights). Percentage of requests, controlled at the gateway or mesh. Coarse (it does not know which user), instant to roll back, and perfect for “ramp confidence” steps: 1% → 10% → 50% → 100%.
• Feature flags (a switch your code reads). Application-level, so it can be per-user, per-tenant, or per-shipper — far finer than weights. You dark-launch the new rating path to one internal test shipper, or ramp by cohort, while everyone else stays on the monolith. A flag is just config the service reads at request time:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cutover-flags
  namespace: prod
data:
  track_read_via_service:  "true"    # reads served by the new service
  track_write_via_service: "false"   # writes still go to the monolith
  rate_via_service:        "false"   # rating not cut over yet

• Shadow / mirror traffic. Send a copy of real production traffic to the new service, throw away its response, and diff it against the monolith’s. This validates parity under real load with zero user risk — it is how the carrier’s tracking service earned trust for two weeks before its first real weight.

Two more techniques round it out. Branch by abstraction works inside the monolith: introduce an interface in front of the capability you are extracting, so you can swap its implementation from an in-process call to a remote call to the new service behind that interface — no big-bang edit. And the whole thing is progressive delivery when you gate each weight increase on metrics (error rate, p95 latency, saga completion rate) and auto-roll-back on a breach; the same machinery is covered for canaries in the Argo Rollouts progressive delivery lesson. The safe recipe combines them: shadow first (parity), then feature-flag a cohort (business control), then traffic-shift the percentage up (scale confidence), each step reversible and metric-gated.

When microservices are the wrong call: the distributed monolith

The failure mode to fear more than the monolith itself is the distributed monolith — a system that has paid the full price of microservices (network calls, many deployables, operational overhead, a distributed-tracing requirement) and received none of the benefits, because the services are still coupled. It is strictly worse than the monolith you left: the coupling is still there, now with added latency and partial-failure modes. Diagnose it by four smells:

• Services that must be deployed together or in a specific order — a change to A requires a synchronized change and release of B.
• A shared database several services read and write, so a schema change ripples across all of them.
• Chatty synchronous call chains — one user request fans out to a dozen sync hops, so end-to-end availability is the product of each service’s availability (ten 99.9% services in a chain give you ~99% together) and latencies stack.
• A change to one “microservice” routinely forces edits to three others.

The root cause is almost always one of the two mistakes above: decomposing by layer instead of domain, or splitting too fine (nanoservices) so a single business operation is smeared across many services that must coordinate over the network. The cure is often to merge wrongly-split services back into their bounded context — going that direction is legitimate and common, not an admission of failure. And the right default for a new or small system is a modular monolith: enforce module boundaries in one process, keep the option to split later, and extract a service only when you have a concrete, named pressure — this module must scale 40× alone, this module’s bug must not take the rest down, this module’s team needs its own release cadence. “Microservices are modern” is not a pressure.

Conway’s Law: the org is part of the architecture

Conway’s Law observes that a system’s structure inevitably mirrors the communication structure of the organization that builds it — four teams building a compiler produce a four-pass compiler. The corollary for this migration is unforgiving: if you split the system into services but keep the org as one shared pool with a central DBA team owning the one schema, the services will re-converge on that shared schema and you will get a distributed monolith. The technology cannot outrun the org chart.

To get independently-deployable services you need independently-responsible teams. A stream-aligned team owns a service end to end — its code, its database, its on-call pager, and its release cadence — with no central gatekeeper that re-couples it to the others. The inverse Conway maneuver is the deliberate move of shaping teams to match the architecture you want before writing the code, so the system grows toward the target rather than back toward the old shape. This is why the main body’s per-service on-call and required Moodle onboarding are not soft extras: the architecture’s complexity is only sustainable if the org is reshaped to own it. Decomposing the software without decomposing the teams is the most expensive way to fail at this.

Practice challenges

Work these in order — they escalate from beginner to advanced, and each assumes the previous. Try before opening the solution.

Challenge 1 (beginner) — Pick the first extraction. The monolith contains rating, label generation, manifesting, track-and-trace, customs, billing, and dispatch. Which capability do you carve off first, and what are the three properties that make it the right choice?

<details> <summary>Solution</summary>

Track-and-trace. The three properties: (1) high business value and the clearest scaling pressure — it is the most-hit, spikiest surface (40× on a holiday morning), so extracting it delivers the elastic-scale win immediately; (2) a clean data boundary — it is largely read-mostly (scan events in, status queries out), which makes the CDC transition straightforward; (3) no hard transactional coupling — it is not entangled in the rate/reserve/debit transaction, so you prove the seam machinery on a safe slice. Do billing last — it is the most transactional and regulated, and you want the team’s earned experience before touching it. Why: sequence by value and decoupling, never by what looks easy. </details>

Challenge 2 (beginner) — Write the weighted cutover route. On the Gateway API, send 10% of /v2/track traffic to a new tracking-svc (port 8080) and 90% to monolith (port 8080). Then state exactly what the rollback is.

<details> <summary>Solution</summary>

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: tracking-cutover
  namespace: edge
spec:
  parentRefs:
    - name: carrier-gateway
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /v2/track
      backendRefs:
        - name: monolith
          port: 8080
          weight: 90
        - name: tracking-svc
          port: 8080
          weight: 10

The rollback is a single number: set tracking-svc weight to 0 (or git revert the commit that changed it). Why: the strangler’s cutover knob is a weight, so rollback is never a redeploy — it is one field. </details>

Challenge 3 (intermediate) — Kill the dual-write. A developer proposes that the new tracking service, in its request handler, write each scan event to both its own database and the monolith’s database “to keep them in sync.” Explain the flaw precisely, then rewrite it correctly.

<details> <summary>Solution</summary>

The flaw: there is no transaction spanning two databases, so a crash (or timeout) after the first write and before the second leaves the two stores permanently divergent, with nothing to reconcile them. Dual writes drift silently. The fix — the outbox pattern: in the service’s own local transaction, write the scan row and an outbox row atomically, then let a relay publish the event:

BEGIN;
  INSERT INTO scans (shipment_id, location, scanned_at)
    VALUES ('SHP-42', 'BLR-HUB', now());
  INSERT INTO outbox (id, event_type, payload, dispatched)
    VALUES (gen_random_uuid(), 'package.scanned',
            '{"shipmentId":"SHP-42","location":"BLR-HUB"}', false);
COMMIT;

A relay (Datastream tailing outbox, or a poller) publishes undispatched rows to Pub/Sub; the monolith consumes them. Why: the business row and the event commit together, so the event fires if and only if the write committed — no lost events, no phantoms. </details>

Challenge 4 (intermediate) — Design the saga. The monolith’s single transaction “rate → reserve label number → debit shipper” must survive being split across Rating, Label, and Billing. Sketch the choreography, name the compensation when Billing fails, and say what makes redelivery safe.

<details> <summary>Solution</summary>

Choreography: Rating does its local commit and publishes shipment.rated; Label consumes it, reserves a number, publishes label.reserved; Billing consumes that and debits, publishing shipment.billed. On Billing failure: Billing publishes billing.failed; Label consumes it and runs its compensating action — releasing the reserved number (a new forward transaction, not a DB rollback) — and the shipment is marked unrated for retry. Redelivery safety: every handler carries an idempotency key (e.g. shipmentId), so a duplicate label.reserved for the same shipment is a no-op. Why: there is no 2PC and no shared transaction; consistency is restored by compensations, and at-least-once delivery is made safe by idempotency. </details>

Challenge 5 (advanced) — Diagnose the distributed monolith. A squad reports: deploying the Tracking service requires deploying Dispatch and Billing in lockstep; all three read the shipments table directly; and a single tracking query makes synchronous calls to Dispatch and Billing. Name the anti-pattern, the three specific smells, and the remediation.

<details> <summary>Solution</summary>

Anti-pattern: a distributed monolith — all the cost of microservices, none of the benefit. The three smells: (1) lockstep deploys → no independent deployability; (2) shared shipments table → shared-schema coupling; (3) synchronous fan-out → coupled availability (end-to-end uptime is the product of each service’s) and stacked latency. Remediation: give each service its own store (bridge the transition with CDC/outbox), replace the synchronous fan-out with Pub/Sub events so Tracking is not on Dispatch’s or Billing’s critical path, and — if the three genuinely cannot be decoupled — recognize they may be one bounded context wrongly split and merge them. Why: the fix is to restore the three prizes (independent deploy, fault isolation, independent scale) the split was supposed to buy. </details>

Challenge 6 (advanced) — Order the safe cutover. You have built the new Rating service. Put these steps in the correct order and justify it: (a) raise the mesh weight to 100%, (b) shadow-mirror production traffic and response-diff, © feature-flag the new path for one internal test shipper, (d) retire the monolith’s rating code, (e) ramp the weight 1% → 10% → 50%, (f) obtain the ServiceNow change approval.

<details> <summary>Solution</summary>

Order: b → c → f → e → a → d. Shadow first (b) to prove parity at zero risk; dark-launch to one internal shipper via a feature flag © to test the real path on a controlled cohort; get the change gate (f) before touching real production weights; ramp the weight with a metric gate at each step (e); reach 100% (a); and only then retire the old code (d) — and keep it revertible until you are certain. Why: risk decreases monotonically — every step before (d) is reversible, so you do the one irreversible step last and last only. </details>

Common beginner mistakes

These are misconceptions, not symptoms — the mental model is wrong, so the fix is to re-frame, not to patch a flag.

• “We’ll rewrite it on Kubernetes and cut over on launch night.” The belief is that a clean-slate rewrite is faster than untangling the old code. It is not: you inherit a decade of undocumented edge cases you can only discover by running the old system, and you carry maximum risk until the single night you flip everything. Right model: incremental extraction that ships value continuously and keeps the monolith authoritative until each slice has earned trust.
• “Microservices — so split it into as many small services as possible.” Finer is not better. Split too fine and one business operation smears across a dozen services that must coordinate over the network, giving you a distributed monolith with worse latency and availability than the original. Right model: one service per bounded context (a noun and the verbs that mutate it); extract only where there is a named pressure, and merge back if you over-split.
• “The services can just share the monolith’s database for now.” It feels pragmatic and it permanently defeats the point: a shared schema is a coupling, so a column change ripples across every service and no one can deploy independently — and “for now” becomes forever. Right model: database-per-service from the start; bridge the transition with CDC and the outbox, never a shared table.
• “We split by layer — a data service, a logic service, a gateway service.” Horizontal layers look tidy but force every feature to cross every service, so every change touches all of them — the definition of a distributed monolith. Right model: decompose vertically by domain, each service owning its API, logic, and data top to bottom.
• “Eventual consistency is a bug; make it strongly consistent like the monolith was.” Reaching for a distributed transaction (2PC) to restore the old ACID guarantee recreates exactly the coupling you are escaping — every participant and the coordinator must be up for anything to commit. Right model: accept eventual consistency across services, restore correctness with sagas and compensating actions, and make handlers idempotent so at-least-once delivery is safe.
• “Leave the monolith on its VMs; only new services go on GKE.” Straddling two platforms means two networks, two security baselines, two observability stacks, and a slow cross-environment hop on every proxied request through the seam. Right model: lift the monolith into GKE as one large Deployment on day one so old and new share one VPC, mesh, and Datadog plane — the seam becomes an in-cluster route.
• “The migration is a technical project; the org chart is HR’s problem.” If you split the system but keep one shared team and a central DBA owning the schema, Conway’s Law pulls the services back into a distributed monolith. Right model: give each service a team that owns it end to end — code, data, and pager — because independently-deployable services require independently-responsible teams.

Glossary

• Monolith: one large application deployed as a single unit, where all features share one process, one codebase, and usually one database.
• Microservice: a small service that owns one business capability end to end — its own API, logic, and data — and deploys, scales, and fails independently of the others.
• Strangler fig (pattern): incrementally replacing a system by growing new services around it behind a facade and carving off one capability at a time, until the old system can be retired.
• Facade: the routing layer placed in front of the old system that decides, per route, whether a request is served by the monolith or a new service — so clients never see the migration.
• Anti-corruption layer (ACL): a translation layer a new service puts between itself and the legacy system, so the old data model and quirks do not leak into the new service’s clean domain model.
• Domain-Driven Design (DDD): an approach that models software around the business domain and its language; used here to decide where to cut the monolith.
• Bounded context: a boundary within which a term has one precise meaning; the natural seam along which to split services.
• Ubiquitous language: the shared, precise vocabulary of a bounded context — where the meaning of a word changes, a context boundary sits.
• Event storming: a workshop technique of laying out business events in time order to surface bounded contexts and where they meet.
• Database-per-service: each service owns a private database no other service reads directly, so its schema is an implementation detail and services stay decoupled.
• Shared database (anti-pattern): multiple services reading and writing one schema — a coupling that defeats independent deployability.
• Dual write: an application writing the same change to two stores in one request with no transaction spanning them; it drifts on any partial failure — avoid it.
• Change Data Capture (CDC): streaming row-level changes out of a database (here via Datastream) so another store can keep a read-correct copy without changing the source application.
• Outbox pattern: writing a business row and an event row in the same local transaction, then relaying the event row to the broker — so the event is published if and only if the business change committed.
• Saga: a sequence of local transactions across services, coordinated by events, where a failure triggers compensating actions instead of a distributed rollback.
• Choreography vs orchestration: two saga styles — services react to each other’s events (choreography), or a central coordinator drives the steps (orchestration).
• Compensating action: a new forward transaction that semantically undoes a completed step (release a reserved number, reverse a debit) — because you cannot roll back across services.
• Idempotency key: a stable identifier on an event or request so that processing it twice has the same effect as once — essential under at-least-once delivery.
• Two-phase commit (2PC): a protocol for one atomic transaction across multiple stores; avoided across services because it re-couples their availability.
• Eventual consistency: stores converge to the same state after a short delay rather than instantly — the trade you accept for decoupling.
• Distributed monolith: services that must deploy together, share a database, or call each other synchronously in chains — all the cost of microservices, none of the benefit.
• Modular monolith: one deployable with enforced internal module boundaries — much of the maintainability of services without the distributed-systems cost; a legitimate destination.
• Traffic shifting / weighted routing: sending a percentage of requests to the new service (e.g. 1% → 10% → 100%) via gateway or mesh weights; the strangler’s cutover knob.
• Shadow / mirror traffic: sending a copy of real traffic to the new service and discarding its response, to validate parity before it serves any user.
• Feature flag: a runtime switch your code reads to route specific users, tenants, or shippers down the new path, giving finer control than infrastructure weights.
• Branch by abstraction: introducing an interface inside the monolith in front of a capability, so its implementation can be swapped (in-process → remote service) without a big-bang change.
• North-south vs east-west: traffic from outside clients into the platform (north-south, handled by the API gateway) vs service-to-service traffic inside it (east-west, handled by the mesh).
• API gateway: the edge that terminates the external API contract — authentication, quotas, versioning, routing (here Apigee / Gateway API).
• Service mesh: an infrastructure layer (here Cloud Service Mesh / Istio) giving service-to-service mTLS, retries, circuit breaking, and traffic splitting without application code.
• Blast radius: the scope of what breaks when one component fails; shrinking it from “the platform” to “one service” is a core goal of the migration.
• Conway’s Law: a system’s structure mirrors the communication structure of the organization that builds it — so decoupling services requires decoupling teams.
• Inverse Conway maneuver: deliberately shaping teams to match the architecture you want, so the code follows the org toward it.
• Stream-aligned team: a team that owns one service end to end — its code, data, on-call, and release cadence.
• GKE Autopilot: a GKE mode where Google manages the nodes and you pay per pod resource request — used here for the new services.
• HPA (Horizontal Pod Autoscaler): scales the number of pods for a service on a signal (CPU, concurrency, queue depth) — how tracking scales 40× while billing stays flat.