GCP Enterprise Architecture: ML Platform / MLOps

In a nutshell

An ML platform is the shared factory that turns one-off model experiments into models a business can actually operate — retrain on a schedule, promote safely, roll back in seconds, and catch silent decay before customers do. MLOps is the name for that operational discipline. This lesson is a complete, reusable reference architecture for it on Google Cloud, built from Vertex AI and its neighbours (BigQuery, Dataflow, Pub/Sub).

The mental model that makes all of it click is a professional restaurant kitchen, not a home cook. A home cook — a data scientist’s notebook — can produce one great dish once. A restaurant has to produce it identically, every service, at volume, safely, with someone accountable for each plate. It manages that with four things this platform mirrors exactly:

• A single stocked pantry so the test kitchen and the line cooks use the same ingredients — that is the Feature Store, and it is what stops a model tasting different in production than it did in training.
• Numbered recipe cards so any cook reproduces the dish the same way — those are Vertex AI Pipelines, and they are why a model can be rebuilt bit-for-bit six months later.
• A pass where the head chef signs off before a plate leaves the kitchen — that is the Model Registry and its promotion gate; nothing reaches a customer without an explicit, recorded approval.
• A taster on every plate who flags the moment the food drifts from the standard — that is Model Monitoring, and when it fires it triggers a fresh batch (a retrain).

The whole thing is a loop, not a stack: data becomes features, features train a model, a gate promotes it, it serves predictions, monitoring watches those predictions, and drift sends you back to retrain. Get the loop right once and the tenth model costs a fraction of the first, because every plane is reused.

Level: Advanced · Time: ~35 min

Before you start — and what you’ll walk away with

This is an architecture-level lesson. It assumes you have trained a model at least once and met the individual services in passing. If any building block feels unfamiliar, skim these first:

• Vertex AI Pipelines: an MLOps production guide — the pipeline/training plane in depth.
• Real-time payments fraud-scoring pipeline on GCP — the fraud use-case this lesson keeps returning to.
• VPC Service Controls: perimeters and exfiltration prevention — the security perimeter the whole platform sits inside.

After working through it you will be able to:

• Draw the five-plane MLOps loop (feature, training, registry/governance, serving, monitoring) and name the exact failure mode each plane removes.
• Kill training-serving skew by design, using one Feature Store definition for both offline training and online serving.
• Separate CI/CD from CT — ship pipeline code one way and produce model versions another — and gate promotion on a metric or a human.
• Promote and roll back a model as a pointer move (an alias / a traffic split), with zero rebuild, and explain precisely why it is instant.
• Wire drift detection back to retraining so the platform self-heals instead of waiting for a business-metric dip.
• Right-size cost (batch vs endpoint, Spot retrains, honest min-replicas) and hold your ground on the trade-offs an experienced team will push on.

Most teams can train a model. Very few can answer, on a Tuesday afternoon when the model starts returning garbage, which version is live, what data it was trained on, whether the features it sees in production match the ones it learned from, and how to roll back without a redeploy that takes an afternoon. The gap between “we have a model” and “we operate models” is the entire discipline of MLOps, and on Google Cloud that discipline has a concrete shape: Vertex AI Pipelines, Feature Store, Model Registry, Endpoints, and Model Monitoring, wired together so that a model’s whole lifecycle is reproducible, governed, and observable. This is the reference architecture for that platform — one that a five-person data team and a five-hundred-person ML org can both run, because the moving parts are the same and only the scale changes.

The business scenario

Picture a mid-market lender, an e-commerce marketplace, or a B2B SaaS company — any business where a handful of models now sit on the revenue path. Fraud scoring on every transaction. Propensity-to-churn driving a retention budget. Dynamic pricing. Demand forecasting that decides what gets bought. These are not science projects; if the fraud model goes blind, money walks out the door, and if the churn model silently drifts, marketing burns spend on the wrong cohort.

The failure mode these organizations hit is organizational, not algorithmic. The data scientist’s notebook computes a “days since last login” feature one way; the production service computes it another way, off a slightly stale replica, and the model degrades for reasons nobody can see — classic training-serving skew. The model that is live in production was trained from a CSV on someone’s laptop that no longer exists, so it cannot be reproduced or audited. Promotion to production is a person copying a model file and updating a config, with no record of who approved it or what it scored on the holdout set. When a regulator or a board member asks “why did the model decline this customer,” there is no lineage to walk back from prediction to model version to training data.

The platform in this article solves exactly that. It gives you one place features are defined and served so training and serving see identical values; one pipeline definition that produces a model the same way every time; one registry that is the single source of truth for what is promotable and what is live; endpoints that scale and roll out safely; and continuous monitoring that tells you when reality has drifted away from the data the model learned on. It is widely useful because the problems are universal — they show up the moment a second model and a second engineer enter the picture.

Architecture overview

The architecture is best read as a loop that data flows around, not a top-to-bottom stack. Five planes cooperate: a feature plane, a training plane, a registry/governance plane, a serving plane, and a monitoring plane that closes the loop back to training.

Imagine the diagram as a wide horizontal ring. On the left, raw and curated data lands in BigQuery (warehouse tables, event streams materialized from Pub/Sub via Dataflow, batch loads from Cloud Storage). A feature engineering step — a Dataflow or BigQuery job orchestrated as part of a pipeline — transforms that data into feature values and writes them to the Vertex AI Feature Store. The Feature Store has two faces: an offline store backed by BigQuery that serves point-in-time-correct feature values for training, and an online store backed by Bigtable that serves the same features at single-digit-millisecond latency for live prediction. That dual face is the whole point — both training and serving draw features from one definition, killing skew.

Moving clockwise to the top, Vertex AI Pipelines (Kubeflow Pipelines under the hood, authored with the KFP DSL plus Google Cloud Pipeline Components) orchestrates the training workflow: pull a point-in-time feature set from the offline store, validate the data, train on Vertex AI Training (CPU/GPU/TPU), evaluate against a holdout, and — only if the eval gate passes — register the resulting model. Every pipeline run emits artifacts and lineage into Vertex ML Metadata, so each model is traceable back to the exact dataset, code, and hyperparameters that produced it. Vertex AI Experiments tracks runs so candidates can be compared.

At the top-right sits the Model Registry — the governance choke point. Pipelines do not deploy models; they register them as new versions of a logical model. A model version carries its eval metrics, its lineage, and a model card. Promotion is an explicit act: a human or an automated gate moves an alias (staging, production, champion, challenger) onto a specific version. Aliases, not version numbers, are what serving points at, so promotion and rollback are pointer moves, not redeployments.

Down the right side, the serving plane deploys the version that carries the production alias to a Vertex AI Endpoint. For request/response, that is an online endpoint — a dedicated public endpoint for isolation, or a dedicated private endpoint over Private Service Connect when traffic must stay inside the VPC. The endpoint autoscales replicas against CPU/GPU utilization, and a single endpoint can host multiple deployed models with traffic splitting for canaries and A/B tests. For bulk scoring, Batch Prediction reads from BigQuery or Cloud Storage and writes results back, no standing endpoint required.

At the bottom, the live request path: an application calls the endpoint; the serving container fetches the latest feature values for the entity (e.g. this user, this transaction) from the online Feature Store, combines them with request-time features, runs inference, and returns the prediction in tens of milliseconds. Prediction requests and responses are sampled to logging.

The monitoring plane closes the ring. Vertex AI Model Monitoring compares the live serving distribution against the training baseline and raises feature drift and training-serving skew alerts; with v2 it can also watch prediction drift and models served outside Vertex. Those alerts feed back to the top of the ring: drift is the signal to trigger continuous training — the same pipeline runs again on fresh data, produces a new model version, and the loop repeats. That feedback edge is what makes the diagram a loop rather than a pipeline.

Component breakdown

Each component earns its place by removing a specific failure mode. The table maps component to purpose to the configuration decisions that actually matter.

Two components are worth dwelling on because teams under-invest in them and pay later.

The Feature Store is the load-bearing wall. Without it, every model team rebuilds feature pipelines, and serving inevitably computes features differently from training. With it, a feature is defined once, materialized to BigQuery for training and to Bigtable for serving from the same job, and reused across models. The configuration that bites people is freshness: the online store is only as current as the last sync, so for features that must be real-time (a running transaction count this session), you write them straight to the online store from a streaming Dataflow job rather than relying on a periodic offline→online sync.

The Model Registry is the governance choke point, not a filing cabinet. Its value is the discipline it enforces: pipelines register, they do not deploy; promotion moves an alias; serving follows the alias. This indirection is what makes rollback instant — repoint production from version 7 back to version 6 and the next requests route there, with zero rebuild. The model card attached to each version (intended use, training data summary, eval metrics, fairness slices) is what you hand a regulator or risk committee, and it is generated by the pipeline, not written after the fact.

Implementation guidance

Project and environment topology. Use separate GCP projects per environment — ml-dev, ml-staging, ml-prod — under a folder, with a shared ml-shared project for the artifact registry and Terraform state. The Model Registry, Feature Store, and Endpoints are regional resources; pick a region (e.g. europe-west1) and keep data residency in mind, since the offline store is BigQuery in that region. Promotion across environments is a registry alias move plus a controlled redeploy in the higher project, gated in CI.

Infrastructure as Code. Provision the platform with Terraform using the google and google-beta providers. The durable, slow-changing resources belong in Terraform; the fast-changing model versions do not (those are produced by pipelines).

A clean split is: Terraform owns the platform (stores, endpoints, IAM, network, schedules); the KFP pipeline owns the model lifecycle (train, evaluate, register, and — guarded by an approval gate — deploy). Trying to manage model versions in Terraform fights the tool; the registry is the state store for those.

Pipeline authoring and CI/CD vs CT. Author pipelines with the KFP SDK, lean on Google Cloud Pipeline Components for the train/evaluate/register/deploy steps, and compile to the pipeline IR as a build artifact. Distinguish two cadences clearly:

• CI/CD (continuous integration/delivery) ships pipeline code. A change to feature logic or model code goes through Cloud Build: unit tests, compile the pipeline, run it in ml-dev, and on green, publish the compiled pipeline and container images.
• CT (continuous training) runs the published pipeline on a schedule or a trigger to produce fresh model versions. New data, or a drift alert from Model Monitoring landing on Pub/Sub, kicks off a run that registers a new candidate. Promotion to production stays gated — usually a human approval in Cloud Build or a metric threshold the candidate must beat the incumbent on.

Identity and networking wiring. Give each plane its own least-privilege service account: the pipeline SA can read the offline store, run training, and register models, but cannot deploy to prod; the deploy SA (used only behind the approval gate) can move aliases and deploy; the serving runtime SA can read the online store and write prediction logs and nothing else. For private serving, deploy the endpoint as a dedicated private endpoint over Private Service Connect so application VPCs reach it without traversing the public internet, and front it with the app’s existing internal load balancing. Keep BigQuery, GCS, and Vertex behind VPC Service Controls so data cannot exfiltrate to a project outside the perimeter even if credentials leak.

The prose above sets the rules; the four snippets below show what they look like in code. Every command, SDK call, and resource block here is real and current — treat the placeholder IDs (MODEL_ID, ENDPOINT_ID, PROJECT_NUMBER, bucket names) as your own values.

The eval-gated pipeline, in code

“Pipelines register, they do not deploy” is the single most important rule, and it is a structural choice you make in the pipeline itself. The KFP pipeline below trains, evaluates, and — only past the gate — registers a new version of an existing logical model. Deployment is deliberately absent: it is a separate, alias-gated act.

from kfp import dsl, compiler
from google_cloud_pipeline_components.v1.custom_job import CustomTrainingJobOp
from google_cloud_pipeline_components.v1.model import ModelUploadOp

# A tiny gate component: emit True only when the candidate clears the bar.
@dsl.component(packages_to_install=["google-cloud-aiplatform"])
def passes_eval(candidate_auc: float, min_auc: float) -> bool:
    return candidate_auc >= min_auc

@dsl.pipeline(
    name="fraud-continuous-training",
    pipeline_root="gs://PLACEHOLDER-BUCKET/pipeline-root",
)
def fraud_ct(project: str, region: str, min_auc: float = 0.92):
    # 1. Train on managed compute (Spot GPUs, distributed if needed).
    train = CustomTrainingJobOp(
        project=project,
        location=region,
        display_name="fraud-train",
        worker_pool_specs=[...],          # image, machine_type, accelerator elided
    )

    # 2. Evaluate against a holdout (your own component; emits candidate_auc).
    #    evaluate = evaluate_op(model=train.outputs["model"], holdout=...)

    # 3. The gate. Everything inside runs ONLY if the candidate clears min_auc.
    gate = passes_eval(candidate_auc=0.0, min_auc=min_auc)   # wire to evaluate output
    with dsl.If(gate.output == True, name="candidate-passed"):
        # 4. REGISTER as a new version of the logical model — do NOT deploy.
        ModelUploadOp(
            project=project,
            location=region,
            display_name="fraud-scorer",
            parent_model="projects/PLACEHOLDER/locations/europe-west1/models/MODEL_ID",
            unmanaged_container_model=...,   # serving image + artifact URI elided
        )

compiler.Compiler().compile(fraud_ct, package_path="fraud_ct.yaml")

Three details carry the lesson. First, the imports come from the google_cloud_pipeline_components.v1.* namespace even though the installed package is v2 — the components under v1 are interface-stable and decoupled from the package version, so this is the current, correct path. Second, parent_model is what turns ModelUploadOp into a new version of an existing logical model rather than a brand-new model — it is the hinge the whole registry discipline hangs on. Third, there is no deploy step: the pipeline’s job ends at register, and the compiled fraud_ct.yaml IR is a build artifact your CI publishes and your CT schedule runs.

Promotion and instant rollback

Registration is not deployment, and promotion is not a rebuild — it is moving an alias onto a version. The Vertex AI SDK exposes it directly, and because aliases are unique per model, assigning one to a new version atomically moves it off whatever version held it before:

from google.cloud import aiplatform

aiplatform.init(project="ml-prod", location="europe-west1")
model = aiplatform.Model(model_name="MODEL_ID")   # the logical model

# Promote the freshly-registered version 8 to production.
# `production` is unique, so this MOVES the pointer off whatever held it.
model.versioning_registry.add_version_aliases(new_aliases=["production"], version="8")

# 02:00, something's wrong. Registry-side rollback is the same one-liner,
# pointed back at the known-good version 7 — no rebuild, no re-registration:
model.versioning_registry.add_version_aliases(new_aliases=["production"], version="7")

remove_version_aliases(target_aliases=["production"], version="8") exists if you want to retire an alias outright rather than move it. Two operational notes matter here. The deploy SA is the only identity allowed to run this in prod — that IAM boundary is the promotion gate. And gcloud’s CLI support for aliases is thin, so teams script the move in their deploy job (SDK, as above) or hit the models:mergeVersionAliases REST endpoint directly. One subtlety the next section unpacks: moving the alias updates the registry’s record of what production is — it does not, by itself, re-route a live endpoint.

Canary rollout on an endpoint

A single endpoint can host several deployed models and split traffic between them — that is how you canary a new version, run an A/B test, and roll back instantly. Deploy version 8 to the existing endpoint taking 10% while the incumbent keeps 90%:

gcloud ai endpoints deploy-model ENDPOINT_ID \
  --region=europe-west1 \
  --model=MODEL_ID \
  --display-name=fraud-scorer-v8 \
  --machine-type=n1-standard-4 \
  --min-replica-count=3 \
  --max-replica-count=20 \
  --autoscaling-metric-specs=cpu-usage=60 \
  --traffic-split=0=10,OLDER_DEPLOYED_MODEL_ID=90

0 is the temporary ID of the model being deployed in this call; OLDER_DEPLOYED_MODEL_ID is the incumbent’s deployed-model ID (read it from gcloud ai endpoints describe ENDPOINT_ID), and the split must sum to 100. --autoscaling-metric-specs=cpu-usage=60 sets the 60% CPU utilization target the architecture uses as its default; for GPU-served models the key is gpu-duty-cycle=60 instead. To graduate the canary, resend with --traffic-split=0=100; to roll back, shift the split back to the incumbent — instant, because that older deployed model is still live on the endpoint. Note that --min-replica-count cannot be 0 for an online endpoint: there is no scale-to-zero for request/response serving, which is exactly why a model that gets one request an hour belongs in Batch Prediction, not on a warm endpoint.

Terraform for the durable platform

Terraform owns the slow-changing platform; pipelines own the fast-changing model versions. Two of the load-bearing resources, with schemas as the current google provider defines them:

# The regional online store (Bigtable-backed) that serves features at low latency.
resource "google_vertex_ai_feature_online_store" "online" {
  name   = "fraud_features_online"     # [a-z0-9_], up to 60 chars
  region = "europe-west1"

  bigtable {
    auto_scaling {
      min_node_count         = 2       # size to real peak QPS, not aspiration
      max_node_count         = 10
      cpu_utilization_target = 60      # 10–80; the node-scaling target
    }
  }
}

# The endpoint is created HERE; MODELS are deployed to it from the pipeline.
resource "google_vertex_ai_endpoint" "fraud" {
  name         = "1470000000"          # numeric endpoint ID, no leading zeros
  display_name = "fraud-scorer"
  location     = "europe-west1"
  region       = "europe-west1"
  network      = "projects/PROJECT_NUMBER/global/networks/ml-vpc"
  # dedicated_endpoint_enabled = true              # isolated DNS, higher limits
  # private_service_connect_config { ... }         # for the PSC-private variant
}

The online store autoscales Bigtable nodes for read QPS — you set the envelope, Vertex scales within it. The endpoint is a container that Terraform creates empty; the pipeline (or your gated deploy job) deploys model versions into it, which is why you will never see a google_vertex_ai_endpoint_*_deployed_model version pinned in this file. For the private serving path the architecture describes, flip on dedicated_endpoint_enabled and add a private_service_connect_config block so application VPCs reach it without touching the public internet.

Enterprise considerations

Security & Zero Trust. Treat every plane as mutually distrusting. Per-plane service accounts with minimal IAM mean a compromised serving container cannot read training data or promote a model. Wrap the data and ML projects in a VPC Service Controls perimeter; serve privately via Private Service Connect; encrypt model artifacts and feature data with CMEK (customer-managed keys in Cloud KMS) where compliance requires control of the key. The registry’s alias mechanism is a Zero-Trust control: nothing reaches production without an explicit, audited alias move, and Cloud Audit Logs record who moved it. For regulated decisions, the per-version model card plus ML Metadata lineage gives you the “why did the model decide this” trail end to end.

Cost optimization. The two cost sinks are idle endpoints and over-eager training. For endpoints, set min replicas honestly — scale-to-low for spiky internal models, but remember online prediction nodes cost while provisioned, so a model that gets one request an hour should be a batch job, not a standing endpoint. Use the 60% utilization autoscaling target as the default and tune per workload. For training, run retrains on Spot machine types and cache successful pipeline steps so a re-run that only changed the eval step does not retrain from scratch. The online Feature Store (Bigtable nodes) is billed for provisioned capacity — size it to real QPS, not aspiration. Batch prediction for anything that can tolerate latency is dramatically cheaper than keeping an endpoint warm.

Scalability. Each plane scales independently. The online store scales by Bigtable node count for read QPS; endpoints scale by replica autoscaling and by sharding traffic across deployed models; training scales by machine type, accelerators, and distributed training with a reduction server. Because features are centralized, onboarding the tenth model is cheaper than the first — it reuses existing features and the same pipeline skeleton. This is the payoff that makes the architecture span small to large orgs: marginal model cost falls as the platform matures.

Reliability & DR (RTO/RPO). Define targets per plane, because they differ:

• Serving is the hot path. Run endpoints with min replicas ≥ 2 across zones; the regional endpoint survives a zonal failure transparently. For regional DR, register the same model version in a second region’s registry and pre-create a standby endpoint — failover is a traffic redirect at your load balancer. RTO: minutes (redirect), RPO: 0 for the model (versions are reproducible artifacts).
• Feature Store online must be backfillable: it is a serving cache of values that live authoritatively in BigQuery. If the online store is lost, re-materialize from the offline store. RPO is bounded by your sync cadence for batch-synced features; stream-written features need a replayable Pub/Sub source.
• Training/registry is not latency-sensitive. Pipelines are reproducible, metadata and registry are managed and regionally durable. A lost pipeline run is simply re-run. RTO: hours is fine.

Observability. Three layers. Infrastructure metrics (endpoint latency, error rate, replica count, utilization) flow to Cloud Monitoring with SLO alerts. Model quality — drift, skew, prediction drift — comes from Vertex AI Model Monitoring against the training baseline, with thresholds per feature and alerts routed to Pub/Sub and on to your incident channel. Lineage from ML Metadata answers the audit questions. The crucial wiring is the drift-alert-to-retraining edge: a monitoring alert publishes to Pub/Sub, which triggers the CT pipeline, so the platform self-heals against gradual data shift rather than waiting for a human to notice degraded business metrics.

Governance. The registry is the policy enforcement point. Enforce that only the gated deploy SA can move the production alias; require a model card and a passing eval before a version is promotable; keep champion/challenger aliases so you always have a tested rollback target. Audit logs over alias moves and IAM give you the compliance story. For fairness and responsible-AI obligations, compute sliced metrics in the eval step and persist them on the model card, so the evidence is attached to the version, not living in a notebook.

Reference enterprise example

Meridian Mutual, a fictional mid-market consumer lender (about 1,800 employees, ~2.4 million active accounts), runs three revenue-critical models: real-time fraud scoring on card-not-present transactions, credit-line propensity for cross-sell, and a batch collections-prioritization model. Before the platform, fraud recall had quietly slipped because the production “transactions in last hour” feature was computed off a 15-minute-stale replica while training used exact values — textbook skew — and nobody could reproduce the live model because it had been trained from an ad-hoc export. A regulator’s question about a declined applicant took the team three weeks to answer.

They rebuilt on this Vertex AI architecture in europe-west1, with ml-dev/ml-staging/ml-prod projects under VPC Service Controls. Decisions and numbers:

• Features: 140 features across the three models, defined once. Real-time fraud features (session transaction count, velocity) are written straight to the online store from a streaming Dataflow job for sub-second freshness; slower features (90-day spend averages) sync from BigQuery hourly. Online store sized at the QPS for ~900 transactions/second peak.
• Training & CT: the fraud pipeline retrains nightly on Spot GPU machines (NVIDIA_L4), ~40 minutes a run, with step caching so eval-only changes skip training. A Model Monitoring skew alert can trigger an off-schedule retrain via Pub/Sub.
• Promotion: nightly runs register a challenger; it is auto-promoted to production only if it beats the incumbent’s holdout AUC by a set margin, otherwise a human reviews. Promotion is an alias move — no redeploy.
• Serving: fraud runs on a dedicated private endpoint over PSC (transaction data never leaves the VPC), min 3 / max 20 replicas at 60% utilization, p99 inference ~28 ms including online feature fetch. Credit-line propensity shares an endpoint with a 90/10 traffic split for A/B. Collections runs as nightly Batch Prediction from BigQuery to BigQuery — no standing endpoint.

Outcome after one quarter: training-serving skew was eliminated, recovering roughly 6 points of fraud recall that the stale-feature bug had been costing. The “why was this applicant declined” answer dropped from three weeks to under an hour by walking ML Metadata from prediction to version to training dataset, with the model card in hand. Total platform run-rate landed near the cost of the single over-provisioned always-on endpoint they had before, because collections moved to batch and retrains moved to Spot. The tenth feature reused by a new model cost effectively nothing to onboard — the marginal-cost curve had bent the right way.

When to use it

Use this architecture when you have — or are about to have — more than one model on a path that matters, served to more than one consumer, maintained by more than one person. The moment those plurals appear, ad-hoc notebooks and hand-copied model files become the bottleneck and the risk, and the centralized feature/registry/monitoring loop pays for itself. It is equally valid at small scale: a two-person team gets reproducibility, skew-free serving, and instant rollback without running any clusters, because every plane is managed.

Trade-offs and anti-patterns to avoid:

• Standing endpoints for batch workloads. A model scored once a day does not need a warm endpoint burning replicas around the clock — use Batch Prediction. Keeping it online is the most common avoidable cost.
• Skipping the Feature Store “to move faster.” Computing features in the serving service is the express lane to training-serving skew and the exact bug this design exists to prevent. The shortcut costs you model accuracy you cannot see.
• Treating the Registry as storage. If pipelines deploy directly and serving points at version numbers, you have lost the governance and the instant rollback. Pipelines register; aliases promote; serving follows aliases.
• Monitoring without a feedback edge. Drift dashboards nobody wires to retraining are decoration. Connect the alert to the CT trigger or you will still find out from a business-metric dip a week late.
• One giant shared endpoint for everything. Multi-model per endpoint is great for A/B and canary within a model family, but co-tenanting unrelated, differently-scaling models couples their failure and scaling domains. Separate them.

Alternatives. If you are all-in on Gemini and generative models rather than classic predictive ML, the gravity shifts toward Model Garden, grounding, and agent tooling, and the feature-store-centric loop matters less. If you have one model and no near-term roadmap for a second, a single Vertex training job plus one endpoint is legitimately enough — adopt the registry and monitoring early but defer the full feature platform until the second model justifies it. And if you are deliberately multi-cloud with portability as a hard requirement, an open KFP-on-GKE plus an open feature store (e.g. Feast) trades managed convenience for portability — a real choice, but you take on the operational weight that Vertex otherwise carries for you. For the common enterprise case — several predictive models, real users, real governance, on Google Cloud — this Vertex AI platform is the path of least regret.

Going deeper

The overview is deliberately clean. Production is where the sharp edges live — the places where a subtly-wrong mental model costs you accuracy, money, or an outage. Here are the ones that separate a platform that looks right in a diagram from one that survives contact with real traffic.

Point-in-time correctness — the silent bug the offline store exists to prevent. When you build a training set, each label has an event time (the fraud happened then; the customer churned then). The features for that row must be the values as they were at that moment — not “now.” A naive JOIN of labels to a current feature table leaks future information into training: your offline AUC looks spectacular and then collapses in production, because at serving time the future features do not exist yet. This is label leakage, and it is the most expensive mistake in feature engineering. The offline Feature Store’s point-in-time lookup (keyed on feature_timestamp) does the time-travel join correctly for you, which is a large part of why “just compute features in the serving service” is a trap — you would have to reimplement point-in-time correctness by hand, and you won’t.

Two pointers, and knowing which one re-routes live traffic. The overview says “repoint the alias and the next requests route there.” That is the intent, but be precise about the mechanism, because conflating the two pointers causes botched rollbacks. The registry alias (production) is the source of truth for “which version is production” — it is metadata. The endpoint traffic-split is what actually decides where a live request goes, and it points at deployed models, not at aliases. Deploying a model by its alias resolves and pins the concrete version at deploy time; the running endpoint does not silently follow later alias moves. So a genuine instant rollback is two pointer moves: shift the endpoint’s traffic-split back to the still-deployed prior version (reroutes live requests immediately, no rebuild), and move the production alias back to match (keeps the registry honest). Both are pointer moves; neither is a redeploy — but they are different pointers, and forgetting the traffic-split one is how “I rolled back” turns into “the bad model is still serving.”

Online-store freshness and the sync race. The offline→online sync has a staleness window equal to its cadence. If you sync hourly, the online store can be up to an hour behind BigQuery — fine for a 90-day spend average, catastrophic for “transactions in the last 5 minutes.” That gap is exactly the skew bug from the reference example. The fix is not a faster sync; it is writing the genuinely real-time features straight to the online store from a streaming Dataflow job, bypassing the batch sync entirely, while slow features ride the periodic sync. Add TTLs on online entities so stale keys expire, and size Bigtable nodes to real peak read QPS — a hot key (one very active merchant) can concentrate load and needs schema attention, the same row-key discipline you would apply to Bigtable anywhere.

Endpoint autoscaling internals and cold behaviour. Online endpoints scale on a utilization target — cpu-usage for CPU serving, gpu-duty-cycle for accelerators — with 60% a sane default. There is no scale-to-zero: min-replica-count is at least 1, and for a hot path you want at least 2 across zones so a zonal blip doesn’t drop you to a single replica. Scale-up is not instant: under a traffic burst, new replicas take time to come up, so p99 latency spikes before capacity catches up — which is why bursty-but-critical models keep a higher floor rather than a min of 1. A dedicated endpoint (public or PSC-private) gives you isolated DNS, higher request-size and timeout limits, and predictable performance versus the shared endpoint; it is the right default for anything latency-sensitive.

Model Monitoring v1 vs v2 — mind the surface. The older monitoring attaches a monitoring job to an endpoint. Model Monitoring v2 introduces a first-class ModelMonitor resource tied to a registry model + version, can monitor models served outside Vertex, and cleanly separates feature drift, prediction (output) drift, and feature-attribution drift — but it is tabular-only today, so it is not your tool for unstructured or generative workloads. Baselines matter: drift is measured against a reference distribution (typically the training dataset, sometimes a recent serving window), and the sampling rate trades monitoring cost against sensitivity. The whole point is the alert path — v2 routes to Cloud Monitoring and Pub/Sub, and that Pub/Sub message is the edge you wire to the CT trigger.

Quotas and limits that bite at scale. The constraint that most often surprises teams is not endpoints per region or deployed models per endpoint — it is regional accelerator quota for custom training. Nightly CT on L4/A100 GPUs across several models contends for a finite regional pool, and a quota ceiling silently serializes or fails your retrains. Request increases early, in the region you actually train in, and treat accelerator quota as a first-class capacity plan. Watch too the concurrent pipeline-run and Batch Prediction limits when you fan out; they are raiseable, but only if you ask before the deadline, not during the incident.

IAM and VPC-SC nuances that make or break Zero-Trust. The promotion gate is not a policy document; it is an IAM fact — the deploy SA is the only principal granted aiplatform.endpoints.deploy and the right to move the production alias, and the pipeline SA is deliberately denied both. Inside a VPC Service Controls perimeter, the pieces that trip people are the ingress/egress rules: Cloud Build (running CI) reaching the Vertex API, the pipeline SA reading BigQuery across the perimeter boundary, and the online-serving path — each needs an explicit rule, or you get opaque PERMISSION_DENIED/403s that look like IAM but are perimeter blocks. Pair this with Workload Identity Federation for keyless CI from your Git provider so there are no long-lived service-account keys to leak, and CMEK on model artifacts and feature data where you must control the key.

When the managed loop is not the answer. Two honest exits. If your roadmap is generative rather than predictive, the centre of gravity moves to grounding, retrieval, and agents — see the enterprise GenAI RAG architecture — and this feature-store loop is not the shape you want. If hard multi-cloud portability is a real requirement (not an aspiration), open KFP-on-GKE plus an open feature store like Feast keeps you portable at the cost of operating the machinery Vertex otherwise runs for you. Choose the managed loop when the models are predictive, the users are real, and Google Cloud is where you live.

Practice challenges

Work these in order — they climb from “can you read the diagram” to “can you design the governance.” Try each before opening the solution.

1. (Beginner) Name the plane. For each symptom, name the plane that removes it: (a) a feature is computed differently in prod than in training; (b) nobody can reproduce the model that is live; © there is no record of who promoted the current model or what it scored; (d) accuracy has quietly decayed with no code change.

   <details><summary>Solution</summary>

   (a) Feature Store — one definition for train and serve kills skew. (b) Pipelines + ML Metadata — the pipeline makes training reproducible and Metadata records the exact data/code/params. © Model Registry aliases + Cloud Audit Logs — promotion is a recorded alias move. (d) Model Monitoring — drift/skew detection against the training baseline. Why: the architecture is literally defined by which failure each plane removes — if you can map symptom→plane, you understand it. </details>

2. (Beginner) Endpoint or batch? Decide online endpoint vs Batch Prediction for: (a) fraud scoring on every card transaction; (b) nightly collections prioritization over the whole book; © an internal risk model hit roughly once an hour by an analyst tool.

   <details><summary>Solution</summary>

   (a) Online endpoint — real-time, request/response, latency-critical. (b) Batch Prediction — bulk, scheduled, no standing endpoint, reads/writes BigQuery. © Batch (or a scheduled batch), not a warm endpoint — one request an hour cannot justify a min-replica-≥-1 endpoint billing 24×7. Why: online endpoints have no scale-to-zero, so rare or bulk traffic is dramatically cheaper batched. </details>

3. (Intermediate) Promote by alias. Write the Vertex AI SDK calls to promote version 8 to production, then to roll the registry back to version 7. State why neither needs a redeploy — and the one extra step a live rollback still requires.

   <details><summary>Solution</summary>

   from google.cloud import aiplatform
   aiplatform.init(project="ml-prod", location="europe-west1")
   m = aiplatform.Model(model_name="MODEL_ID")
   m.versioning_registry.add_version_aliases(new_aliases=["production"], version="8")  # promote
   m.versioning_registry.add_version_aliases(new_aliases=["production"], version="7")  # roll back
   

   Aliases are unique per model, so re-assigning production atomically moves the pointer — no rebuild, no re-registration. The extra step: a live endpoint serves deployed models, not aliases, so an actual rollback also needs a traffic-split shift back to the still-deployed version 7. Why: the registry alias is the source of truth; the endpoint traffic-split is what reroutes requests — two pointers. </details>

4. (Intermediate) Canary at 10%. Write the gcloud command to deploy MODEL_ID onto an existing endpoint taking 10% of traffic while the incumbent keeps 90%, with min 3 / max 20 replicas on n1-standard-4 at a 60% CPU target.

   <details><summary>Solution</summary>

   gcloud ai endpoints deploy-model ENDPOINT_ID \
     --region=europe-west1 --model=MODEL_ID \
     --display-name=fraud-scorer-v8 \
     --machine-type=n1-standard-4 \
     --min-replica-count=3 --max-replica-count=20 \
     --autoscaling-metric-specs=cpu-usage=60 \
     --traffic-split=0=10,OLDER_DEPLOYED_MODEL_ID=90
   

   0 is the temporary ID of the model in this call; get OLDER_DEPLOYED_MODEL_ID from gcloud ai endpoints describe; the split must sum to 100. Why: multi-model-per-endpoint plus traffic-split is how you canary and how you roll back — no rebuild in either direction. </details>

5. (Advanced) Close the loop. Describe the exact chain of services that makes a drift signal trigger a retrain, from detection to a new registered candidate. Name each hop.

   <details><summary>Solution</summary>

   Model Monitoring detects feature drift/skew against the training baseline → publishes an alert to Cloud Monitoring and a Pub/Sub topic → a Pub/Sub push subscription (or Eventarc / a Cloud Function / Cloud Scheduler) triggers a run of the published CT pipeline → the pipeline retrains on fresh data, evaluates, and — past the gate — registers a new challenger version in the Model Registry → gated promotion decides whether it reaches production. Why: monitoring without this feedback edge is a decorative dashboard; the Pub/Sub→CT hop is what makes the platform self-heal. </details>

6. (Advanced) Design the promotion gate. You retrain nightly and produce a challenger every run. Specify a promotion policy that only lets a genuinely-better model reach production, and name the IAM control that stops the pipeline from promoting itself.

   <details><summary>Solution</summary>

   Policy: auto-promote the challenger only if it beats the incumbent’s holdout metric (e.g. AUC) by a fixed margin δ on the same frozen eval set and clears any fairness-slice thresholds; otherwise hold for human review. Keep champion/challenger aliases so a tested rollback target always exists, and require a model card + passing eval before a version is promotable at all. IAM control: a separate gated deploy SA is the sole holder of aiplatform.endpoints.deploy and the right to move the production alias; the pipeline SA can register versions but is denied both. Why: splitting “can register” from “can promote” is the Zero-Trust core of the whole design — the registry becomes a real gate, not a suggestion. </details>

Common beginner mistakes

These are misconceptions, not typos — each one is a wrong mental model that leads a well-intentioned engineer to build the platform slightly wrong.

• “Moving the production alias reroutes live traffic.” It does not on its own. The endpoint serves deployed models and routes by traffic-split; the alias is the registry’s record of what production should be. Right model: two pointers — move the traffic-split to reroute requests, move the alias to keep the registry honest. A rollback that forgets the traffic-split leaves the bad model serving.

• “The Feature Store is just a cache I can bolt on later.” It is the definition boundary that prevents training-serving skew, not a performance cache. Add it after your models have already diverged and you inherit the skew bug plus a migration. Right model: define features once and materialize to both offline and online from the same job, from the start — especially for anything real-time.

• “Reproducibility means I saved the model file.” The file cannot tell you what data, code, or hyperparameters produced it, so it cannot be reproduced or audited. Right model: the pipeline plus ML Metadata capture lineage automatically; “trained on what?” is answered by walking the metadata graph, not by inspecting a .pkl.

• “Continuous training means promote whatever retrains.” CT produces candidates, not blessed models. Auto-promoting every nightly retrain ships regressions straight to production. Right model: CT registers a challenger; a metric threshold or a human moves the alias. Training cadence and promotion decision are deliberately different things.

• “We have a monitoring dashboard, so we’re covered.” A dashboard nobody wired to an action lets drift rot silently until a business metric dips a week later. Right model: alert → Pub/Sub → CT trigger. The value is the feedback edge, not the chart.

• “One big endpoint for all our models is simpler.” Co-tenanting unrelated, differently-scaling models on one endpoint couples their failure and scaling domains — one model’s traffic spike degrades another. Right model: multi-model-per-endpoint is for A/B and canary within a model family; give unrelated models their own endpoints.

• “Terraform should manage the model versions too.” Versions are produced by pipelines many times a day; forcing them through Terraform state fights the tool and loses the registry’s rollback semantics. Right model: Terraform owns the durable platform (stores, endpoints, IAM, network); the Model Registry is the state store for versions.

Glossary

• MLOps — the operational discipline of building, deploying, monitoring, and governing models reliably and repeatably; the “how you operate models,” as distinct from “how you train one.”
• ML platform — the shared set of managed services (here, Vertex AI and neighbours) that implements MLOps so individual teams don’t each rebuild it.
• Vertex AI — Google Cloud’s managed ML platform: training, pipelines, feature store, registry, endpoints, and monitoring under one roof.
• Feature Store — the single place features are defined and served. The offline store (BigQuery) serves point-in-time-correct values for training; the online store (Bigtable) serves the same features at low latency for live prediction.
• Training-serving skew — when a feature is computed differently (or from staler data) at serving time than at training time, silently degrading the model. The Feature Store exists to prevent it.
• Point-in-time correctness / label leakage — building each training row from feature values as they were at the label’s event time. Getting it wrong (joining to “now”) leaks future information — label leakage — and inflates offline metrics that collapse in production.
• Vertex AI Pipelines (KFP) — managed Kubeflow Pipelines; you author a training/eval/register DAG with the KFP SDK and compile it to a portable IR.
• Google Cloud Pipeline Components (GCPC) — prebuilt pipeline steps (train, upload/register, create-endpoint, deploy) imported from google_cloud_pipeline_components.v1.*.
• Vertex AI Training — managed custom or AutoML training jobs on elastic CPU/GPU/TPU, no cluster to run.
• ML Metadata — the lineage graph: which dataset, code, and parameters produced which artifact. The audit backbone.
• Vertex AI Experiments — run tracking so candidate models and their metrics can be compared.
• Model Registry — the versioned catalog of models. The governance choke point: pipelines register, promotion moves an alias, serving follows the alias.
• Model version — one registered iteration of a logical model. Created by ModelUploadOp with a parent_model.
• Alias — a unique, movable pointer (production, champion, challenger, staging) onto a specific version. Promotion and registry-side rollback are alias moves.
• Model card — the per-version document (intended use, training-data summary, eval metrics, fairness slices) generated by the pipeline; your evidence for a regulator or risk committee.
• Endpoint (online prediction) — a managed, autoscaling service that hosts one or more deployed models for request/response inference.
• Deployed model — a concrete model version placed on an endpoint. Traffic-split routes requests across the deployed models on an endpoint.
• Dedicated endpoint — an isolated endpoint (public, or private over PSC) with its own DNS and higher size/timeout limits, versus the shared endpoint.
• Private Service Connect (PSC) — private connectivity so an application VPC reaches the endpoint without traversing the public internet.
• Traffic split / canary — sending a fraction of traffic to a new version (--traffic-split=0=10,OLD=90) to test it safely; rolling back is shifting the split back.
• Batch Prediction — large-scale offline scoring straight from/to BigQuery or GCS, with no standing endpoint. The cheap path for bulk or rare traffic.
• Model Monitoring — detection of feature drift, prediction (output) drift, and training-serving skew against a baseline; v2 is a first-class ModelMonitor resource, tabular-only, that can also watch models served outside Vertex.
• Baseline — the reference distribution drift is measured against, usually the training dataset (sometimes a recent serving window).
• Continuous training (CT) — running the published pipeline on a schedule or a trigger (new data, a drift alert) to produce fresh candidate versions.
• CI/CD (vs CT) — CI/CD ships pipeline code (tests, compile, publish); CT runs the published pipeline to produce model versions. Different cadences, different triggers.
• VPC Service Controls — a data-exfiltration perimeter around BigQuery/GCS/Vertex; crossing it needs explicit ingress/egress rules.
• CMEK — customer-managed encryption keys (in Cloud KMS) for model artifacts and feature data where you must control the key.
• Spot — preemptible, discounted machine types, ideal for cheap retrains that tolerate interruption.
• Reduction server — a Vertex feature that speeds up distributed (multi-node) training by optimizing gradient reduction.
• Autoscaling metric — the target an endpoint scales on: cpu-usage for CPU serving, gpu-duty-cycle for accelerators; 60% is a common default.
• RTO / RPO — Recovery Time Objective (how fast you must recover) and Recovery Point Objective (how much data loss is tolerable); set per plane, since serving, feature store, and training differ.