Real-Time Payments Fraud Scoring Pipeline on GCP

In a nutshell

Every time someone taps or types a card number, the payment network has a sliver of a second to decide approve or decline — and buried in that decision is a quieter question: is this fraud? You cannot pause the checkout to run a slow background investigation while the customer stands at the till; the answer has to come back in well under a tenth of a second or the transaction simply times out and goes through anyway. This lesson is the reference architecture for answering that question, at scale, on Google Cloud.

The whole design turns on one idea: do the slow, expensive homework in advance; keep the live decision featherlight. Picture a nightclub with a bouncer wearing an earpiece. The bouncer cannot run a background check on each person at the door — there’s a queue and only seconds to spare. So a back office continuously keeps a running dossier on every regular (“been here three times tonight already, last seen ten minutes ago across town”) and radios the one-line summary to the earpiece. At the door, the bouncer just listens to the summary, glances at the face, and waves them in or stops them. The back office is the streaming pipeline computing behavioral features; the earpiece is the feature store; the split-second call at the door is the synchronous decision path that must answer inside the latency budget.

On GCP those roles map to concrete services: Pub/Sub absorbs the firehose of card transactions, Dataflow continuously computes “how has this card behaved in the last 90 seconds / 5 minutes / hour,” Bigtable holds those features for a single-digit-millisecond lookup, and Vertex AI (or an embedded model) turns the assembled features into a fraud probability. Everything else in this lesson — exactly-once counting, drift monitoring, fail-open rules, PCI scope — exists to make that loop correct, adaptive, and auditable.

Level: Advanced · Time: ~37 min

Before this lesson, it helps to be comfortable with the building blocks it assembles: streaming ingestion with Pub/Sub exactly-once, stream processing with Dataflow / Apache Beam, the Bigtable schema and row-key model, and the MLOps retraining loop from Vertex AI Pipelines. You do not need payments-industry experience — every unfamiliar term is defined in the Glossary at the end.

After this lesson you’ll be able to:

• Explain why real-time fraud scoring splits into a slow streaming path and a thin synchronous decision path, and why they meet at exactly one place.
• Design a Bigtable feature store whose row key survives thousands of reads per second without hot-spotting.
• Budget a 100 ms p99 latency SLO across the Bigtable read → vector assembly → model prediction → policy chain.
• Keep streaming and training features identical to avoid train/serve skew, and monitor drift once the model is live.
• Choose the right behavior for the hard cases — fail-open-to-rules, idempotent retries, and calibrated thresholds — in a PCI-regulated context.

A mid-tier card network — the kind that sits between a few hundred issuing banks and the merchant acquirers, switching roughly 9,000 authorization messages a second at peak — gets a board-level ultimatum after a bad quarter. Card-not-present fraud losses are up, a single coordinated bot attack on a weekend pushed chargeback ratios past the threshold that triggers brand-scheme fines, and the existing rules engine — a wall of hand-written IF amount > X AND country != home THEN decline rules accreted over fifteen years — is simultaneously catching too little real fraud and declining too many good customers. The head of risk frames it bluntly: “Every authorization that crosses our switch must get a fraud score, the score must come back before we have to answer the issuer, and the model has to learn from last week’s attack — not last year’s.” The hard constraint is the one that makes this an engineering problem and not a data-science notebook: the network’s processing SLA gives the fraud decision a budget of about 100 milliseconds, end to end, at p99, inside an authorization flow that itself has only a few hundred milliseconds before a timeout becomes a forced approval. This article is the reference architecture for building that scoring service properly on Google Cloud — a streaming, low-latency, governed fraud pipeline that a card network’s risk officer, CISO, and scheme auditor will all sign.

The pressures stack the way they always do in payments. Latency is non-negotiable and adversarial: blow the budget and the authorization either times out (a forced approval, which is exactly when the fraudster wins) or declines a legitimate cardholder at the point of sale (a customer-experience and revenue failure). Scale means 9,000 transactions per second sustained with spikes at retail peaks, every one needing a score, with no warm-up. Accuracy under drift means fraud patterns mutate weekly — a model trained on June’s attack is blind to July’s — so the system has to retrain and redeploy without a maintenance window. And regulation means PCI-DSS scope around the cardholder data, model-governance and explainability obligations under emerging AI rules, and an audit trail that proves why any given transaction was declined. A batch job that scores yesterday’s transactions overnight is worthless here; the decision has to happen inside the swipe.

Why not the obvious shortcuts

Three shortcuts will be proposed in the first design meeting, and each fails in a way worth naming so the room can move past them.

Keep the rules engine and just add more rules. Rules are fast and explainable, which is why they survive, but they are static, brittle, and exploitable: an attacker probes the thresholds and walks transactions just under them. Rules cannot weigh a hundred weak signals together the way a model can, and the maintenance burden compounds until no one dares touch a rule for fear of what it silently catches.

Score in batch and cache a risk grade per card. Precomputing a nightly risk score per card and looking it up at authorization time is gloriously fast — but it is blind to the transaction in front of it. The whole signal in card-not-present fraud is velocity and context: five transactions on this card in the last ninety seconds across three countries. A cached overnight grade cannot see the ninety seconds that just happened.

Call a hosted model API synchronously per transaction. Putting a network hop to a generic model endpoint in the authorization path adds tail latency you do not control and a dependency you cannot bound at p99. At 9,000 TPS the per-call overhead and the occasional multi-hundred-millisecond tail will blow the budget on their own, before the model even runs.

The architecture that actually works splits the problem in two: compute the expensive, stateful features continuously on a stream so they are already sitting in a low-latency store before the transaction arrives, and at decision time do only the cheap part — fetch precomputed features, assemble a vector, and call a model that returns in single-digit milliseconds. The streaming pipeline pays the cost of statefulness ahead of time; the synchronous path stays thin enough to fit the budget.

Architecture overview

Trace the diagram as two currents feeding one reservoir: along the top, the streaming path (Pub/Sub → Dataflow → Bigtable) continuously refills the feature store; along the bottom, the decision path (authorization switch → GKE → Bigtable read + Vertex AI → back to the switch, inside 100 ms) drinks from it — and the two meet only at Bigtable. The platform runs two paths that share state but live on completely different latency regimes, and keeping them separate in your head is the first step to operating this well: an asynchronous streaming path that ingests the firehose of authorization events and continuously maintains features, and a synchronous decision path that must return a score inside the 100ms budget. They meet at exactly one place — the Bigtable feature store — which the stream writes to and the decision path reads from.

The defining property of the whole topology is that nothing slow is allowed on the synchronous path. No stream join, no aggregation, no model training, and no cross-region call happens while a transaction is waiting. All of that lives on the streaming side. The decision path does three cheap things — read features, build a vector, predict — and returns.

Synchronous decision path, following the request:

1. The network’s authorization switch holds the in-flight ISO 8583 / ISO 20022 message and makes a low-latency gRPC call to the scoring service — a stateless app on GKE (a regional, private cluster) fronted by an internal load balancer. This call is on the critical path, so it never leaves the VPC and never touches the public internet.
2. The scoring service reads the card’s and merchant’s precomputed features from Bigtable with a single-digit-millisecond point lookup keyed by card token and merchant id. These features were computed seconds ago by the streaming path and are already warm.
3. It assembles the feature vector — the freshly-read velocity and behavioral features plus a handful of request-time fields (amount, MCC, channel) — and calls a Vertex AI online prediction endpoint hosting the fraud model. The endpoint runs in the same region with the model loaded in memory, returning a probability in a few milliseconds.
4. The service applies the decision policy — a calibrated threshold plus a thin layer of non-negotiable hard rules (a known-compromised BIN, a sanctioned geography) that risk insists stay deterministic and explainable — and returns score + decision + reason codes to the switch. The switch approves, declines, or steps up to 3-D Secure.
5. The complete decision record — features used, score, threshold, reason codes — is published to a Pub/Sub outcome topic for the audit trail and for later label-joining, asynchronously, off the critical path.

Asynchronous streaming path, independent and continuous:

1. Every authorization event the switch processes is published to a Pub/Sub ingestion topic — a fully managed, globally durable buffer that absorbs the 9,000 TPS firehose and decouples the volatile switch from the pipeline behind it.
2. A Dataflow streaming job (Apache Beam) consumes the topic and does the stateful heavy lifting: windowed aggregations per card and per merchant (transaction count and amount over sliding 1-minute, 5-minute, and 1-hour windows), velocity features (distinct countries, distinct merchants, time since last transaction), and enrichment joins. Dataflow’s exactly-once processing and watermark-based windowing are what make these counts correct under out-of-order and late events.
3. Dataflow writes the updated feature values to Bigtable continuously, so by the time the next transaction on that card arrives milliseconds or seconds later, its velocity features already reflect the one that just happened. This write-ahead-of-read is the entire trick.
4. In parallel, raw enriched events land in BigQuery as the historical store for offline training, analytics, and the eventual fraud labels (chargebacks, confirmed-fraud reports) that arrive days later and become training targets.

Training & retraining loop, on a slower cadence still: Vertex AI Pipelines orchestrate scheduled retraining — pull labeled history from BigQuery, recompute training features with the same logic the stream uses, train and evaluate the model, register it in the Vertex AI Model Registry, and roll it out behind a traffic split. The streaming features and the training features must be computed identically, or the model sees one distribution in training and another in production — the train/serve skew that quietly destroys fraud models.

Component breakdown

A few of these choices deserve the why, because they are the ones teams get wrong.

Why Bigtable as the feature store, not a relational cache or a generic key-value store. The decision path’s read is the most latency-sensitive operation in the whole system, and it happens 9,000 times a second. Bigtable gives single-digit-millisecond point reads at that throughput with a flat latency profile that holds as data grows, because the row key is designed for exactly this lookup — cardToken#reversedTimestamp style keys put a card’s hot features on a predictable, well-distributed row. A relational store would add query-planner variance and connection-pool contention you cannot bound at p99; a smaller cache would not hold the full card population. The discipline that matters: design the row key around the read pattern, and split features into column families so the decision path reads only the groups it needs.

Why Dataflow for the features, not a microservice doing its own counting. The hard part of velocity features is correctness under disorder — events arrive late, out of order, and occasionally twice, and a naive counter double-counts or misses, producing features that lie to the model. Dataflow’s Beam model gives you watermarks (a principled notion of “how late is too late”), windowing (sliding windows for “last 5 minutes” that update as time advances), and exactly-once state, so the count of “transactions on this card in the last minute” is actually correct. Rebuilding that correctly in application code is a project unto itself, and getting it subtly wrong is how fraud models silently degrade.

Why the feature logic must be shared between stream and training. This is the single most common and most damaging mistake in real-time ML. If the streaming job computes “distinct countries in the last hour” one way and the training job computes it another way from BigQuery history, the model is trained on a distribution it never sees in production — train/serve skew — and its real-world accuracy collapses while offline metrics look fine. The fix is to factor the feature transformations into a shared library invoked by both the Beam pipeline and the Vertex AI Pipelines training step, and to validate parity continuously.

Implementation guidance

Provision with Terraform, and treat the VPC and private connectivity as the first deliverable. Everything on the decision path — GKE, Bigtable, the Vertex AI endpoint — must reach each other over private networking with no public egress, or you have both a latency tax and a PCI scope you do not want.

1. A VPC with subnets for GKE, a Private Service Connect / private-services range for managed services, and Private Google Access so the cluster reaches Bigtable and Vertex AI without traversing the internet.
2. Bigtable provisioned with an SSD cluster sized to the read QPS (not just storage), with autoscaling on node count.
3. The regional private GKE cluster with Workload Identity enabled and an internal load balancer for the switch’s gRPC call.
4. The Vertex AI endpoint deployed in the same region as GKE and Bigtable — cross-region hops are latency you cannot afford.
5. Pub/Sub topics (ingestion + outcome + dead-letter) and the Dataflow streaming job with the streaming engine and autoscaling enabled.

A minimal Terraform shape for the Bigtable feature store communicates the intent — SSD for latency, autoscaling for the firehose:

resource "google_bigtable_instance" "feature_store" {
  name = "fraud-feature-store-prod"

  cluster {
    cluster_id   = "fraud-fs-prod-c1"
    storage_type = "SSD"            # SSD, not HDD — single-digit-ms reads
    zone         = "asia-south1-a"
    autoscaling_config {
      min_nodes      = 6
      max_nodes      = 30
      cpu_target     = 60           # scale ahead of the read firehose
    }
  }
  deletion_protection = true
}

The pipeline that applies this runs in GitHub Actions, authenticating to GCP via Workload Identity Federation (OIDC) so there is no long-lived service-account key sitting in a secret to leak — a hard rule for anything that touches a payments environment. The same pipeline runs the model evaluation harness (below) as a required gate before any promotion.

Identity: federate the humans, kill the static keys. Analysts, data scientists, and on-call engineers reach BigQuery, the Vertex dashboards, and Datadog through SSO: the workforce IdP is Okta, federated (for the shops that also run Microsoft estates, via Entra ID) into Google Cloud Identity over OIDC, with Okta groups mapped to GCP IAM roles so a data scientist gets BigQuery read on the analytics dataset but never on the PAN-adjacent columns, and only the SRE group can touch the production endpoint. Conditional-access and adaptive MFA live in Okta. Workloads authenticate with Workload Identity — the GKE scoring service and the Dataflow workers assume scoped service accounts (Bigtable read, Vertex predict, Pub/Sub publish) with no key files anywhere. The handful of residual secrets that are not service identities — issuer-callback API tokens, a third-party device-fingerprint feed credential, a signing key for the outcome records — live in HashiCorp Vault, leased dynamically and injected by the Vault Agent sidecar, so they are short-lived and never written to a Kubernetes Secret or a container image.

Feature and serving wiring. Define the feature schema once and share it: column families in Bigtable grouped so the decision path reads only what it needs (velocity, behavior, merchant), the same field definitions emitted by the Beam pipeline, and the identical transformation library imported by the training pipeline. Serve the model on a Vertex AI dedicated endpoint sized so the model stays resident in memory (cold starts are fatal at p99), and roll new models out behind a traffic split — 5% to the challenger, watch the metrics, then ramp — rather than a hard cutover. Keep the threshold and the deterministic hard rules in version-controlled config, reviewable and instantly revertable, because the threshold is a business lever (the fraud-catch versus false-decline tradeoff) that risk will want to tune without a redeploy.

Enterprise considerations

Security, PCI scope, and Zero Trust. Payments raises the bar past ordinary cloud security. (a) Tokenize the PAN before it ever reaches this pipeline — the scoring service operates on a card token, not the primary account number, which keeps the bulk of this system out of PCI-DSS cardholder-data scope; only the tokenization boundary and the narrow stores that touch PAN-adjacent data stay in scope. (b) Everything on the decision path is private-networking only, identity-based access, least-privilege service accounts per workload — Zero Trust by construction. © Wiz runs continuous CSPM and sensitive-data scanning across Bigtable, BigQuery, and Cloud Storage, alerting the moment a dataset drifts toward public exposure or an IAM binding widens access to PAN-adjacent fields — the posture backstop behind the policy controls. (d) CrowdStrike Falcon sensors on the GKE node pools and Dataflow workers give runtime threat detection, feeding the network’s SOC. (e) Organization policy denies public IPs on the data-plane resources, and Wiz independently verifies the policy is actually holding. (f) A material control breach — a public-exposure drift, a sustained guardrail failure — auto-raises a ServiceNow incident so security has a ticket, not just a log line.

Cost optimization. Two cost centers dominate — the always-on streaming/serving footprint and the data volume — and both reward engineering.

Tag and label every resource by environment and cost center, pipe spend metrics to Datadog, and let the FinOps team see fraud-platform cost per million transactions scored — the unit economic the CFO actually asks about.

Scalability. Each tier scales independently and the whole point of the split is that nothing slow blocks the firehose. Pub/Sub absorbs spikes natively — it is a buffer, so a switch surge becomes backlog, not backpressure on the source. Dataflow autoscales workers on the backlog and watermark lag. Bigtable scales out on node count (read QPS) with the row-key design ensuring the load spreads rather than hot-spotting one tablet — the classic Bigtable failure is a sequential row key that funnels all writes to one node. The GKE scoring service scales pods on requests-per-second behind the internal LB, and the Vertex endpoint scales replicas on QPS with a warm floor so a scale-up never cold-starts into the latency budget. The natural ceiling is regional capacity, which is why a network at this volume reviews quotas and plans a second region early.

Failure modes, and what each one looks like. Name them before they page you.

• Stale features — Dataflow falls behind and the velocity features the model reads are minutes old, so the model is blind to the attack happening now. Mitigation: a hard Datadog SLO on pipeline lag (watermark age) with a page well before features go stale, and Dataflow autoscaling headroom.
• Vertex endpoint slow or cold — a model scaled to zero replicas, or a cold node, adds hundreds of milliseconds and blows the budget. Mitigation: a warm replica floor, dedicated endpoints, and a fail-open-to-rules fallback so a model timeout degrades to the deterministic rule set rather than forcing a blind approval.
• Bigtable hot-spotting — a poorly chosen row key funnels reads or writes to one node and p99 latency spikes. Mitigation: salt/reverse the row key, validate the distribution under load, and alert on per-node CPU skew.
• Train/serve skew — features computed differently in training and serving silently tank real-world accuracy while offline metrics look healthy. Mitigation: a shared feature library and a continuous parity check between stream-computed and batch-computed features.
• Model drift — last month’s model is blind to this month’s fraud pattern; precision/recall quietly erode. Mitigation: drift monitors in Datadog on score distribution and feature distributions, and an automated retrain cadence with a champion/challenger gate.
• Regional outage — see DR below.

Reliability & DR (RTO/RPO). Decide the numbers per tier and pick them around the business reality that the scoring service going dark is itself a fraud event — you must keep deciding. The fallback is the safety net: if Vertex or Bigtable is unreachable, the scoring service fails open to the deterministic hard-rule set and flags those transactions for offline review, so authorizations keep flowing with degraded protection rather than timing out into forced approvals. For genuine regional loss, run the decision path active in a second region with Bigtable replication keeping the feature store warm and the model deployed in both regions; the switch routes to the healthy region. BigQuery and the durable Pub/Sub history are the rebuild source of truth. A pragmatic target: RTO under 5 minutes to the second region with fail-open rules covering the gap, and RPO near zero for the decision audit trail (every outcome is published durably to Pub/Sub before the service returns).

Observability and SLOs. This system lives or dies on latency, so the decision-latency SLO is the headline metric: instrument the decision span end to end in Datadog with OpenTelemetry — one trace covering Bigtable read → vector assembly → Vertex predict → policy → return — with a hard p99 < 100ms objective and an error budget that pages before it is exhausted, not after. Beyond latency, monitor the metrics the risk team actually cares about: pipeline/watermark lag, feature freshness, score-distribution drift, fraud-catch rate and false-decline rate (the business tradeoff), fallback-activation rate (how often you degraded to rules), and Pub/Sub backlog. Run an offline evaluation harness in the GitHub Actions pipeline so every candidate model is scored on precision/recall at the operating threshold before it can be promoted, and every model promotion passes a ServiceNow change gate so risk and audit have a documented, reversible record of which model decided which transactions.

Governance and explainability. Payments regulators and the card schemes increasingly require that an automated decline be explainable. Keep reason codes on every decision — the top contributing features — and persist them with the outcome record so a disputed decline can be reconstructed. Pin model versions explicitly in the registry (never a floating “latest”), promote through the eval and ServiceNow gates, and keep the decision policy and thresholds in version control. Log every scored decision — features, score, model version, reason codes — durably to the outcome topic and BigQuery for audit, dispute resolution, and as future training data, under the retention the scheme rules require.

Explicit tradeoffs

Accept these or do not build it. The streaming-feature architecture is genuinely more complex than a rules engine: you now operate a Pub/Sub firehose, a stateful Dataflow job whose correctness depends on watermarks you must understand, a feature store whose row-key design you must get right, and a retraining loop. The 100ms budget forces real discipline — everything slow must be precomputed, every component must be in-region, and the model must stay warm — and the price of getting it wrong is not a slow page, it is a forced approval or a declined customer at checkout. A model is also less transparent than a rule; you buy accuracy and adaptability at the cost of explainability, which is why the reason-code and hard-rule layer is not optional in a regulated payments context. And the platform’s standing cost — the always-on streaming and serving footprint, the second region, the Wiz/CrowdStrike/Datadog tooling — is overhead you cannot amortize away at low volume; this design earns its keep at a card network’s scale and would be over-engineered for a single merchant’s checkout.

The alternatives, and when they win. If your volume is modest and latency is forgiving, a synchronous feature computation (read recent transactions and aggregate them at decision time) skips the streaming pipeline and is far simpler — it just will not hold at 9,000 TPS inside 100ms. If you need maximum explainability and can accept lower accuracy, a modern rules-plus-gradient-boosted-model hybrid scored mostly on request-time features keeps the architecture lighter. If you would rather not operate the ML lifecycle at all, a managed fraud-detection service (a turnkey fraud API) trades control and customizability for speed-to-launch — reasonable for a smaller issuer, but a card network needs the control to model its own scheme-specific fraud and to own the latency budget. And if your fraud is dominated by account-level patterns rather than per-transaction velocity, a graph-based approach (entity-resolution and link analysis over the BigQuery history) complements this pipeline rather than replacing it — run it on the streaming side and feed its signals in as features.

Going deeper

The core architecture above is the what. This section is the how it actually behaves under load — the internals, edge cases, and numbers an engineer needs before taking this to production. If the sections above are the blueprint a risk officer signs, these are the details the on-call engineer lives with.

Spending the 100 milliseconds: a latency budget you can defend. “Sub-100ms at p99” is not a wish; it is an accounting exercise where every stage gets an allotment and the sum, plus headroom, must fit. You budget the tail, not the mean — a p50 of 15 ms is irrelevant if the p99 is 140 ms, because the tail is exactly when a fraudster’s burst arrives. A representative allocation for one scoring call:

The headroom line is the one juniors delete and seniors defend: it absorbs a stop-the-world garbage-collection pause, a momentary Bigtable node blip, a TCP retransmit. When measured Vertex p99 creeps from 30 ms toward 60 ms, you have not “still got room” — you have eaten the buffer that keeps the SLO honest, and it is time to act (scale the endpoint, embed the model, shrink the model).

Embed the model, or call the endpoint? The network hop is the decision. A call to a Vertex AI online endpoint, even in-region and private, still crosses the pod boundary and the network — a few milliseconds plus tail risk you do not fully control. For the tightest budgets, the fraud workhorse — a gradient-boosted tree (XGBoost / LightGBM) — is small enough to load in-process inside the GKE scoring service, turning “predict” from a network round-trip into a microsecond function call. You buy back the last few milliseconds and remove a dependency from the critical path. What you give up is exactly what Vertex gives you for free: managed rollouts, traffic-split canaries, centralized model monitoring, and updating the model without shipping a new container. The middle ground is a co-located sidecar — the model server runs in the same pod, so it’s a localhost call, not a cross-node hop. A rule of thumb: embed the tabular booster when microseconds matter and you can tolerate container-based model updates; keep the managed Vertex endpoint when you want governed A/B rollout and centralized drift monitoring, and pay for endpoint headroom instead. Whatever you choose, deploy the endpoint with a warm replica floor so a scale-up never cold-starts into the budget:

# Warm floor on a dedicated, in-region endpoint — the model stays resident,
# so a scale event never cold-starts into the 100 ms budget.
gcloud ai endpoints deploy-model "$ENDPOINT_ID" \
  --region=asia-south1 \
  --model="$MODEL_ID" \
  --display-name=fraud-scorer-v7 \
  --machine-type=n1-standard-4 \
  --min-replica-count=3 \      # never scale to zero on the auth path
  --max-replica-count=20

Roll a challenger onto the same endpoint and shift a slice of traffic to it (start at 5%) with a traffic split, ramping only as the metrics hold — a canary, not a cutover.

Bigtable, Memorystore, or both? Feature-store latency tiers. Bigtable is the right default for the feature store: single-digit-millisecond reads, holds the entire card population durably, autoscales on QPS. Memorystore (Redis) is a different tier — sub-millisecond, but volatile, capacity-bounded (you pay to keep it all in RAM), and eviction is your problem. When the latency waterfall proves you need the last few milliseconds, the pattern is Memorystore as an L1 cache for the hottest tokens in front of Bigtable as L2: most reads hit sub-millisecond Redis, misses fall through to Bigtable and populate the cache. The catch is consistency — the streaming path must now update or invalidate Redis too, or the decision path reads stale velocity from a warm-but-wrong cache. Most networks skip L1 entirely because Bigtable already fits the budget; add Memorystore only when you have measured that you must, not because sub-millisecond sounds better than single-digit. (The Bigtable and Memorystore deep-dives in this course cover each store’s internals in detail.)

A worked example: computing one velocity feature. The “transactions on this card in the last 5 minutes” feature is a sliding-window count — trivial to describe, treacherous to get right under late and duplicate events, which is exactly why it lives in Dataflow and not in application code. In Apache Beam it is roughly:

# Representative: sliding 5-minute velocity count per card token (Apache Beam)
import apache_beam as beam
from apache_beam.transforms import window

(events
 | "KeyByCard"      >> beam.Map(lambda e: (e["card_token"], e))
 | "SlidingWindow"  >> beam.WindowInto(
        window.SlidingWindows(size=5 * 60, period=60))   # 5-min window, new pane each minute
 | "CountPerCard"   >> beam.combiners.Count.PerKey()
 | "UpsertBigtable" >> WriteVelocityToBigtable())         # write the `velocity` family on the card row

The SlidingWindows(size, period) is what makes “last 5 minutes” mean the trailing 5 minutes, recomputed every minute rather than fixed clock buckets. Beam’s watermark decides when a window’s result is final versus still awaiting late data, and its exactly-once state guarantees the count reflects each event once. Rebuilding this correctly in a microservice — handling the event that arrives twice, or four minutes late — is a project unto itself, and getting it subtly wrong is how a fraud model is quietly fed lies.

Exactly-once, idempotency, and the retried authorization. There are two different exactly-once problems here, and conflating them causes real bugs. On the streaming side, Dataflow provides exactly-once processing of each message into feature state, so a Pub/Sub redelivery does not double-count a transaction in the velocity window; pairing it with Pub/Sub exactly-once delivery on the subscription makes the counts trustworthy end to end (the mechanics are in the Pub/Sub exactly-once lesson linked above). On the decision side there is a subtler trap: the authorization switch may retry the same transaction after a timeout or network blip. If the pipeline treats that retry as a brand-new event, it inflates the card’s velocity and may decline the legitimate retry it just caused. The fix is an idempotency key carried end to end — the ISO message’s STAN or RRN, or a network transaction id: the ingestion side dedupes on it before the velocity counters see it, and the decision side returns the same cached score for the same key within a short window. Idempotency is what makes “exactly-once” true from the model’s point of view, not merely the pipeline’s — one economic event, one count, one score.

The read-your-writes race: feature freshness has a floor. The write-ahead-of-read trick assumes the previous transaction’s features are already in Bigtable when the next read happens — but there is a propagation window (event published → Dataflow processes → Bigtable commit) of tens to a few hundred milliseconds. Two transactions on the same card inside that window, and the second reads features that do not yet reflect the first. You cannot drive this to zero. Three honest responses: (a) accept it — a 1/5/60-minute velocity barely moves for one missing event; (b) for the sharpest signal, rapid-fire same-card, keep a tiny same-second counter in a very-low-latency store (Memorystore) that the decision path updates synchronously; © monitor watermark lag so the window stays small and bounded. Naming this race is what separates an engineer who has operated a feature store from one who has only drawn one on a whiteboard.

Calibration and the threshold as a cost matrix. A model emits a raw score; whether 0.80 actually means “80 out of 100 such transactions are fraud” requires calibration (Platt scaling or isotonic regression). Skip it and your threshold is meaningless, because the number it compares against has no probabilistic footing. The threshold itself is not a magic constant — it is the point on the precision/recall curve where a cost matrix balances: the cost of a missed fraud (chargeback loss plus scheme-fine exposure) against the cost of a false decline (lost interchange plus a churned customer at the till). Risk owns that lever and tunes it, which is why it lives in version-controlled config, is instantly revertable, and can differ per segment (a riskier MCC or a card-not-present channel carries a tighter threshold). The step-up to 3-D Secure is the third door — it turns a binary approve/decline into a graded response for the ambiguous middle.

Drift, measured — PSI, not vibes. “The model is drifting” must be a number before it is an alert. Distinguish three drifts: feature drift (an input distribution shifts — a new merchant category floods in), score drift (the output distribution moves), and the dangerous one, concept drift (the relationship between features and fraud itself changes, visible only once labels arrive). Measure input and score drift with Population Stability Index (PSI) or KL-divergence against a training-window baseline; a PSI above roughly 0.2 is the conventional “investigate” line. Because true fraud labels lag by days — chargebacks and confirmed-fraud reports trickle in long after the decision — you watch leading indicators continuously (drift, fallback-activation rate, precision on the fast-confirmed subset) and reserve full precision/recall recomputation for when labels mature. The retrain trigger is drift plus schedule, and every candidate passes the offline eval gate before promotion — the loop the Vertex AI Pipelines lesson above builds out.

Quotas, limits, and API-surface caveats. The failure that pages you during your biggest sale is usually a quota, not a bug. Vertex AI online prediction has per-region, per-model QPS limits — request the increase before a peak event and load-test to peak-plus-headroom, not to average. Bigtable autoscaling reacts to CPU and QPS but ramps over minutes; pre-scale (raise min_nodes) ahead of a known retail peak rather than trusting reactive scaling to catch a vertical spike. Pub/Sub throughput is effectively elastic, but watch subscription backlog and streaming-pull quotas. On the API surface: prefer Vertex AI dedicated endpoints (Private Service Connect) for private, in-VPC, predictable-latency serving, and reach them through the google-cloud-aiplatform SDK or gcloud ai endpoints, not the legacy “AI Platform Prediction” surface. And in Terraform, Bigtable’s autoscaling_config block (shown earlier) is mutually exclusive with a fixed num_nodes — set one or the other, never both.

The shape of the win

For the card network’s risk desk, the payoff is not “an ML model.” It is that a card-not-present transaction crosses the switch, gets a fraud score grounded in the velocity of the last ninety seconds with reason codes attached, comes back inside the 100ms budget so the issuer is answered on time, and — because the pipeline retrains weekly and degrades safely to deterministic rules when a component falters — fraud losses fall and false declines fall and the scheme auditor can reconstruct any decision. That combination is the one that ends the board-level pressure. Everything upstream — the Pub/Sub firehose, the watermark-correct Dataflow features, the single-digit-millisecond Bigtable reads, the warm in-region Vertex endpoint, the Okta-federated access, the Vault-held secrets, the Wiz posture scanning, the Datadog latency SLO — exists to make a risk officer, a CISO, and a scheme auditor each say yes. The architecture here is the destination; start narrower if you must, but this is where real-time fraud scoring at a card network’s scale has to land.

Practice challenges

Work these in order — the first two build intuition, the middle two are design decisions, and the last two are the kind of question an interviewer or a live incident asks. Try each before opening the solution.

1. Beginner — trace the two paths. From the architecture diagram, list which components sit on the synchronous decision path and which sit on the asynchronous streaming path, and name the single component both touch.

<details> <summary><strong>Solution</strong></summary>

Decision path: authorization switch → GKE scoring service → Bigtable (read) → Vertex AI endpoint → decision back to the switch (and the outcome published to Pub/Sub off the critical path). Streaming path: Pub/Sub ingestion → Dataflow → Bigtable (write) and BigQuery. Shared: the Bigtable feature store.

Why: the entire design rests on keeping slow work off the decision path and letting the two paths meet only at the feature store.

</details>

2. Beginner — design a row key that won’t hot-spot. A colleague proposes keying the Bigtable feature rows as timestamp#cardToken. Explain the problem and give a better key.

<details> <summary><strong>Solution</strong></summary>

timestamp#… increases monotonically, so every recent write lands in the same row-key range and funnels to one node — a hot-spot that spikes p99. Lead with the high-cardinality field instead: cardToken (optionally with a hashed/salted prefix to spread further); if you need per-card time ordering, append a reversed timestamp: cardToken#<reversedTs>.

Why: Bigtable distributes load by row-key range, so the key must lead with a well-distributed, high-cardinality field — the card, not the clock.

</details>

3. Intermediate — budget the latency. Your p99 target is 100 ms. You measure: Bigtable read 10 ms, vector assembly 3 ms, policy 2 ms, and serialize + deserialize 8 ms total. How much p99 can Vertex predict take if you insist on 40 ms of headroom? What do you do if the measured predict p99 is 55 ms?

<details> <summary><strong>Solution</strong></summary>

100 − (10 + 3 + 2 + 8) − 40 = 37 ms for predict. At a measured 55 ms you are 18 ms over and eating the headroom: scale the endpoint or raise the warm-replica floor, move to a faster machine type / optimized runtime, co-locate or embed the model to kill the network hop, or shrink the model.

Why: you engineer to the tail and the headroom absorbs GC/jitter — spending it is a signal to act, not spare capacity to enjoy.

</details>

4. Intermediate — make a retry idempotent. The switch retries an authorization with the same RRN after a timeout. Sketch how the streaming and decision sides avoid both double-counting the velocity and returning a divergent second score.

<details> <summary><strong>Solution</strong></summary>

Carry the RRN (or STAN / network transaction id) as an idempotency key end to end. Streaming: dedupe on the key before the velocity counters (a short-TTL seen-set or a Dataflow dedupe on the key) so the retry doesn’t increment counts. Decision: cache the score by idempotency key for a short window and return the same score + decision for a repeat key.

Why: a retried authorization is one economic event; exactly-once has to hold from the model’s point of view, not just the pipeline’s.

</details>

5. Advanced — prove feature parity. You suspect the streaming distinct_countries_1h differs from the BigQuery training computation. Design a continuous check that would catch train/serve skew.

<details> <summary><strong>Solution</strong></summary>

Factor the transform into one shared library imported by both the Beam pipeline and the Vertex AI training step. Then log stream-computed features to BigQuery and run a scheduled parity job: for a sample of (card, timestamp), recompute the feature from raw BigQuery events and compare it to what the stream wrote at that time; alert when the mismatch rate crosses a threshold and block promotion on a parity SLA in the eval gate.

Why: healthy offline metrics with collapsing production accuracy is the skew signature — you must measure parity, never assume it.

</details>

6. Advanced — fail open, safely. Vertex is unreachable for 90 seconds during a spike. Specify the exact fallback behavior, what you record, how you detect and recover — and why fail-open is right here.

<details> <summary><strong>Solution</strong></summary>

On a short deadline / circuit-breaker trip, the scoring service fails open to the deterministic hard-rule set (known-bad BIN, sanctioned geography, hard velocity ceilings), returns a decision within budget, and tags every such transaction fallback=true, publishing it to the outcome topic for offline review. Datadog alerts on fallback-activation and endpoint-error rate; recovery is automatic when endpoint health returns, protected by a warm replica floor. Fail-open because a scoring outage that blocks the path becomes a forced timeout → forced approval (fraud wins) or mass false declines (customer harm); degrading to rules keeps authorizations flowing with reduced protection.

Why: in payments the scoring service going dark is itself a fraud event — you must keep deciding.

</details>

Common beginner mistakes

“The ML model is the system.” The instinct is that the model is the hard, valuable part and everything else is plumbing. In this architecture the model runs in a handful of milliseconds; the genuinely hard parts are feature freshness and the latency budget. The pipeline exists to hand the model warm, correct features inside 100 ms — the right mental model is “a feature-delivery system that happens to end in a prediction,” not “a model with some inputs.”

“Real-time scoring means computing everything at request time.” “Real-time” tempts beginners to do the aggregation synchronously inside the request. Computing velocity windows over the full card population at 9,000 TPS inside 100 ms will not hold. The stream pays that cost ahead of time and writes the answer into the feature store; the request just reads it. Write-ahead-of-read is the whole trick.

“A time-sorted row key is efficient — it’s already ordered.” Sorting by timestamp sounds tidy, but a monotonically increasing key funnels every recent write to one Bigtable node — a hot-spot that spikes p99 exactly when volume is highest. Lead the key with a high-cardinality, well-distributed field (the card token) and reverse timestamps only for per-card ordering.

“Offline accuracy is high, so we’re done.” A strong AUC in the notebook feels like victory, but train/serve skew and drift routinely let offline metrics look healthy while production accuracy quietly collapses. Enforce feature parity between stream and training, monitor drift with a real metric (PSI), and gate every promotion on an eval harness scored at the operating threshold.

“If the model service fails, decline everything — fail-closed is safer.” Fail-closed is the security reflex, and it is wrong on the auth path: a hard failure becomes a timeout (a forced approval) or a wall of false declines that harms customers and revenue. Fail open to deterministic rules, flag those transactions for review, and keep deciding — the scoring service going dark is itself a fraud event.

“A 0.9 score means 90% fraud, so just pick a threshold.” Raw model outputs are not calibrated probabilities, and the threshold is not an arbitrary default. Calibrate (Platt / isotonic) so the number means what it says, then set the threshold from the cost matrix (missed-fraud cost versus false-decline cost) — a versioned, tunable business lever, not a constant.

“Tokenization is someone else’s problem.” Letting the raw PAN flow into the pipeline “because it’s encrypted” drags the entire system into PCI-DSS cardholder-data scope. Tokenize before ingestion so the pipeline only ever sees a token — keeping the bulk of the platform out of scope is a design decision made at the boundary, not a checkbox afterwards.

Glossary

Authorization (auth) — the real-time request-and-response when a card is used, asking the issuer to approve or decline; the fraud score must return inside its time budget. Issuer / acquirer — the cardholder’s bank (issuer) and the merchant’s bank (acquirer); the card network switches messages between them. Card network / switch — the network in the middle (and its message-routing “switch”) that carries authorizations; this pipeline scores every message crossing it. ISO 8583 / ISO 20022 — the messaging standards for card authorizations; the switch holds one of these in flight while it waits for the score. Card-not-present (CNP) — a transaction where the card isn’t physically read (online / phone); the dominant fraud surface and the reason velocity and context features matter. PAN — Primary Account Number, the 16-ish-digit card number; kept out of the pipeline via tokenization to limit PCI scope. Tokenization — replacing the PAN with a non-sensitive token before ingestion, so the pipeline operates on the token and stays out of cardholder-data scope. BIN — Bank Identification Number, the leading digits identifying the issuer; used in hard rules (e.g., a known-compromised BIN). MCC — Merchant Category Code, classifying the merchant’s business; a common request-time feature and a segment for per-segment thresholds. STAN / RRN — System Trace Audit Number / Retrieval Reference Number: identifiers on the auth message usable as an idempotency key for retries. Chargeback — a cardholder’s later dispute that reverses a transaction; confirmed chargebacks become the fraud labels for training, and they arrive days late. Interchange — the fee flow on a transaction; a false decline forfeits it, which is why false declines carry a real cost in the threshold’s cost matrix. 3-D Secure / step-up — an extra cardholder-verification challenge; the graded third option between approve and decline for ambiguous transactions. p99 latency — the 99th-percentile response time; the metric this system is engineered to, because the tail is when attacks land. Latency budget — the per-stage allocation of the p99 target (here ~100 ms) plus reserved headroom; the discipline that keeps the SLO honest. Feature — a computed input to the model (e.g., “distinct countries in the last hour”); the pipeline’s job is to have these ready before the transaction arrives. Feature store — the low-latency store (Bigtable, optionally fronted by Memorystore) the stream writes and the decision path reads; the one place the two paths meet. Velocity features — counts and rates over recent time windows (transactions per minute, distinct merchants) that capture the context static rules miss. Windowed aggregation — computing a feature over a moving time window (e.g., a sliding 5-minute count); done in Dataflow / Beam for correctness under disorder. Watermark — Beam’s principled notion of “how late is too late,” deciding when a window’s result is final versus still awaiting late events. Exactly-once — processing each event’s effect on state exactly one time despite retries and redeliveries, so velocity counts don’t double-count. Idempotency — designing so a repeated request (a retried auth) has the same effect as one; enforced with an idempotency key carried end to end. Train/serve skew — when features are computed differently in training and serving, so the model meets a distribution in production it never trained on; the quiet accuracy killer. Drift — a shift over time in inputs (feature drift), outputs (score drift), or the feature-to-fraud relationship (concept drift); monitored to trigger retraining. PSI (Population Stability Index) — a number quantifying how much a distribution has shifted from a baseline; PSI above ~0.2 conventionally means “investigate.” Calibration — adjusting raw model scores (Platt / isotonic) so a 0.8 really means an 80% probability, which is what makes the threshold meaningful. Threshold — the score cut-off for decline; not a constant but the point where the cost of missed fraud balances the cost of false declines, tuned per segment. Reason codes — the top contributing features behind a decision, persisted with the outcome so a disputed decline is explainable and auditable. Fail-open (to rules) — degrading to the deterministic hard-rule set when the model or feature store is unreachable, so authorizations keep flowing with reduced protection. Champion/challenger — running the live model (champion) against a candidate (challenger) on a traffic split before promoting; a canary for models. Workload Identity — GKE / Dataflow workloads assuming scoped service accounts with no key files, so nothing on the path holds a long-lived credential. PCI-DSS — the card-industry security standard governing cardholder data; tokenization and private networking keep most of this pipeline out of its scope. Bigtable — GCP’s wide-column NoSQL store; the feature store here, chosen for flat single-digit-ms reads at high QPS with a read-pattern-shaped row key. Memorystore — GCP’s managed Redis / Memcached; an optional sub-millisecond L1 cache in front of Bigtable when the latency waterfall demands it. Dataflow (Apache Beam) — GCP’s managed stream / batch processing; computes the windowed velocity features with watermarks and exactly-once state. Pub/Sub — GCP’s managed messaging; the durable firehose that ingests every authorization and decouples the switch from the pipeline. Vertex AI online prediction — GCP’s managed low-latency model serving; hosts the fraud model on an in-region dedicated endpoint (or is replaced by an embedded model). Vertex AI Pipelines / Model Registry — the MLOps orchestration and versioned model store driving scheduled retrain, evaluation, and governed rollout. BigQuery — GCP’s serverless data warehouse; the historical store for training data, analytics, and late-arriving labels. GKE — Google Kubernetes Engine; runs the stateless scoring service on a regional private cluster behind an internal load balancer. Private Google Access / Private Service Connect — private-networking mechanisms letting the cluster reach Bigtable and Vertex without traversing the internet.