GCP Enterprise Architecture: Data Mesh

In a nutshell

A data mesh changes who owns the data, not just where it lives. In the old model one central data team ingests, models, and serves every other team’s data — and quickly becomes the bottleneck everyone waits on. A data mesh flips that: the team that produces the data (Orders, Marketing, Logistics, Risk) also owns it, packages it as a documented data product, and publishes it. Everyone else discovers those products in a shared catalog and reads them in place — under one common set of rules. Nobody re-copies anybody’s data, and no central team stands between producer and consumer.

Think of it as the difference between a single government canteen and a food market. The canteen (a central data warehouse) has to cook every cuisine for the whole city from one kitchen: it is always behind, and it never quite gets each region’s recipe right. A food market is a mesh — each stall owns and cooks its own dish (a data product) to a shared health code (governance), a single directory board (the catalog) lets you find any stall, and you eat at the stall (zero-copy) instead of everyone hauling raw ingredients home to re-cook them (the copy-everywhere anti-pattern). A health inspector (federated governance) writes the code once and audits every stall — but never cooks.

On Google Cloud this maps unusually cleanly, because Dataplex exists precisely to be the domain abstraction, BigQuery is the serving surface, Analytics Hub is the zero-copy exchange, and Dataplex Universal Catalog plus policy tags are the shared governance plane. The rest of this lesson makes each of those mappings concrete, from a three-domain company to a fifty-domain regulated enterprise.

Level: Advanced · Time: ~38 min

Prerequisites. You should be comfortable with BigQuery datasets/tables, the GCP resource hierarchy (org → folders → projects) and IAM basics (roles, groups, service accounts), and the idea of a data warehouse vs. a data lake. A passing familiarity with Dataplex, Analytics Hub, and policy tags helps but is not required — each is introduced here.

After this lesson you can:

• Name the four data-mesh principles and map each to a specific GCP service.
• Lay out project-per-domain with a Dataplex lake / zone / asset structure.
• Stand up a policy-tag taxonomy and enforce column- and row-level security at query time.
• Publish a curated table as an Analytics Hub listing and consume it as a zero-copy linked dataset.
• Write a data contract as a tag template plus a Dataplex data-quality scan, gated in CI.
• Decide when a mesh is right — and when a lakehouse or a plain warehouse is the better call.

A central data team is the most over-subscribed resource in almost every enterprise. The teams that create the data — Orders, Marketing, Logistics, Risk — throw it over a wall into a shared warehouse, and one small platform group becomes the single thread through which every model, every join, and every “can you add this column” request must pass. The backlog grows, the platform team learns each domain’s semantics badly and slowly, and producers stop caring about quality because quality became somebody else’s job. A data mesh is the organizational and architectural answer: the domain that produces the data also owns it as a product, publishes it through a shared self-serve platform, and lets other domains discover and consume it under one governance regime — without a central choke point.

On Google Cloud the mesh has an unusually clean mapping, because Google ships a service whose entire purpose is to be the domain abstraction layer: Dataplex. A Dataplex lake is a domain; zones inside it separate raw from curated; the assets it manages are BigQuery datasets and Cloud Storage buckets that physically live in the domain’s own project. Dataplex Universal Catalog (the unification of Data Catalog and Dataplex) is the discovery, metadata, and policy-tag plane for the whole organization. Analytics Hub is the governed exchange where a domain publishes a data product and other domains subscribe to it — a read-only, zero-copy link rather than an export. This article is a reusable reference for standing that up, from a three-domain company on a few terabytes to a fifty-domain regulated enterprise on petabytes. What changes with scale is the number of lakes and the strictness of the policy tags, not the shape.

The business scenario

Picture a mid-market online retailer that has grown into something larger and is now drowning in its own success. Four years ago a two-person platform team built a single BigQuery project, acme-analytics-prod, and offered to ingest everyone’s data. For the first handful of pipelines it was wonderful. Today it is the bottleneck the whole company complains about:

• The platform team carries an 11-week backlog of “please onboard this dataset” and “please add this field” requests. They do not understand Logistics’ carrier-SLA semantics or Risk’s chargeback model well enough to model the tables correctly, so they get them subtly wrong and the rework loop never closes.
• Producers have no incentive to keep schemas clean or document anything, because once the data lands in acme-analytics-prod its quality is “the analytics team’s problem.”
• Access is all-or-nothing. Past ~50 tables, per-dataset IAM became unmanageable, so almost everyone who needs anything has bigquery.dataViewer on the whole project. Finance tables with customer PII and unhashed card BINs are one query away from any analyst in the building.
• The single project is a cost and blast-radius problem: one runaway SELECT * on a 40 TB table, or one mis-scheduled hourly transform, hits everybody’s slots, and nobody can attribute the spend to a team because it is all one billing line.
• Nobody can find anything. There is no catalog; tribal knowledge lives in a Confluence page that was last accurate in 2023.

This is the moment a data mesh pays for itself. The four mesh principles — domain ownership, data as a product, self-serve platform, and federated computational governance — map almost one-to-one onto Google Cloud primitives. The objective is not “more technology.” It is to decentralize the production of data while centralizing the governance of access and the discoverability of products. Critically, this is not a big-bang rewrite. The same pattern that serves three domains scales to fifty by adding projects and Dataplex lakes, not by re-architecting.

What “good” looks like at the end:

• Each business domain owns its own GCP project, its own BigQuery datasets and Cloud Storage buckets, and its own pipelines. The domain registers that storage into a Dataplex lake it administers, and publishes curated, documented data products (well-described tables/views with an owner, an SLA, a freshness guarantee, and a data-quality scorecard).
• A central governance project holds the organization-wide policy-tag taxonomy, Universal Catalog tag templates, and the org policies — and audits access. It does not own anyone’s data.
• Consumers search one catalog across all domains, find a product, and consume it through an Analytics Hub subscription (zero-copy, read-only) or a governed BigQuery grant. Column-level security via policy tags and row-level security via row-access policies are enforced at query time. No copying, no full-project access, no 11-week tickets.

The four mesh principles, mapped to GCP

The mesh is defined by four principles (from Zhamak Dehghani’s original formulation). None of them is a product — they are operating principles — but on Google Cloud each has a natural home. Learn the four, and the rest of the architecture is just detail.

Read the table top-to-bottom and a design falls out: decentralize principles 1–3 (ownership, product, platform) into the domain projects; centralize principle 4 (governance) as code in a shared project that owns rules, not data. That single sentence — “decentralize production, centralize governance” — is the whole architecture in miniature.

“Computational” is the load-bearing word. Federated governance is not a committee that reviews every table by hand; it is policy expressed as artifacts the platform enforces on its own: a policy tag masks a column at query time, a failed DQ scan blocks a promotion in CI, an org policy refuses a bucket in the wrong region. Humans set the standard; the machine applies it on every query and every deploy. That is what keeps governance from becoming the new bottleneck.

Data as a product means a data contract

The phrase “data as a product” is concrete on GCP: a product ships with a data contract, and the contract is two versioned artifacts in Git — a metadata contract (a Universal Catalog tag template the producer must fill in) and a quality contract (a Dataplex data-quality ruleset). A sketch of the metadata half:

# data-contract: orders.fact_order  (versioned in the domain's repo)
product:        fact_order
domain:         orders
owner:          grp-orders-data-owners@acme.com
on_call:        orders-oncall@acme.com
sla_hours:      4            # max staleness the product promises
freshness_min:  60           # target load latency
classification: internal
contains_pii:   true         # → sensitive columns carry policy tags
ports:
  analytics_hub_listing: orders/internal-exchange/fact_order
  authorized_view:       orders.products.v_fact_order
quality:                     # the DQ half, enforced by a Dataplex scan
  - column: order_id     rule: not_null      threshold: 1.00
  - column: order_id     rule: unique
  - column: order_total  rule: value >= 0    threshold: 0.99

The producer changes the contract, not the consumer’s expectations, and a breaking change is a schema-version bump, not a surprise. That is the difference between a product (a promise you can depend on) and a table (something that might change under you tonight).

Conway’s law is the real reason this works

A data mesh is an organizational pattern first and a technical one second — because of Conway’s law: a system’s architecture inevitably mirrors the communication structure of the org that builds it. One central data team produces one monolithic warehouse; you cannot get a decentralized architecture out of a centralized org chart. The mesh applies the inverse Conway maneuver — you deliberately reshape teams, giving each domain a data-product team with its own on-call, so that the architecture you want (independent, owned products) becomes the natural output. This is why “mesh-washing” fails: if the same central team still builds every table, Conway’s law guarantees you still have a monolith, no matter how many Dataplex lakes you draw on the diagram. The ownership has to actually move, or nothing else does.

Data mesh vs. data lake vs. data warehouse

These three get muddled constantly, and the difference is mostly about ownership and copies, not storage technology — all three can sit on BigQuery and Cloud Storage. Here is the honest comparison:

The key insight: a mesh is not a replacement technology for a warehouse or lake — it reuses both (each domain’s curated zone is a small warehouse; each raw zone is a small lake) and adds two things they lack: distributed ownership and a governed, zero-copy exchange. If you have one team and one domain, you do not want the federation overhead — you want a lakehouse (one governed BigQuery + BigLake copy, one team). See the sibling lesson Enterprise Architecture: Lakehouse for that simpler pattern; reach for the mesh only when ownership itself needs to decentralize.

Architecture overview

The end-to-end shape is a mesh of self-governing domains over a shared catalog and exchange. Storage, compute, and pipelines live in the domain projects (the nodes); the catalog, the policy taxonomy, and the publish/subscribe exchange are the shared fabric that connects them.

Reading the diagram (path 1→7): an analyst searches the Universal Catalog and finds a product (1), subscribes via Analytics Hub to a read-only linked dataset (2), and queries it from their own BigQuery project (3); at query time the policy-tag column check (4) and the row-access filter (5) apply, BigQuery reads the producer’s storage in place (6), and the governed result returns (7) — the bytes never leave the producer’s project, and the governance travels with the query.

Resource hierarchy — the nodes and the fabric. Under the GCP Organization, you create a folder layout that mirrors the mesh: a domains/ folder holds one project per domain (acme-orders, acme-marketing, acme-logistics, acme-risk, …); a platform/ folder holds the shared governance/catalog project and the self-serve tooling project (Composer, Terraform CI, the data-product template). Organization policies and IAM provide guardrails you cannot opt out of; project boundaries give you per-domain isolation, blast-radius control, and clean cost attribution by default — every domain is its own billing slice.

Producer (domain) path — how raw data becomes a product. Inside a domain project, source data lands in a domain-owned Cloud Storage bucket or streams into BigQuery (via Dataflow, Datastream CDC, Pub/Sub + BigQuery subscriptions, or BigLake over object storage). The domain creates a Dataplex lake (orders-domain) and inside it two or more zones: a raw zone (Cloud Storage, schema-on-read) and a curated zone (BigQuery datasets and/or BigLake tables, schema-validated). It attaches its buckets and datasets to those zones as assets. Dataplex then runs automatic discovery over those assets — crawling files and tables, inferring schema and partitions, and registering everything into the Universal Catalog without a human writing DDL. The domain team curates the product: it writes a transform (BigQuery SQL, a dbt model, or a Dataform pipeline) that produces a clean, documented curated table, attaches Universal Catalog tag templates (owner, SLA, freshness, classification), tags sensitive columns with policy tags, and wires a Dataplex data-quality scan so the product ships with a measured, published quality score. At this point the curated table is a candidate data product.

Governance path — federated, computational, central where it must be. The shared governance project owns exactly two things that must be global: the policy-tag taxonomy (a BigQuery taxonomy of policy tags like pii.email, pii.card_bin, confidential.financial, each mapped to a fine-grained reader role) and the catalog tag-template definitions (the metadata contract every product must fill in — owner, domain, SLA, freshness, sensitivity). Domains apply those tags to their own resources; governance defines them once for everyone. This is the “federated computational governance” pillar made concrete: policy is authored centrally as code, enforced automatically by BigQuery and Dataplex at query time, and the data-quality and lineage signals roll up so a platform owner can see, across all domains, which products are healthy, which contain PII, and where any column flows downstream — without owning a single byte.

Consumer path — discover, subscribe, query in place. A consumer (say a Marketing analyst) opens Dataplex search / the Universal Catalog and searches across every domain’s published products at once — filtered by tag template, classification, or business glossary term. They find orders.curated.fact_order and read its product page: owner, SLA, freshness, quality score, schema, and lineage. To consume it they either (a) subscribe via Analytics Hub — the Orders domain has published the product into a data exchange as a listing; the analyst’s project subscribes and receives a linked dataset, a read-only, zero-copy pointer that always reflects the source, billed to the consumer’s project for compute — or (b) receive a direct BigQuery IAM grant for tighter internal sharing. Either way, when they run the query, column-level access is enforced by the policy tags (the analyst lacking pii.email reader simply does not see that column; the query does not fail, the column is masked or omitted), and row-level security is enforced by any row-access policies on the table. The bytes are never copied — the consumer’s BigQuery slots read the producer’s storage through the governed link.

So the request/data path for a single cross-domain query is: analyst → Dataplex search (find product) → Analytics Hub subscription (linked dataset) → BigQuery query in the consumer project → policy-tag check (column ACL) + row-access policy (row filter) → producer’s BigQuery storage read in place → governed result back. Metadata and discovery are centralized in the catalog; policy is centralized as code; ownership, storage, pipelines, and compute are decentralized to the domains; and bytes flow point-to-point, zero-copy, from producer storage to consumer compute — governed end to end.

Component breakdown

Implementation guidance

The whole mesh is infrastructure-as-code, and the most important IaC decision is the module boundary: ship a single, opinionated “domain” Terraform module so a new domain is a 30-line call, not a research project. That module is the self-serve platform pillar expressed as code.

The domain module (Terraform). Google provides first-class google_dataplex_* and google_bigquery_* resources; a domain module composes them:

• google_project (or a reference to an existing one) under the domains/ folder, with the domain’s billing label.
• google_dataplex_lake for the domain, plus google_dataplex_zone (one RAW, one CURATED) and google_dataplex_asset resources attaching the domain’s google_storage_bucket and google_bigquery_dataset.
• Discovery (discovery_spec) enabled on each zone with a schedule.
• A google_bigquery_dataset per curated product, plus the authorized view wiring (google_bigquery_dataset_iam_* and view-on-base-table grants) so the product port is exposed without base-table access.
• google_dataplex_datascan resources for data quality and data profiling, pointed at the curated tables, with rules and a schedule.
• Standard IAM bindings that delegate lake admin and dataset owner to the domain’s group, while the org-level taxonomy stays in the governance project.

The governance module (Terraform). Lives in the platform/governance project and is owned by the central team:

• google_data_catalog_taxonomy + google_data_catalog_policy_tag to define the policy-tag taxonomy once (pii.email, pii.card_bin, confidential.financial, …). Domains reference these tags by ID when tagging columns via policy_tags on a google_bigquery_table schema.
• google_data_catalog_tag_template resources for the metadata contract (owner, SLA hours, freshness minutes, classification, on-call) that every product must populate.
• google_bigquery_analytics_hub_data_exchange for each trust boundary; domains create google_bigquery_analytics_hub_listing resources to publish into it.
• Fine-grained reader roles bound to the right groups so policy-tag enforcement actually gates columns.

Because Dataplex and the catalog are regional/multi-regional, pin every domain’s lake, datasets, and buckets to the same location (e.g. EU or us-central1) the org policy mandates — Analytics Hub linked datasets and BigQuery queries cannot cross region boundaries silently, and data-residency rules depend on it.

Networking and identity wiring. A data mesh is mostly an identity problem, not a network one — the heavy lifting is IAM, not VPC:

• Identity. Sync your IdP (e.g. Entra ID / Google Workspace) groups via Cloud Identity, and grant groups, never users. Each domain has a grp-<domain>-data-owners (lake admin, dataset owner) and the platform team has grp-platform-governance (taxonomy and exchange admin). Policy-tag reader roles are bound to consumer groups like grp-marketing-analysts. Workload Identity Federation lets domain CI/CD (GitHub Actions, GitLab) deploy without long-lived service-account keys; pipelines run as dedicated service accounts per domain with least privilege.
• Network. Lock down data egress with VPC Service Controls: put all the data projects in a service perimeter so BigQuery and Cloud Storage data cannot be exfiltrated to a project outside the org, even with valid credentials. Use Private Google Access / Private Service Connect so analytics traffic to BigQuery and GCS stays on Google’s backbone and never traverses the public internet. Analytics Hub sharing within the perimeter is allowed; cross-perimeter listings are an explicit, audited bridge.
• Encryption. CMEK (Cloud KMS) on every dataset and bucket, with keys in a domain (or central security) key project, enforced by org policy.

Promotion and contracts. The golden path is: data lands in the raw zone → a Dataform/SQL transform builds the curated table → a Dataplex DQ scan runs → on pass, the table is tagged with the metadata contract, policy tags are applied to sensitive columns, and an Analytics Hub listing is created (or refreshed) by CI. A failed DQ scan blocks promotion and pages the domain’s on-call. The data contract is literally the tag template plus the DQ ruleset, both versioned in Git.

Enterprise considerations

Security & Zero Trust. The mesh defaults to Zero Trust because nothing is granted at the project level for data — access is per-product, per-column, per-row, and bound to groups. The enforcement layers stack: VPC Service Controls stop exfiltration at the perimeter; IAM on Analytics Hub listings controls who can subscribe; policy tags mask or omit sensitive columns at query time for anyone lacking the fine-grained reader role (the query succeeds without the column rather than erroring, which keeps analysts productive); row-access policies filter rows per group; CMEK owns the encryption boundary; and dynamic data masking can show email as a hash to one role and cleartext to another on the same column. Every access is logged — see observability.

Cost optimization. Project-per-domain is the FinOps win: each domain is a billing slice, so the “whose query cost ₹3 lakh this month” question answers itself. On top of that: use BigQuery editions with reservations and autoscaling (assign a baseline + autoscale slots per domain so one domain’s runaway job cannot starve another), or on-demand with per-project/per-user bytes-billed quotas as a hard cap for smaller domains. Storage is cheap and shared-by-reference: Analytics Hub linked datasets mean a product consumed by ten domains is stored once and never duplicated — the classic mesh anti-pattern of “everyone exports a copy” is structurally impossible here. Partition and cluster every large product table, set partition expiration on raw zones, and lifecycle raw Cloud Storage to coldline. Tag every resource with domain, cost-center, and environment for showback/chargeback in BigQuery billing export.

Scalability. The pattern scales by addition. Going from 4 to 40 domains means 36 more module calls — more lakes, more projects, more listings — not a re-architecture, because there is no central pipeline to overload. BigQuery itself scales to petabytes and to thousands of concurrent queries; the catalog search scales across the whole org; Analytics Hub fans a single product out to many subscribers at zero marginal storage. The thing that must scale organizationally is the federated governance council that owns the shared taxonomy — keep the policy-tag and tag-template set small and stable.

Reliability & DR (RTO/RPO). BigQuery storage is durable and replicated within a location, and multi-region datasets (US, EU) give cross-regional resilience automatically. For stricter needs, BigQuery managed disaster recovery / cross-region dataset replication provides a standby with a defined failover; table snapshots and time travel (default 7 days, configurable to 2–7) cover accidental deletes and bad transforms with near-zero RPO for recent state. Cloud Storage uses dual-region or multi-region buckets for the raw zone. Practical targets for a mid-size deployment: RPO ≈ minutes (continuous replication + time travel) and RTO ≈ 1 hour (failover to the replica region and re-point Analytics Hub listings). Because pipelines are per-domain and IaC-defined, recovering a single domain is independent of the others — a failure blast-radius bounded to one lake, not the whole platform.

Observability. Three layers, all native: (1) Cloud Audit Logs + BigQuery INFORMATION_SCHEMA views give you who queried what, which columns/policy tags were touched, bytes billed, and slot utilization per domain — the governance audit trail. (2) Dataplex data-quality and profiling scans publish a live scorecard per product, and Dataplex lineage shows column-level flow for impact analysis. (3) Cloud Monitoring dashboards and alerts on slot contention, DQ-scan failures, and freshness SLA breaches, routed to each domain’s on-call. The product page in the Universal Catalog ties these together so a consumer sees the SLA, the latest quality score, and the lineage before they subscribe.

Governance. This is the centralized pillar, and it is computational, not manual: policy tags and tag templates are authored once as code in the governance project and enforced automatically by BigQuery/Dataplex; promotion is gated on DQ scans in CI; data residency and CMEK are enforced by org policy; and a lightweight federated governance council (platform + one rep per domain) owns the shared taxonomy and the data-contract standard. The principle is standards centralized, execution decentralized — governance defines the rules and audits compliance; domains own and run everything else.

Going deeper

Everything above is the shape. This section is the production detail that separates a mesh that demos from one that survives an audit and a scale-up.

Column security: what actually happens on SELECT *

A subtle but important correctness point. Policy tags alone deny; they do not silently drop. If a user lacks the fine-grained reader role (roles/datacatalog.categoryFineGrainedReader) for a policy tag on a column, then a bare SELECT * fails with an access error naming the column — it does not quietly omit it. The user has two ways to succeed:

1. Exclude the column explicitly: SELECT * EXCEPT(customer_email) FROM ... — the query runs because it never touches the protected column.
2. Configure dynamic data masking on the policy tag: attach a masking rule (e.g. SHA256, DEFAULT, or NULL) and grant the masked reader role (roles/bigquerydatapolicy.maskedReader). Now SELECT * succeeds and the column comes back masked instead of erroring.

So “the analyst just doesn’t see the PII column” is true only when you have added masking; with plain column ACLs, SELECT * errors and analysts must exclude the column. Decide per product which behavior you want — masking keeps dashboards from breaking; hard denial is stricter. This is the single most common surprise for teams new to BigQuery policy tags, and it is why the enterprise section pairs policy tags with dynamic data masking.

Analytics Hub linked datasets are symbolic links, not copies

A linked dataset is a pointer, not an export. Consequences worth internalizing:

• Storage is billed to the publisher; compute to the subscriber. Ten domains subscribing to one product store zero extra bytes; each pays only for the slots its own queries burn. This is what makes “everyone consumes, nobody copies” structurally true, not just aspirational.
• It cannot cross regions. A listing in the EU multi-region can only be subscribed by datasets in EU. This is why the implementation guidance insists every domain pin to the same location — mismatch and the subscription simply cannot be created.
• Freshness is instant — the link reflects the source table live; there is no replication lag because there is no replication.
• Restricted egress can be set on a listing to block subscribers from copying or exporting the data out, so a zero-copy product stays zero-copy even downstream — important for PII/PCI products.
• Cross-org and commercial sharing work the same way: a listing can be published to another organization’s projects, or monetized on Cloud Marketplace, without changing the pattern.

VPC Service Controls and the cross-perimeter bridge

Put every data project in one VPC-SC service perimeter and BigQuery/GCS data cannot be exfiltrated to a project outside it, even by someone holding valid credentials — this is the control that upgrades IAM (who may read) with a topology boundary (where data may go). Two nuances: Analytics Hub sharing within a perimeter is transparent, but a listing consumed by a project in a different perimeter (e.g. the 3PL partner org) requires an explicit egress rule — an audited, intentional bridge, not an accident. Always roll perimeters out in dry-run mode first; a misconfigured perimeter blocks every pipeline at once, and dry-run shows you exactly what would be denied before you enforce. See VPC Service Controls for the perimeter mechanics.

BigQuery slots: fair-sharing without starvation

Per-domain BigQuery Editions reservations (Standard / Enterprise / Enterprise Plus) are how you stop one domain’s runaway job from starving another. Give each domain a baseline of slots plus autoscaling to a ceiling, and assign the reservation to the domain’s project(s) via an assignment. Idle slots within a reservation are shared automatically among that reservation’s jobs, so a baseline of 100 with autoscale to 500 is efficient and bounded. For small domains, plain on-demand with a per-project bytes-billed quota is a simpler hard cap. The reference example’s “Card/Risk 500-slot baseline, others 100–200” is exactly this dial. The deep mechanics live in BigQuery deep dive, and the column/row security in BigQuery fine-grained access.

Quotas and design limits that shape the taxonomy

The “keep the taxonomy small” advice is not just aesthetic — it is a hard limit. A single policy-tag taxonomy caps at roughly 100 policy tags with a hierarchy only a few levels deep, and there is a small cap on taxonomies per project per region. This is deliberate: column security only scales if you grant on a handful of classifications (pii.email, pii.card_bin, confidential.financial), not on hundreds of individual columns. Similarly, authorized views and row-access policies carry per-table limits, and Analytics Hub caps listings per exchange — all reasons the mesh grows by adding domains (each with its own budget of these objects) rather than by piling everything into one project. Always check the current published quotas before you design a taxonomy that assumes headroom.

The catalog is moving: Dataplex Universal Catalog

A live API caveat: Google has consolidated the old standalone Data Catalog into Dataplex Universal Catalog, and is retiring the standalone Data Catalog API/console (announced for early 2026) — the discovery, tag-template, and policy-tag capabilities now live under Dataplex. The metadata model is also evolving from tag templates toward Aspects / Aspect Types (a richer, typed metadata attachment). Policy tags remain the column-security mechanism. Practically: the Terraform google_data_catalog_* resources still provision taxonomies and tag templates today, but confirm the current surface before you build a new mesh, because the console path and some API names are in flight. This is exactly the kind of thing to verify against live docs rather than trust from memory.

Lineage and impact analysis

Dataplex lineage is captured automatically for BigQuery, Dataflow, Dataform, and Composer, and it is column-level — so before a domain changes fact_order.order_total, it can see every downstream product and dashboard that reads that column across other domains. You can also report custom lineage via the API (or OpenLineage) for systems Google does not auto-instrument. In a mesh this is the trust primitive: a consumer subscribing to a product can see, on the product page, both where its inputs came from and where its outputs go — impact analysis without a spreadsheet.

Reference enterprise example

NorthBank Retail Group is a fictional ₹9,000-crore omnichannel retailer (≈ 4,000 stores, online, and a co-branded card) running on Google Cloud. They had the exact pain in the scenario: one BigQuery project, an 11-week onboarding backlog, and a co-branded-card PII exposure that nearly failed a PCI audit. They adopted the mesh over two quarters.

Topology. Five domains, one lake each, under a domains/ folder, all pinned to the asia-south1-anchored ASIA multi-region for data-residency:

Shared nb-platform-gov holds the policy-tag taxonomy (pii.email, pii.phone, pii.card_bin, confidential.financial), the tag templates (owner, SLA-hours, freshness-min, classification, on-call), and two Analytics Hub exchanges: internal-exchange (all five domains) and partner-exchange (a tightly-scoped exchange that publishes only shipment_sla to their 3PL logistics partner’s separate GCP org).

Key decisions. (1) Card/Risk publishes via Analytics Hub, never via direct grant — txn_authz is listed with card_bin and pan_hash carrying policy tags; only grp-fraud-analysts holds the pii.card_bin and confidential.financial reader roles, so a Marketing analyst subscribing to the same product sees every column except those two, masked at query time. This is what flipped the PCI finding from fail to pass. (2) Marketing’s dim_customer_360 consumes Orders and Card data by subscription, not export — it joins three domains’ products as linked datasets, so there is exactly one physical fact_order, governed once, and Marketing’s queries are billed to nb-marketing. (3) Logistics’ shipment_sla is dual-published — internally and, via the partner exchange, to the 3PL across an org boundary, zero-copy, with an audited cross-perimeter bridge. (4) Promotion is gated on Dataplex DQ — fact_order ships with a 99.2% completeness and 100% uniqueness scorecard on the product page; a failed scan blocks the listing refresh and pages Orders’ on-call.

Reservations & cost. Each domain gets a small BigQuery Enterprise reservation with autoscaling (Card/Risk baseline 500 slots, others 100–200, autoscaling to a per-domain ceiling), so Card/Risk’s nightly fraud scoring can never starve Marketing’s dashboards. Total platform spend landed at ≈ ₹23.4 L/month — lower than the old single-project bill, because the zero-copy subscriptions eliminated five redundant nightly export pipelines and their storage, and per-domain quotas killed the runaway-SELECT * incidents.

Outcome. Onboarding a new data product dropped from 11 weeks to under 4 days (a domain calls the Terraform module, lands data, the DQ scan passes, CI publishes the listing). Cross-domain discovery went from “ask in Slack” to a single Universal Catalog search. The PCI audit passed with the card columns provably masked-by-default. And the central platform team — still two people — stopped writing other teams’ pipelines and instead owned the paved road: the Terraform module, the taxonomy, the exchanges, and the governance council. Decentralized production, centralized governance, exactly as intended.

When to use it

Use a GCP data mesh when you have multiple data-producing domains with distinct semantics, a central data team that has become a delivery bottleneck, and a real need for fine-grained, classification-driven access (PII/PCI/financial) across organizational boundaries. The sweet spot starts around three or more domains with genuinely different data and ownership, and the value compounds as you grow — the pattern is explicitly designed so that scaling is additive.

Trade-offs. The mesh moves cost and complexity to the producers: every domain now owns pipelines, quality scans, and an on-call rotation it did not have before. That is the point — accountability follows ownership — but it demands domains that are willing and able to own data as a product. It also requires a real federated governance function; without someone owning the shared taxonomy and the data-contract standard, you get fifty inconsistent catalogs and the mesh degrades into a swamp.

Anti-patterns to avoid. (1) “Mesh-washing” a central lake — slapping Dataplex lakes onto datasets that one central team still builds and owns is not a mesh; it is a relabeled monolith with extra YAML. The ownership has to actually move. (2) Export-and-copy instead of subscribe — if domains email each other CSVs or schedule cross-project exports, you have rebuilt the four-divergent-copies problem; force consumption through Analytics Hub linked datasets or authorized views so there is one physical copy. (3) Per-column IAM instead of policy tags — grant on classifications, not on hundreds of individual columns, or column security will not scale past the first few products. (4) A sprawling taxonomy — keep policy tags and tag templates small and stable; a 200-tag ontology nobody understands is worse than ten that everyone does. (5) Mesh for one domain — if you have a single analytics team on a few terabytes, you do not need a mesh; you need a clean lakehouse (one governed copy in BigQuery + BigLake, one team), which is dramatically simpler.

Alternatives. For a single team / single domain, use the GCP lakehouse pattern (BigQuery + BigLake + Dataplex governance, one queryable copy) — same building blocks, none of the federation overhead. For a lift-and-shift warehouse with light governance needs, a well-organized BigQuery project with authorized views and policy tags may be enough without the full mesh apparatus. If you are multi-cloud and the data physically lives in S3 or Azure, BigQuery Omni / BigLake over external stores can bring those into the same catalog and policy model so the mesh spans clouds. And if your organization is not ready to make domains own their data, do not start with the mesh — fix the ownership culture first, because the architecture cannot manufacture accountability the org is unwilling to take on.

Common beginner mistakes

Distinct from the architectural anti-patterns in When to use it — these are the conceptual traps that trip up people new to the mesh and to GCP’s data services.

• Thinking a mesh is a product you install. There is no “Data Mesh” service to enable. The mesh is an operating model (four principles) realized with ordinary GCP services. If you go looking for a “turn on data mesh” button, you have misunderstood the pattern. Right model: it is a way of organizing ownership, plumbed with Dataplex + BigQuery + Analytics Hub + policy tags.

• Assuming SELECT * silently hides columns you can’t read. As covered in Going deeper, plain policy tags error on a protected column; only dynamic data masking makes the query succeed with the value masked. Beginners write SELECT *, get an access error, and conclude column security is broken — it is working exactly as designed. Use SELECT * EXCEPT(col) or add a masking rule.

• Confusing a Dataplex lake with a Cloud Storage “data lake.” A Dataplex lake is a logical domain container (a governance construct) that groups BigQuery datasets and GCS buckets across a project — it is not a storage bucket. The word “lake” collides with the older meaning; keep them separate in your head.

• Copying data “just to be safe.” The entire point of Analytics Hub linked datasets is that there is one physical copy. A beginner instinct is to CREATE TABLE AS SELECT a local copy of a shared product “so my dashboards are stable.” That rebuilds the copy-everywhere problem, doubles storage, and guarantees drift. Subscribe; don’t copy.

• Granting bigquery.dataViewer at the project level “to unblock the team.” This is the exact monolith habit the mesh exists to kill. Access is per-product (Analytics Hub listing IAM), per-column (policy tag), per-row (row-access policy) — never whole-project. Project-level dataViewer on a domain with PII is how audits fail.

• Standing up the tech before moving the ownership. Drawing lakes and exchanges while the same central team still builds every table is “mesh-washing.” Conway’s law wins: the architecture will mirror the org. Move the ownership (and the on-call) to the domains first, or the diagram is fiction.

Practice challenges

Work these in order — each builds on the last. Commands are real and schema-correct against the current gcloud / bq / Terraform surface; replace every placeholder (PROJECT_ID, group emails, taxonomy/tag IDs) with your own. No live project is assumed here — treat any output as representative.

1 (Beginner) — Create a domain lake with raw and curated zones. Stand up a Dataplex lake orders-domain in asia-south1 with a RAW zone (for GCS landing data) and a CURATED zone (for BigQuery products).

<details> <summary>Solution</summary>

gcloud dataplex lakes create orders-domain \
  --project=acme-orders --location=asia-south1 \
  --display-name="Orders domain"

gcloud dataplex zones create raw \
  --project=acme-orders --location=asia-south1 --lake=orders-domain \
  --type=RAW --resource-location-type=SINGLE_REGION \
  --display-name="Raw landing"

gcloud dataplex zones create curated \
  --project=acme-orders --location=asia-south1 --lake=orders-domain \
  --type=CURATED --resource-location-type=SINGLE_REGION \
  --display-name="Curated products"

Why: the lake is the domain node and zones separate schema-on-read landing (RAW) from schema-validated products (CURATED) — the mesh’s node in three commands. </details>

2 (Beginner) — Attach a BigQuery dataset as an asset and enable discovery. Attach the domain’s existing curated BigQuery dataset to the curated zone and turn on hourly auto-discovery so schema lands in the catalog with no DDL.

<details> <summary>Solution</summary>

gcloud dataplex assets create fact-order-asset \
  --project=acme-orders --location=asia-south1 \
  --lake=orders-domain --zone=curated \
  --resource-type=BIGQUERY_DATASET \
  --resource-name="projects/acme-orders/datasets/curated" \
  --discovery-enabled \
  --discovery-schedule="0 * * * *"

Why: an asset brings existing storage under domain governance without moving it; discovery crawls it and registers entries into the Universal Catalog automatically — the “no hand-written DDL” promise. </details>

3 (Intermediate) — Define a policy-tag taxonomy and tag a PII column. In the governance project, create a FINE_GRAINED_ACCESS_CONTROL taxonomy with a pii.email policy tag (Terraform), then bind it to the customer_email column of fact_order.

<details> <summary>Solution</summary>

resource "google_data_catalog_taxonomy" "pii" {
  project      = "acme-platform-gov"
  region       = "asia-south1"
  display_name = "acme-pii"
  activated_policy_types = ["FINE_GRAINED_ACCESS_CONTROL"]
}

resource "google_data_catalog_policy_tag" "email" {
  taxonomy     = google_data_catalog_taxonomy.pii.id
  display_name = "pii.email"
}

Then in the table schema (JSON, or the google_bigquery_table schema), the column carries the tag:

{ "name": "customer_email", "type": "STRING",
  "policyTags": { "names": ["projects/acme-platform-gov/locations/asia-south1/taxonomies/TAX_ID/policyTags/TAG_ID"] } }

Why: you grant the fine-grained reader role on the classification (pii.email) once, not on every email column across the org — this is what makes column security scale past the first few products. </details>

4 (Intermediate) — Enforce row-level security by region. On fact_order, add a row-access policy so members of grp-eu-analysts see only region = 'EU' rows, with no data copy.

<details> <summary>Solution</summary>

CREATE ROW ACCESS POLICY eu_only
ON `acme-orders.curated.fact_order`
GRANT TO ('group:grp-eu-analysts@acme.com')
FILTER USING (region = 'EU');

Why: same physical table, different rows per group — row-level security and column-level policy tags compose, so one product serves EU and non-EU consumers without a fork. </details>

5 (Advanced) — Publish and consume a product zero-copy via Analytics Hub. Publish the curated dataset as a listing in an internal exchange (Terraform), then have acme-marketing subscribe to receive a read-only linked dataset.

<details> <summary>Solution</summary>

resource "google_bigquery_analytics_hub_data_exchange" "internal" {
  project          = "acme-platform-gov"
  location         = "asia-south1"
  data_exchange_id = "internal_exchange"
  display_name     = "internal-exchange"
}

resource "google_bigquery_analytics_hub_listing" "fact_order" {
  project          = "acme-platform-gov"
  location         = "asia-south1"
  data_exchange_id = google_bigquery_analytics_hub_data_exchange.internal.data_exchange_id
  listing_id       = "fact_order"
  display_name     = "Orders — fact_order"
  bigquery_dataset { dataset = "projects/acme-orders/datasets/curated" }
}

The consumer subscribes (console, API, or google_bigquery_analytics_hub_listing_subscription), producing a linked dataset in acme-marketing — read-only, zero-copy, billed to Marketing for compute.

Why: the product is stored once in acme-orders and read in place; policy tags on customer_email still mask or deny for anyone lacking the reader role, even through the subscription. </details>

6 (Advanced) — Gate promotion on a Dataplex data-quality scan. Create a DQ scan on fact_order (not-null + unique order_id, order_total >= 0) and wire CI so a failing scan blocks the Analytics Hub listing refresh.

<details> <summary>Solution</summary>

dq-rules.yaml:

rules:
  - column: order_id
    dimension: COMPLETENESS
    nonNullExpectation: {}
    threshold: 1.0
  - column: order_id
    dimension: UNIQUENESS
    uniquenessExpectation: {}
  - column: order_total
    dimension: VALIDITY
    rangeExpectation: { minValue: "0" }
    threshold: 0.99

gcloud dataplex datascans create data-quality fact-order-dq \
  --project=acme-orders --location=asia-south1 \
  --data-source-resource="//bigquery.googleapis.com/projects/acme-orders/datasets/curated/tables/fact_order" \
  --data-quality-spec-file=dq-rules.yaml

gcloud dataplex datascans run fact-order-dq \
  --project=acme-orders --location=asia-south1

In CI, read the job’s pass/fail (gcloud dataplex datascans jobs describe ...) and only run the terraform apply that refreshes the listing when it passed.

Why: “data as a product” means the product ships with a measured SLA; gating promotion on the scan makes the quality contract executable, not aspirational — a failed scan pages the domain, it does not ship bad data downstream. </details>

Glossary

• Data mesh — an operating model where data-producing domains own their data as products, published through a shared self-serve platform under federated governance; the opposite of a single central data team.
• Domain — a business capability (Orders, Marketing, Risk) that produces data and owns it end-to-end. On GCP, one domain ≈ one project + one Dataplex lake.
• Data product — a curated, documented, quality-scored, discoverable dataset with an owner and an SLA — not a raw table. Served as a BigQuery view/table plus its catalog metadata and Analytics Hub listing.
• Data contract — the versioned promise a product makes: a metadata contract (tag template: owner, SLA, freshness, classification) plus a quality contract (Dataplex DQ ruleset), both in Git.
• Dataplex — GCP’s data-fabric service; provides the lake / zone / asset domain abstraction, auto-discovery, data-quality scans, and lineage over BigQuery and Cloud Storage.
• Dataplex lake — a logical container representing one domain; groups that domain’s BigQuery datasets and GCS buckets. A governance construct, not a storage bucket.
• Zone — a tier inside a lake: a raw zone (schema-on-read, GCS) or a curated zone (schema-validated, BigQuery/BigLake).
• Asset — a specific BigQuery dataset or GCS bucket attached to a zone, bringing existing storage under domain governance without moving it.
• Dataplex Universal Catalog — the org-wide discovery, metadata, and policy plane (the merger of Data Catalog into Dataplex). Where any product across all domains is searched and described.
• Tag template — a structured metadata schema (owner, SLA, freshness, classification…) a product must populate; the machine-readable half of the data contract. (Evolving toward Aspects / Aspect Types.)
• Policy tag — a column-level access classification (e.g. pii.email) in a BigQuery taxonomy; grant the fine-grained reader role on the tag, not on individual columns.
• Taxonomy — the tree of policy tags, authored centrally in the governance project (caps at roughly 100 tags). The unit of column-security policy.
• Dynamic data masking — a rule attached to a policy tag that returns a masked value (hash/null/default) instead of erroring, letting SELECT * succeed for users with the masked-reader role.
• Row-access policy — a SQL predicate bound to a group that filters which rows a consumer sees on a shared table, with no copy.
• BigLake table — a BigQuery-governed table over Cloud Storage / external object data, so object data becomes a first-class, policy-tagged product.
• Analytics Hub — GCP’s publish/subscribe data exchange. Producers create listings; consumers subscribe to receive a linked dataset.
• Linked dataset — a read-only, zero-copy pointer in the consumer’s project that always reflects the producer’s source table; storage billed to the producer, compute to the consumer.
• Data exchange — a trust boundary in Analytics Hub that groups listings (e.g. internal-exchange, partner-exchange).
• Federated computational governance — the mesh principle that global standards are decided together and enforced automatically by the platform (policy tags, DQ gates, org policies), not by manual review.
• Self-serve data platform — the paved road (Terraform “domain” module, Composer templates) that lets a domain ship a product in days without central tickets.
• VPC Service Controls (VPC-SC) — a service perimeter that stops data exfiltration to projects outside the boundary, even with valid credentials.
• Conway’s law — the observation that a system’s architecture mirrors the org’s communication structure; the mesh applies the inverse Conway maneuver, reshaping teams to get the architecture.
• Inverse Conway maneuver — deliberately organizing teams (domain data-product teams with their own on-call) so the desired architecture (owned, independent products) emerges naturally.
• Lakehouse — the single-domain simplification: one governed BigQuery + BigLake copy owned by one team; the right choice when ownership does not need to decentralize.