GCP Enterprise Architecture: IoT Ingestion & Analytics

In a nutshell

Every connected-product company eventually asks its data two questions that fight each other. “What is this one device doing right now?” wants an answer in a blink. “Across my whole fleet over the last year, what’s trending toward failure?” wants to grind through billions of rows. No single database is good at both — so this architecture stops trying to force them into one store and instead splits a single firehose of device readings into two purpose-built ones.

Picture 60,000 refrigerators scattered across the country, each shouting its temperature every ten seconds. A phone switchboard that answers every call and checks each caller’s ID badge is the MQTT broker. A conveyor belt that never drops a package even when everyone calls at once is Pub/Sub. A sorting line that opens each package, throws out the junk, stamps it with the correct time, sticks on a label, and photocopies it into two filing systems is Dataflow. The “flip to any one fridge and see its latest reading instantly” binder is Bigtable. The “walk to the warehouse and pull every reading from the last 18 months” archive is BigQuery. The dashboards on the wall are Looker. And the intercom you use to tell a fridge to change its setting is the command-and-control path, running back the other way.

The one honest wrinkle: Google used to sell that switchboard as a managed service called Cloud IoT Core, and retired it in August 2023 with no direct replacement. So the modern Google Cloud IoT architecture starts with a box you have to fill yourself — a broker you run, or buy from a partner — and everything downstream is the part Google does brilliantly. This lesson walks the whole path, up and down, and explains every decision.

Level: Advanced · Time: ~40 min

Before you start, it helps to know what a message queue is and why you’d put one between producers and consumers (the Pub/Sub deep dive covers this), the basic idea of a stream-processing job (the Dataflow / Apache Beam deep dive), and the difference between a key-value store and an analytical warehouse (the Bigtable and BigQuery deep dives). You do not need prior IoT experience — building that mental model from scratch is the whole point of the lesson.

After this lesson you will be able to:

• Explain why Google Cloud IoT architectures no longer start with “IoT Core,” and name the broker options that replace it.
• Describe the “two questions, two stores” split and justify why Bigtable and BigQuery both exist in the same design.
• Design a hotspot-free Bigtable row key for device time-series and set garbage-collection policies that keep the hot store small.
• Lay out the Pub/Sub → Dataflow → {Bigtable, BigQuery, alerts} fan-out, including dedup, event-time windowing, and dead-lettering.
• Reason about cost, scale, security, and disaster recovery for a fleet from hundreds to millions of devices.
• Recognise the anti-patterns (serving point lookups from BigQuery, a timestamp-prefixed row key) before they reach production.

The two things every GCP IoT diagram gets wrong

The first thing every IoT architecture diagram on Google Cloud gets wrong is the box on the far left. It says “IoT Core,” and IoT Core has not existed since August 2023. Google retired its managed device-connectivity service — the MQTT broker, the device registry, the per-device auth — and did not replace it. So the modern Google Cloud IoT reference architecture starts with a hole where the front door used to be, and the single most important design decision you make is how you fill it: which MQTT broker terminates a million long-lived device connections, authenticates each device, and bridges their telemetry into Pub/Sub — because everything to the right of that broker is the part Google does superbly and the part this article spends most of its words on.

The second thing those diagrams get wrong is treating IoT like clickstream. It is not. Clickstream is bursty human traffic that you analyse after the fact. A device fleet is millions of always-on emitters producing dense, regular time-series, and the questions you ask of it are two completely different shapes: “show me the last reading and current status of this one asset, right now, in single-digit milliseconds” (a point lookup against a key), and “chart the temperature trend of these 50,000 assets over the last quarter and tell me which models are drifting” (an analytical scan over history). Those two access patterns do not belong in the same store. The whole architecture below exists to fan a single ingest stream into Bigtable for the hot point-lookup and BigQuery for the analytical scan — and to do command-and-control back down to the devices on the same backbone. This is the reference, built end to end on a broker tier, Pub/Sub, Dataflow, Bigtable, BigQuery, and Looker.

The business scenario

The shape of the problem is identical whether you operate 800 connected devices or 8 million; only the broker sizing and the autoscaling ceilings change. So picture an operator who manufactures and runs physical things in the field that emit data and currently cannot see them in aggregate or in real time.

A commercial refrigeration and HVAC operator is the clean example — the same pattern fits connected vehicles, smart meters, industrial machinery, agricultural sensors, or medical devices. They have tens of thousands of refrigeration units deployed across supermarkets, restaurants, and cold-chain warehouses. Each unit has a controller that knows its compressor temperature, door-open events, power draw, defrost cycles, and fault codes. Today that data lives on the unit. A technician sees it only on a site visit. When a compressor is about to fail, the first signal the business gets is a 2 a.m. phone call about spoiled inventory and a freezer full of ruined stock — a claim that can run into lakhs per incident.

What the business actually needs splits cleanly into the two access shapes that drive the entire design:

• Operational, per-device, now. A field-service dispatcher needs to pull up unit #4471 in the Pune warehouse and see its current temperature and live status in milliseconds, and the on-call platform needs to alert the instant a unit crosses a fault threshold. This is a key lookup and a last-value read, not an analytical query.
• Analytical, fleet-wide, over time. Reliability engineering needs to chart degradation curves across every unit of a given compressor model over 18 months, correlate failures with ambient conditions, and feed a predictive-maintenance model. This is a partitioned scan over billions of rows.
• Command-and-control, downlink. Having seen a problem, an operator needs to act on the device — push a new temperature setpoint, trigger a manual defrost, schedule a firmware update — and know the device acknowledged it. Data does not only flow up.
• Correctness under field conditions. Devices go offline and replay a backlog when connectivity returns, so events arrive late, out of order, and duplicated. Clocks drift. Payloads occasionally corrupt. The pipeline must not drop, double-count, or stall.
• Cost that scales with a fleet, not a demo. At millions of devices and billions of readings a day, storage layout and retention tiers are the difference between a sane bill and an absurd one.

The architecture below meets all five with managed, mostly serverless Google Cloud services behind one self-managed broker tier — so a small team runs a fleet of thousands without a streaming-infrastructure group, and the same design scales to millions of devices without a redesign.

Architecture overview

Read the data path as up (telemetry ingest), across (fan-out to two stores plus the warehouse), and down (command-and-control), with one connectivity tier you operate and everything else managed.

The connectivity tier (devices → MQTT broker → Pub/Sub). Because IoT Core is gone, devices connect over MQTT to a broker you run: the managed ClearBlade IoT Core (a drop-in successor that keeps the IoT Core device registry and MQTT API), or a self-hosted EMQX / HiveMQ cluster on GKE for full control. The broker is the part that terminates a million long-lived TLS connections, authenticates each device by per-device certificate or JWT, enforces per-device topic ACLs, and handles last-will/keep-alive for unreliable links. A thin bridge — the broker’s native Pub/Sub connector, or a sidecar — republishes each device’s telemetry onto Pub/Sub topics keyed by message type (telemetry, events, state). From this point on, the workload is pure Google Cloud and the broker is just an edge that feeds the log.

Stage 1 — Durable ingest (Pub/Sub). Pub/Sub is the decoupling buffer: it absorbs the thundering herd when a regional network blip reconnects thousands of devices at once, retains messages (default 7 days, up to 31) so any consumer can replay, and lets multiple downstream consumers read the same stream independently. Device messages carry the device ID and an event timestamp as attributes so downstream can order by event time, not arrival time. Pub/Sub schemas (Avro/Protobuf) attached to topics reject malformed payloads at publish.

Stage 2 — Stream processing and fan-out (Dataflow). A single Dataflow streaming job (Apache Beam) subscribes to the telemetry topics and is the brain of the pipeline. It parses and validates, deduplicates on (device_id, sample_time) so a device’s reconnect-replay doesn’t double-count, resolves event-time windowing with watermarks and allowed lateness so a reading that was buffered on the device for an hour lands in the correct historical window rather than “now,” enriches each reading with device metadata (model, site, install date, warranty tier) joined from a reference store, computes rolling aggregates and threshold breaches, and then fans the same record out to three sinks: it writes the latest reading and live status to Bigtable for millisecond point lookups, appends the full validated history to BigQuery via the Storage Write API (exactly-once), and emits alert events back onto a Pub/Sub topic when a threshold is crossed. Anything that fails validation goes to a dead-letter topic and a BigQuery errors table.

Stage 3a — Hot operational store (Bigtable). Bigtable holds the time-series telemetry for the operational access pattern: “give me the recent readings and current state of this device” in single-digit milliseconds. The row-key design (device_id#reverse_timestamp) puts the newest reading first, so a “latest state” read is the first row of a one-device scan, and a “last 24 hours” read is a tight contiguous scan. Bigtable serves the dispatcher console, the live-status API, and the alerting checks — none of which should ever hit a scan engine.

Stage 3b — Analytical warehouse (BigQuery). BigQuery is the analytical store of record: time-partitioned (by sample day/hour) and clustered (by model, site, device_id) tables that hold months-to-years of every reading for fleet-wide trend analysis, reliability engineering, and BigQuery ML predictive-maintenance models scored in place. Materialized views and scheduled queries roll raw readings into marts (uptime by model, fault rate by site, energy by region).

Stage 4 — Command-and-control (downlink). Operators don’t only watch; they act. A command (new setpoint, defrost, firmware target) is published to a commands Pub/Sub topic, the bridge/broker delivers it to the device’s downlink MQTT topic (MQTT QoS 1 for acknowledged delivery), and the device’s ACK flows back up the same telemetry path and is recorded. This closes the loop without a second control plane.

Stage 5 — Serving and presentation (Looker). Looker is the semantic and dashboard layer over BigQuery — fleet health, model-level reliability, energy and SLA reporting — with LookML defining metrics once and governing per-customer/per-region row-level access. The live single-device console reads Bigtable directly (via a small API) for its millisecond freshness; Looker handles the fleet analytics. BI Engine can accelerate the heavy Looker dashboards.

The flow in one breath: devices → MQTT broker (ClearBlade / EMQX on GKE) → Pub/Sub (durable buffer) → Dataflow (validate, dedupe, window, enrich, fan-out) → {Bigtable for hot point lookups, BigQuery for analytical history, Pub/Sub for alerts} → Looker + live console, with a commands topic running the other direction back through the broker to the device, and a dead-letter branch off Dataflow for anything malformed. One stream in; two stores plus a warehouse out; a control path back down.

Component breakdown

A few choices deserve the why, because they are where an IoT architecture differs from a generic streaming one.

Bigtable and BigQuery are not redundant — they answer opposite questions, and the fan-out is the point. This is the decision that defines the whole design. BigQuery is a magnificent scan engine and a terrible key-value store: a point lookup of one device’s latest reading is a full query with seconds of latency and a per-query cost, which is unacceptable for a console refresh or an alert check fired thousands of times a second. Bigtable is the opposite: a constant-low-latency key-value/range store with no good story for “scan a billion rows and GROUP BY model.” So the operational, per-asset, now-questions go to Bigtable (point lookups, last-value, recent-range), and the analytical, fleet-wide, over-time questions go to BigQuery (partitioned scans, aggregates, ML). Dataflow writes the same validated record to both in one pass. Skipping Bigtable and serving the live console from BigQuery is the most common and most expensive mistake in Google Cloud IoT.

The Bigtable row key is the entire performance story. device_id#reverse_timestamp — where the reverse timestamp is LONG_MAX − sample_millis — does three things at once: it co-locates one device’s readings contiguously (fast single-device range scan), it sorts newest-first so “current state” is the first row read (a one-row lookup, not a sort), and prefixing with device_id (high-cardinality, well-distributed) avoids the hotspotting that a raw-timestamp prefix would cause by funnelling all writes to one tablet. Use tall, narrow rows (one logical reading per row, few columns) and set per-column-family garbage-collection policies so raw samples auto-expire (maxage=90d) while a latest-state family keeps only the current version (maxversions=1). Get this key wrong and Bigtable is either hotspotted on write or slow on read.

Dataflow exists to do what a direct Pub/Sub→BigQuery subscription cannot — and IoT needs every bit of it. The zero-code BigQuery subscription is real and useful, but it is at-least-once only, applies no transformation, and writes to exactly one destination. IoT specifically needs the four things it can’t do: deduplicate a device’s reconnect-replay, window a late buffered reading into the correct historical bucket (not “now”), enrich a bare (device_id, value) with model/site/warranty metadata, and fan out to Bigtable and BigQuery and an alert topic from one stream. Beam’s event-time windowing with watermarks and allowed lateness is exactly the mechanism that makes “the 14:05 reading” correct even when a device that was offline uploads it at 16:30.

The broker is the one box you own, so treat its sizing as capacity planning, not autoscaling magic. Unlike everything to its right, the broker holds stateful long-lived connections; you size it to concurrent connections and message rate, run it HA (multi-zone GKE or managed ClearBlade), and per-device auth + topic ACLs are non-negotiable so a compromised device can only publish its own topic and receive its own commands.

Implementation guidance

Project, region, and instance layout. Put the platform in a dedicated Google Cloud project (or one per environment: iot-dev, iot-prod) under your landing-zone folder hierarchy, on a Shared VPC from the network host project. Co-locate the GKE broker cluster (if self-hosted), Dataflow workers, Bigtable instance, and BigQuery dataset in one region (e.g. asia-south1) to avoid cross-region latency and egress; pin the region to your data-residency requirement. Bigtable and BigQuery are both regional resources here — keep them together with the Dataflow job.

Infrastructure as code (Terraform). Provision the whole right-hand side declaratively so it is reproducible and reviewable; the broker is the only piece needing app-level deployment (Helm/manifests on GKE, or ClearBlade console + API).

• google_pubsub_schema + google_pubsub_topic per message type, with google_pubsub_subscription carrying dead_letter_policy.
• google_bigtable_instance (SSD, autoscaling cluster) and google_bigtable_table with column families; set GC policies via google_bigtable_gc_policy (e.g. max_age 90d on raw, max_version 1 on latest-state).
• google_bigquery_dataset + google_bigquery_table with time_partitioning on sample time, clustering on ["model","site","device_id"], and require_partition_filter = true so no one scans all history by accident.
• The Dataflow job via google_dataflow_flex_template_job (or a CI step running gcloud dataflow flex-template run) pointing at your packaged Beam pipeline, with enableStreamingEngine and bounded autoscaling.
• google_bigquery_data_transfer_config (scheduled queries) and materialized-view tables for marts; the Looker (Google Cloud core) instance with its LookML project in Git and a dedicated BigQuery service account.
• For self-hosted brokers: google_container_cluster + node pools sized for connection count, with the EMQX/HiveMQ Helm release and the Pub/Sub bridge configured via Workload Identity.

Keep the Beam pipeline in its own repo with unit tests on the transforms — Beam’s TestStream lets you assert dedup and late-data/windowing behaviour deterministically — and a CI pipeline that builds the Flex Template image and updates the streaming job.

The Dataflow pipeline, concretely. A streaming Beam pipeline (Java or Python) that: reads the telemetry subscription, extracting device_id and sample_time from message attributes for event time; deduplicates on (device_id, sample_time) with a stateful keyed dedup over a bounded window; applies fixed event-time windows with a watermark and generous allowed_lateness (devices buffer offline); enriches via a slowly-refreshed side input of device metadata (or a per-key Bigtable lookup for very large fleets); computes threshold breaches and rolling aggregates; branches invalid records to a dead-letter PubsubIO + BigQuery errors table with a reason; and then writes to three sinks in the same pipeline — BigtableIO.write() keyed device_id#reverse_timestamp for the hot store, BigQueryIO.write() with the Storage Write API (exactly-once) for history, and PubsubIO.write() to the alerts topic on breach. Prefer the Storage Write API over legacy streaming inserts — it is the current path, cheaper, and supports exactly-once.

Networking. Run Dataflow workers in the Shared VPC subnet with Private Google Access and --no_use_public_ips, so they reach Pub/Sub, Bigtable, and BigQuery over Google’s private network with no public egress. The broker is the only public ingress: front a self-hosted EMQX/HiveMQ with a TCP/TLS Load Balancer (MQTT over TLS on 8883) and tight firewall rules; managed ClearBlade exposes its own secured endpoint. Lock data services inside a VPC Service Controls perimeter so Bigtable and BigQuery cannot exfiltrate data even with valid credentials. The live-status API (reading Bigtable) sits behind a global HTTPS LB with Cloud Armor; Looker reaches BigQuery over Google’s network and its UI is gated by IAP/allowlist.

Identity and access (least privilege). Each component gets its own service account. The broker bridge SA gets pubsub.publisher on telemetry topics and pubsub.subscriber on commands only. The Dataflow worker SA gets pubsub.subscriber on its subscriptions, bigtable.user on the instance, bigquery.dataEditor on the target dataset, and pubsub.publisher on the alerts topic. The live-status API SA gets bigtable.reader on the serving app-profile and nothing else. The Looker SA gets bigquery.dataViewer + bigquery.jobUser on the serving dataset only. Devices authenticate at the broker with per-device certs/JWT — they never hold Google Cloud credentials. Humans get access through groups mapped to IAM roles. Use column-level policy tags (Dataplex/Data Catalog) to mask any customer-identifying fields in BigQuery, and Looker row-level access so each customer/region sees only its own fleet.

Enterprise considerations

Security and Zero Trust. Trust is established by identity and context at every hop, not by network position. Devices are the largest and least trustworthy population, so each authenticates individually at the broker (per-device cert/JWT), is constrained by per-device topic ACLs (it can publish only its telemetry and receive only its commands), and is never granted a Google Cloud identity — a stolen device key compromises one device, not the fleet, and is revoked at the broker. Inside Google Cloud, no service account has standing broad access; each is scoped to exactly the topics/tables it needs, and VPC Service Controls wrap Bigtable and BigQuery in a data perimeter so even a leaked key can’t move telemetry out. Dataflow runs without public IPs. The command path uses QoS 1 acknowledged delivery so control actions are confirmed, and every command is logged in Cloud Audit Logs — critical when “who told that freezer to stop cooling?” is an auditable question. Pub/Sub schemas plus the dead-letter branch contain malformed or hostile payloads instead of propagating them.

Cost optimization. IoT economics live and die on storage layout and retention, because the data volume is enormous and most of it is read rarely. The meters and the levers:

• Bigtable — billed on nodes (provisioned) + storage, not per-query. It is the priciest idle component, so size nodes to the serving load (autoscale on CPU), and crucially use GC policies to expire raw samples after the hot window (e.g. 90 days) so the hot store stays small. Bigtable holds recent data for fast access; it is not your archive.
• BigQuery — billed on bytes scanned (on-demand) or slots, plus storage. Partition + cluster so a model-level trend query scans a few partitions, not the whole table; require a partition filter; serve dashboards from materialized views and BI Engine so analysts never re-scan raw history. Use partition expiration to tier old raw data out (or to long-term storage pricing) while keeping aggregates.
• Dataflow — billed on worker time; Streaming Engine + autoscaling means you pay for current ingest rate. Cap maxNumWorkers.
• Pub/Sub — billed on throughput; cheap relative to the stores. Retention length is the lever.
• Broker — fixed GKE node cost (or ClearBlade per-device/connection pricing); size to peak concurrent connections.

The decisive move is tiering by access pattern: hot recent data in a small, GC-bounded Bigtable; full history in cheap BigQuery storage scanned only by partitioned analytical queries; dashboards served from MVs/BI Engine. Curiosity does not generate a scan bill, and you are not paying Bigtable node hours to store two years of cold samples.

Scalability. Every tier scales independently. The broker scales horizontally (more GKE nodes / managed capacity) to more concurrent connections — this is the tier you actively plan. Pub/Sub is effectively unbounded throughput and is precisely what absorbs a reconnect storm when a region’s devices come back online and replay buffers simultaneously: the log just grows and drains rather than overwhelming the processors. Dataflow autoscaling adds workers as backlog grows. Bigtable scales by adding nodes (linear throughput) and resharding tablets — provided the row key avoids hotspots. BigQuery scales storage and (with editions autoscaling) compute without capacity planning. The same design runs at thousands of devices for a mid-market operator and millions for a global one; the broker fleet and the autoscaling ceilings change, the shape does not.

Reliability and DR (RTO/RPO). The durable Pub/Sub log is the backbone of recoverability: with retention configured you can replay from a timestamp to rebuild a derived table or re-hydrate a store after a logic bug or bad deploy, and the dead-letter topic preserves anything that failed. Dataflow checkpoints state in Streaming Engine (a worker loss doesn’t lose in-flight data) and exactly-once writes mean a retry doesn’t double-count in BigQuery. RPO: within the Pub/Sub retention window, effective loss approaches zero — design RPO inside that window (e.g. 7 days); for the device side, MQTT QoS 1 + on-device buffering means a disconnected device’s readings arrive (late) when it reconnects rather than vanishing. RTO: Bigtable supports cross-region replication for an active-active or warm-standby serving store (the live console survives a regional fault); BigQuery can be a multi-region location or have critical datasets replicated; and because the entire Google Cloud side is reproducible via Terraform, you can stand the pipeline back up in a paired region in well under an hour. Most operators target minutes of RTO for the live console (replicated Bigtable), an hour or two for fleet analytics (BigQuery), and seconds of effective RPO inside the retention window. The broker tier needs its own HA/multi-zone design and a documented failover for the device endpoint (DNS/anycast) — it is the only component without a free Google-managed failover.

Observability. Cloud Monitoring + Logging give the operational picture across the whole path. The leading indicators of trouble are Pub/Sub subscription backlog / oldest-unacked-message age (processing falling behind a reconnect storm), Dataflow system lag / data freshness / watermark, Bigtable CPU utilization and p99 read/write latency (the signal a hot row key or undersized cluster is forming), BigQuery bytes scanned and slot utilization, and dead-letter depth (should be near zero). On the device side, track connected-device count and disconnect rate at the broker — a cliff there means a connectivity or auth incident, not a data one. Set SLOs on end-to-end freshness (device sample → queryable / → Bigtable-visible) and on live-console read latency, and alert on backlog age, dead-letter growth, and Bigtable p99 — those catch most real incidents early.

Governance. Dataplex / Data Catalog provide the catalog, lineage, and policy tags that drive any column masking, and describe the telemetry as discoverable data products. Looker’s LookML is the governed semantic layer — uptime, MTBF, fault rate are defined once, version-controlled in Git, and access-scoped — so reliability engineering and the customer-facing SLA report compute the same numbers. A device registry (ClearBlade’s, or your own table) is the governance backbone for the fleet: which device exists, its model/firmware/owner, its certificate state, and its decommission status. IAM through groups, audit logs (including the command path), and VPC Service Controls round out a setup that satisfies SOC 2 / ISO and data-residency requirements.

Reference enterprise example

Frostline Systems is a mid-market commercial-refrigeration operator: roughly 1,100 employees, about ₹3,000 crore in annual revenue, and a fleet of 62,000 connected units (supermarket display cases, restaurant walk-ins, and cold-chain warehouse systems) across India and the Gulf. Each unit samples compressor temperature, door state, power draw, defrost cycle, and fault codes every 10 seconds — roughly 535 million readings a day, peaking around 9,000 messages/sec with reconnect bursts to 40,000/sec when a regional ISP flaps. Their old model was reactive: the first signal of a failing compressor was spoiled stock and an emergency call-out, and a single warehouse spoilage event could cost ₹6–8 lakh plus the customer relationship.

They built exactly this architecture in asia-south1 (Mumbai) for residency, and chose ClearBlade IoT Core as the managed broker successor so the existing units’ IoT-Core-style MQTT clients connected with minimal firmware change — keeping the per-device certificate auth and device registry they already had. The broker bridges three Pub/Sub topics (telemetry, events, commands), each with an Avro schema and a dead-letter topic. A single Dataflow streaming job (Streaming Engine, autoscaling 5→60 workers) deduplicates on (device_id, sample_time), applies 1-minute event-time windows with 15 minutes of allowed lateness (units buffer readings when offline and replay on reconnect), enriches each reading with model/site/warranty metadata, routes ~0.2% malformed messages to a dead-letter table, and fans out to all three sinks: Bigtable (row key device_id#reverse_timestamp, raw column family maxage=90d, latest-state family maxversions=1, autoscaling on a 3-node minimum), BigQuery (partitioned by sample-hour, clustered on model, site, device_id, via Storage Write API exactly-once), and an alerts topic on threshold breach.

The dispatcher console and the alerting service read Bigtable directly through a small Cloud Run API; reliability engineering and the customer SLA reports run on BigQuery, where a BigQuery ML failure-prediction model scores each compressor’s recent trend nightly. Looker (Google Cloud core) serves fleet-health, model-reliability, and per-customer SLA dashboards, with row-level access so each supermarket chain sees only its own units. Operators push setpoint changes and manual-defrost commands from Looker actions through the commands topic; the broker delivers them at QoS 1 and the device ACK is recorded.

The outcome after two quarters:

• Live console freshness: ~3–4 seconds device-sample-to-visible in Bigtable; single-device read p99: ~9 ms — a dispatcher pulls up any of 62,000 units instantly.
• Analytical history: the full 535M-readings/day stream lands in BigQuery and stays queryable for 18 months; a model-level degradation query over a quarter scans tens of GB (partitioned + clustered), not the multi-TB table.
• Predictive maintenance working: the BQML model now flags ~70% of compressor failures 5–10 days ahead, converting emergency call-outs into scheduled service. Spoilage incidents on monitored units dropped sharply — the single biggest line on the business case.
• Reconnect storms absorbed: a regional ISP outage that reconnected ~8,000 units at once produced a 40,000/sec replay burst; Pub/Sub buffered it and Dataflow drained it in minutes with zero data loss and no double-counting, thanks to dedup + windowing.
• Cost: the platform runs about ₹14–17 lakh/month — the ClearBlade broker and Bigtable nodes are the largest fixed lines; BigQuery cost stayed bounded because dashboards hit MVs and analytical queries are partitioned, and Bigtable stayed small because GC policies expire raw samples at 90 days while history lives cheaply in BigQuery.
• Operability: a six-person platform team runs it on four alerts that matter — Pub/Sub backlog age, dead-letter depth, Bigtable p99, and broker connected-device count — and rebuilt the Bigtable hot store once after a key-design fix by replaying Pub/Sub, with no loss.

The decision that paid off most was refusing to serve the live console from BigQuery. Fanning Dataflow out to Bigtable (for the millisecond per-device lookup) and BigQuery (for the fleet-wide scan) — instead of forcing both questions onto one store — is the entire difference between a console a dispatcher actually uses and a per-query bill that scales with every refresh.

When to use it

Use this architecture when you operate a fleet of devices that emit time-series telemetry and you need both per-device millisecond operational lookups and fleet-wide analytical history, and you need an acknowledged way to send commands back to devices. It is the reference for connected products, fleet/asset telemetry, smart metering, industrial and agricultural IoT, predictive maintenance, and cold-chain/medical-device monitoring. It scales from a few hundred devices to millions because every tier behind the broker is managed and elastic, and the broker tier scales horizontally.

The trade-offs and anti-patterns:

• Don’t serve point lookups from BigQuery. A live single-device console or a high-frequency alert check against BigQuery is slow and expensive per query. That access pattern belongs in Bigtable. Conversely, don’t try fleet-wide GROUP BY model scans in Bigtable — that’s BigQuery’s job. Putting the wrong question on the wrong store is the defining IoT mistake.
• Don’t skip Dataflow with a direct BigQuery subscription if you need more than raw landing. The zero-code subscription is at-least-once, single-destination, and transform-free. IoT needs dedup of reconnect-replays, correct late-data windowing, enrichment, and fan-out to two stores plus alerts — that’s Dataflow.
• Don’t treat the broker as an afterthought. It is the one stateful, self-managed, capacity-planned tier and the entire security perimeter for untrusted devices. Per-device auth and topic ACLs are mandatory; HA and a device-endpoint failover plan are mandatory.
• Don’t get the Bigtable row key wrong. A timestamp-prefixed key hotspots writes onto one tablet and throttles the whole fleet. Prefix with the high-cardinality device_id and reverse the timestamp.
• Don’t store everything hot forever. Bigtable is the recent store; set GC policies and let BigQuery be the cheap archive. Skipping retention tiering is how the bill explodes.

Alternatives, and when they fit better. If your fleet is small (thousands, not millions) and your needs are purely analytical with no millisecond console, you can drop Bigtable and run the simpler Pub/Sub → Dataflow → BigQuery real-time-analytics pattern, or even the zero-code Pub/Sub→BigQuery subscription for raw landing — cheaper and less to operate. If you need sub-second time-series with built-in downsampling and a Prometheus-style query layer rather than a key-value store, a managed time-series database is a closer fit than Bigtable for some metrics workloads. If your team standardises on Kafka semantics and open-source streaming, Pub/Sub Lite or self-managed Kafka on GKE plus Dataproc/Flink substitutes for the Pub/Sub + Dataflow pair at the cost of more operations. And if you want to push intelligence to the edge — filtering, aggregating, or running ML on the device before sending — pair this with on-device inference and an edge gateway so only meaningful events traverse the network, which materially cuts Pub/Sub and Dataflow volume for very large or bandwidth-constrained fleets. But for a managed, scalable Google Cloud IoT platform that answers both the per-device and the fleet-wide question from one ingest stream, broker → Pub/Sub → Dataflow → {Bigtable, BigQuery} → Looker is the reference to reach for first.

Going deeper

This section is for the reader who already follows the architecture and wants the mechanics underneath the decisions.

The Cloud IoT Core retirement, precisely. Google announced the retirement in August 2022 and shut the service down on 16 August 2023. What died was specifically the managed device-connectivity layer: the MQTT/HTTP bridge, the per-device device registry, per-device credential management, and the built-in Pub/Sub forwarding. What did not change is everything to the right of it — Pub/Sub, Dataflow, Bigtable, and BigQuery are exactly as they were. Google’s own migration guidance pointed customers at partners, and ClearBlade licensed an API-compatible “IoT Core” so existing device firmware — which speaks a specific MQTT dialect and JWT auth scheme — can reconnect with minimal change. The migration is mostly an export of the device manifest (device IDs, public keys, metadata) from the old registry into the successor’s registry, plus repointing the device’s MQTT endpoint hostname. If you are greenfield, you skip the compatibility question entirely and pick a broker on its merits (EMQX and HiveMQ are the common self-hosted choices; ClearBlade and others are managed). The lasting lesson: on GCP, device connectivity is now a “you” problem, and ingest onward is a “Google” problem.

The Bigtable row key, at tablet level. Bigtable stores rows sorted lexicographically by key and splits that keyspace into tablets, each served by one node. A tablet splits when it grows hot or large, and the split point is a key boundary. This is why a monotonically increasing prefix — a raw timestamp, an auto-increment ID — is pathological: every new write lands at the same right-hand edge of the keyspace, so it always hits the same tablet on the same node while the other N−1 nodes sit idle. That is right-edge hotspotting, and it caps your write throughput at one node no matter how many you provision. Prefixing with device_id (thousands to millions of distinct, well-distributed values) scatters writes across the whole keyspace and therefore across all tablets. Reversing the timestamp within a device (LONG_MAX − sample_millis) then makes the newest sample sort first inside that device’s contiguous range, so “latest state” is a single-row read and “last 24 hours” is a short forward scan. Bigtable’s Key Visualizer is the tool that shows a heatmap of read/write pressure across the keyspace — a bright vertical stripe is a hotspot forming. An alternative to a natural high-cardinality prefix is salting (prefixing a hash bucket), but a real device_id is almost always the better, self-documenting choice.

The Bigtable “latest state” row is your digital twin. The single most-recent row per device — the one the maxversions=1 state family keeps — is, in practice, the device’s digital twin: the current known temperature, door state, firmware, connection status, and last-seen time of the physical asset, readable in single-digit milliseconds. You do not need a separate “digital twin” product; the hot store’s latest-state view is the twin, kept live by the same Dataflow write that feeds history. Command-and-control then operates on the twin — an operator sets a desired state (new setpoint), and the loop reconciles when the device’s reported state (its ACK and next telemetry) flows back up.

Watermarks, lateness, and why dedup needs state. Beam separates event time (when the device sampled the reading) from processing time (when Dataflow saw it). The watermark is Beam’s moving estimate of “event time up to which I believe I’ve seen everything.” A window (say a fixed one-minute event-time bucket) fires its result when the watermark passes the window’s end. A reading that arrives after that — a device that buffered offline and replayed an hour later — is a late element; withAllowedLateness(Duration) tells Beam how long to keep the window’s state around to accept and re-fire for stragglers. Set it too low and legitimate replayed readings are dropped from their correct window; set it too high and Beam must retain window state (memory and cost) for that whole horizon. Dedup is a separate stateful operation: a keyed state cell per (device_id, sample_time) remembers “seen,” with a timer to expire the cell after the dedup horizon so state doesn’t grow unbounded. This is precisely why MQTT QoS 1 (below) is survivable — the at-least-once broker delivery produces duplicates that this state removes.

Three different “exactly-once” guarantees — don’t conflate them. (1) Pub/Sub exactly-once delivery is a subscription setting that an ack’d message won’t be redelivered within the ack deadline — it is about the transport. (2) The BigQuery Storage Write API offers exactly-once via named streams and offset tracking — it is about the sink write. (3) Dataflow end-to-end exactly-once comes from deterministic keyed dedup plus Streaming Engine checkpointing of state, so a worker crash and retry doesn’t double-apply. In this design you rely on the Storage Write API’s default stream for streaming into BigQuery (higher-throughput and cheaper than the legacy tabledata.insertAll streaming inserts, and the current recommended path), and on your own (device_id, sample_time) dedup for the semantic guarantee that a device’s replay doesn’t double-count — because two genuinely separate deliveries of the same reading are a data duplicate, not a transport one, and only application-level dedup removes them.

Ordering, schemas, and evolution. Pub/Sub ordering keys guarantee in-order delivery per key (e.g. per device_id) at the cost of throughput on any hot key — most IoT pipelines don’t need it because Dataflow reorders by event time anyway. Pub/Sub schemas (Avro/Protobuf) validate payloads at publish; schema evolution is safe when you add optional fields with defaults, and breaks consumers when you remove or retype a field — so version the schema (reading-v1, reading-v2) and let the Dataflow parser tolerate both shapes during a rollout rather than doing a flag-day cutover across a whole fleet at once.

Bigtable app profiles and replication. An app profile is a named routing-and-priority policy attached to your requests. Run the serving path (dispatcher console, alert checks) and the ingest path (Dataflow writes) on separate app profiles so a heavy ingest burst can’t starve a dispatcher’s read, and so you can pick single-cluster routing for read-your-writes consistency on the serving profile while ingest uses multi-cluster. Cross-region replication adds a second cluster for a warm-standby serving store (this is what lets the live console survive a regional fault) — but replication consumes CPU on both clusters to apply writes, so size for the replicated load, not just the primary.

Practice challenges

These build the right-hand (Google Cloud) side of the architecture piece by piece. No live cluster is assumed — read them as design-and-command exercises; every command is real and current. Replace PROJECT_ID and other placeholders with your own values.

1. Beginner — stand up the ingest topics. Create the four Pub/Sub topics the broker bridge publishes to (telemetry, events, state, commands), plus a dead-letter topic and a Dataflow subscription that dead-letters after 5 failed deliveries.

<details> <summary>Solution</summary>

gcloud pubsub topics create telemetry events state commands
gcloud pubsub topics create telemetry-dlq

gcloud pubsub subscriptions create telemetry-sub \
  --topic=telemetry \
  --dead-letter-topic=telemetry-dlq \
  --max-delivery-attempts=5 \
  --ack-deadline=60

# The Pub/Sub service agent must be able to publish to the DLQ and ack the source
PROJECT_NUMBER="$(gcloud projects describe PROJECT_ID --format='value(projectNumber)')"
PUBSUB_SA="service-${PROJECT_NUMBER}@gcp-sa-pubsub.iam.gserviceaccount.com"
gcloud pubsub topics add-iam-policy-binding telemetry-dlq \
  --member="serviceAccount:${PUBSUB_SA}" --role="roles/pubsub.publisher"
gcloud pubsub subscriptions add-iam-policy-binding telemetry-sub \
  --member="serviceAccount:${PUBSUB_SA}" --role="roles/pubsub.subscriber"

Why: a topic per message type keeps schemas and consumers independent; the dead-letter policy quarantines poison messages instead of blocking the whole subscription, and the DLQ only works once the Pub/Sub service agent is allowed to move messages into it. </details>

2. Beginner → Intermediate — enforce a payload schema. Define an Avro schema for a reading and bind it to the telemetry topic so malformed publishes are rejected at the door.

<details> <summary>Solution</summary>

gcloud pubsub schemas create reading-v1 \
  --type=avro \
  --definition='{"type":"record","name":"Reading","fields":[{"name":"device_id","type":"string"},{"name":"sample_time","type":"long"},{"name":"temp_c","type":"double"},{"name":"door_open","type":"boolean"}]}'

gcloud pubsub topics update telemetry \
  --schema=reading-v1 \
  --message-encoding=json

Why: the schema rejects bad payloads at publish time — the cheapest possible place to stop corrupt data, before it ever costs a Dataflow cycle or lands in a store. </details>

3. Intermediate — the Bigtable hot store. Create an autoscaling SSD Bigtable instance and a readings table with a raw family that garbage-collects after 90 days and a state family that keeps only the latest version. State the row key you would use.

<details> <summary>Solution</summary>

gcloud bigtable instances create iot-hot \
  --display-name="IoT hot store" \
  --cluster-config=id=iot-hot-c1,zone=asia-south1-a,autoscaling-min-nodes=3,autoscaling-max-nodes=10,autoscaling-cpu-target=60

# cbt reads project/instance from ~/.cbtrc, or pass them as flags:
cbt -project=PROJECT_ID -instance=iot-hot createtable readings
cbt -project=PROJECT_ID -instance=iot-hot createfamily readings raw
cbt -project=PROJECT_ID -instance=iot-hot createfamily readings state
cbt -project=PROJECT_ID -instance=iot-hot setgcpolicy readings raw maxage=90d
cbt -project=PROJECT_ID -instance=iot-hot setgcpolicy readings state maxversions=1

Row key: device_id#reverse_timestamp, where reverse_timestamp = LONG_MAX − sample_millis.

Why: the device_id prefix spreads writes across tablets (no hotspot), the reversed timestamp puts newest-first so “current state” is one row, and per-family GC keeps the hot store small — you pay for nodes to serve recent data, not to archive years of it. (Default cluster storage is SSD.) </details>

4. Intermediate — the BigQuery history table. Create a table partitioned by hour on sample_time, clustered on model, site, device_id, that refuses full-table scans.

<details> <summary>Solution</summary>

bq mk --dataset --location=asia-south1 PROJECT_ID:iot

bq mk --table \
  --time_partitioning_field=sample_time \
  --time_partitioning_type=HOUR \
  --clustering_fields=model,site,device_id \
  --require_partition_filter \
  PROJECT_ID:iot.readings \
  device_id:STRING,sample_time:TIMESTAMP,model:STRING,site:STRING,temp_c:FLOAT,door_open:BOOLEAN

Why: partitioning + clustering means a model-level trend query scans a few partitions instead of the whole table, and require_partition_filter makes the expensive accident — a WHERE-less full scan over billions of rows — impossible rather than merely discouraged. </details>

5. Advanced — least privilege for the Dataflow worker. The Dataflow worker service account must read the telemetry subscription, write Bigtable, write BigQuery, and publish alerts — and nothing else. Bind exactly those, at the resource level where you can.

<details> <summary>Solution</summary>

SA="dataflow-worker@PROJECT_ID.iam.gserviceaccount.com"

# Pub/Sub: subscribe to telemetry, publish to alerts (resource-scoped)
gcloud pubsub subscriptions add-iam-policy-binding telemetry-sub \
  --member="serviceAccount:${SA}" --role="roles/pubsub.subscriber"
gcloud pubsub topics add-iam-policy-binding alerts \
  --member="serviceAccount:${SA}" --role="roles/pubsub.publisher"

# Bigtable: instance-scoped user (reads + writes data, no admin)
gcloud bigtable instances add-iam-policy-binding iot-hot \
  --member="serviceAccount:${SA}" --role="roles/bigtable.user"

# BigQuery: dataset-scoped editor (set on the dataset, not the project)
bq add-iam-policy-binding \
  --member="serviceAccount:${SA}" --role="roles/bigquery.dataEditor" \
  PROJECT_ID:iot

# The worker also needs the project-level role to run at all:
gcloud projects add-iam-policy-binding PROJECT_ID \
  --member="serviceAccount:${SA}" --role="roles/dataflow.worker"

Why: every grant is the minimum the fan-out needs, scoped to a named resource where the API allows it — so a leaked worker key can touch four specific resources, not the whole project. </details>

6. Advanced — replay to rebuild the hot store. A row-key bug corrupted the Bigtable table. You’ve fixed the pipeline. Rebuild the last 7 days of hot state from the durable log without touching the devices, and explain why this works.

<details> <summary>Solution</summary>

# 1. Recreate the table with the corrected schema (see challenge 3).
# 2. Point the fixed pipeline at the same subscription, then rewind it in time:
gcloud pubsub subscriptions seek telemetry-sub \
  --time=2026-07-12T00:00:00Z

Redeploy the corrected Dataflow job; it re-reads from the seek time and rewrites Bigtable correctly. BigQuery history is idempotent on (device_id, sample_time), so the replay does not double-count it.

Why: Pub/Sub retention makes the subscription a rewindable source of truth — the log, not the store, is authoritative, so any derived store can be rebuilt by replay. It only works if the target time is inside the retention window and the subscription was created to retain acked messages (--retain-acked-messages with a long-enough --message-retention-duration). </details>

Common beginner mistakes

• “I’ll just use Cloud IoT Core.” It was retired on 16 August 2023 and there is no drop-in Google replacement. If a tutorial or diagram shows an “IoT Core” box on the left, it predates the shutdown. Right model: you now own the broker — a managed partner (ClearBlade) or self-hosted EMQX/HiveMQ on GKE — and Google’s managed services begin at Pub/Sub.
• “Pub/Sub is where the data lives.” Pub/Sub is a transient buffer (7 days default, 31 max), not a database — you can’t run analytics on it, and messages age out. Right model: Pub/Sub is the durable conveyor belt between the broker and processing; the data lives in Bigtable (recent) and BigQuery (history).
• “One database should handle both the live view and the analytics.” Beginners reach for whichever store they know — usually BigQuery, because SQL is familiar — and try to make it do both. A per-device console refresh then becomes a multi-second, per-query-billed full query. Right model: two questions, two engines — point lookups to Bigtable, fleet scans to BigQuery, both written from one Dataflow pass.
• “Bigtable and BigQuery are basically the same Google product.” The names collide but the engines are opposite: Bigtable is a low-latency key-value/range store with no GROUP BY; BigQuery is a scan-and-aggregate warehouse with seconds of point-lookup latency. Right model: learn them as complementary, not interchangeable — the whole design hinges on the difference.
• “The row key should start with the timestamp so data is time-ordered.” A time-prefixed key sends every write to the same tablet (right-edge hotspotting) and throttles the entire fleet to one node. Right model: prefix with the high-cardinality device_id to spread writes, then reverse the timestamp within a device for newest-first reads.
• “Each device needs a Google Cloud service account / IAM credentials.” A million GCP credentials are both impossible to manage and a catastrophic blast radius. Right model: devices authenticate at the broker with per-device X.509 certs or JWT; they never hold a Google Cloud identity, and a stolen device key compromises one device and is revoked at the broker.
• “Event time is just when the row hits BigQuery.” Treating arrival time as the reading’s timestamp silently mis-buckets every replayed reading — a device that was offline for an hour would have all its data stamped “now.” Right model: event time is when the device sampled the reading (carried in the message attributes); Dataflow windows on event time with allowed lateness so a late reading lands in its correct historical minute.

Glossary

• MQTT — a lightweight publish/subscribe messaging protocol built for constrained devices and unreliable networks; the lingua franca of IoT device connectivity.
• MQTT broker — the server that accepts device MQTT connections, authenticates them, and routes messages; on GCP you now run or buy this (ClearBlade, EMQX, HiveMQ) because IoT Core is gone.
• QoS (Quality of Service, MQTT) — the delivery guarantee level: 0 (at-most-once, fire-and-forget), 1 (at-least-once, may duplicate), 2 (exactly-once, most overhead). IoT typically uses QoS 1 — which is why downstream dedup is required.
• Cloud IoT Core — Google’s former managed device-connectivity service (broker + device registry + auth), retired 16 August 2023 with no direct Google replacement.
• ClearBlade IoT Core — a third-party, API-compatible successor to Google’s IoT Core that lets existing device firmware reconnect with minimal change.
• EMQX / HiveMQ — popular MQTT brokers you can self-host (e.g. on GKE) when you want full control of the connectivity tier.
• Device registry — the system of record for the fleet: which devices exist, their keys/certs, model, firmware, and status. The governance backbone.
• Digital twin — a live software representation of a physical device’s current state; here, the latest-state row per device in Bigtable is the twin.
• Uplink / downlink — telemetry flowing up from devices to the cloud; commands flowing down from the cloud to devices.
• Command-and-control — the downlink path that sends acknowledged actions (setpoint change, defrost, firmware target) to a device.
• Telemetry — the stream of readings a device emits (temperature, power draw, door state, fault codes).
• Pub/Sub — Google’s fully managed, horizontally scalable message queue; the durable buffer between the broker and processing.
• Topic / subscription — a Pub/Sub topic is a named stream you publish to; a subscription is a named reader position, so multiple consumers read the same topic independently.
• Dead-letter topic — a side topic that captures messages a subscription failed to process after N attempts, so poison messages don’t block the pipeline.
• Message attributes — key/value metadata on a Pub/Sub message (e.g. device_id, sample_time) used for routing and event-time ordering.
• Schema (Avro/Protobuf) — a contract attached to a topic that validates message structure at publish time, rejecting malformed payloads early.
• Dataflow — Google’s managed runner for Apache Beam pipelines; the stream-processing “brain” that validates, dedupes, windows, enriches, and fans out.
• Apache Beam — the open-source SDK for defining unified batch/stream pipelines that Dataflow executes.
• Streaming Engine — a Dataflow mode that moves shuffle and state off the workers into the service, enabling smoother autoscaling and checkpointing.
• Event time vs processing time — event time is when a reading was sampled on the device; processing time is when Dataflow saw it. They diverge when devices replay buffered data.
• Watermark — Beam’s moving estimate of “I’ve now seen all events up to this event time,” used to decide when a window can fire.
• Windowing — grouping stream elements into finite buckets (e.g. fixed one-minute event-time windows) so aggregates can be computed over an unbounded stream.
• Allowed lateness — how long Beam keeps a window’s state to accept stragglers after the watermark passes, so replayed readings still land in the correct window.
• Deduplication — removing repeated deliveries of the same reading (keyed on (device_id, sample_time)), necessary because QoS 1 and reconnect-replays create duplicates.
• Bigtable — Google’s low-latency, horizontally scalable key-value / wide-column store; here the hot store for per-device millisecond lookups and live state.
• Row key — the single sorted key that determines where a Bigtable row lives and how fast it’s read; device_id#reverse_timestamp is the design that avoids hotspots.
• Hotspotting — funnelling too many reads/writes onto one tablet/node (classically from a monotonic key prefix), throttling the whole instance.
• Tablet — a contiguous range of the Bigtable keyspace served by one node; the unit that splits and rebalances as data grows.
• Column family — a group of columns in Bigtable stored and configured together (e.g. a raw family and a state family), each with its own GC policy.
• Garbage-collection (GC) policy — a per-family rule that auto-expires cells by age (maxage=90d) or version count (maxversions=1), keeping the hot store small.
• App profile — a named Bigtable routing/priority policy; separating serving and ingest profiles stops a write burst from starving reads.
• BigQuery — Google’s serverless analytical data warehouse; here the store of record for months-to-years of full history, scanned for fleet-wide trends.
• Partitioning — splitting a BigQuery table by a time (or integer) column so queries scan only relevant partitions, cutting cost and time.
• Clustering — sorting data within partitions by chosen columns (model, site, device_id) so filtered queries read fewer blocks.
• Materialized view (MV) — a precomputed, auto-refreshed query result BigQuery serves cheaply, so dashboards don’t re-scan raw history.
• Storage Write API — BigQuery’s current high-throughput streaming ingestion path, supporting exactly-once; preferred over the legacy streaming inserts.
• Exactly-once — a guarantee that each record is applied once despite retries; achieved here via dedup + Storage Write API + Dataflow checkpointing.
• BigQuery ML (BQML) — training and scoring ML models with SQL inside BigQuery, so predictive-maintenance models run where the data already lives.
• Looker / LookML — Google’s BI and semantic layer; LookML defines metrics once (uptime, MTBF, fault rate) and governs per-customer row-level access.
• BI Engine — an in-memory acceleration layer for BigQuery that speeds up heavy dashboard queries.
• Row-level security — access rules that restrict which rows a user can see (e.g. each customer sees only its own units).
• VPC Service Controls — a data-exfiltration perimeter around services like Bigtable and BigQuery, so even a valid credential can’t move data outside it.
• Private Google Access — lets VMs/workers without public IPs reach Google APIs over Google’s private network.
• Workload Identity — binds a Kubernetes service account to a Google service account so GKE pods (the broker bridge) get GCP credentials without static keys.
• Reconnect storm — a flood of devices reconnecting and replaying buffered data at once (e.g. after a regional outage), absorbed by Pub/Sub.
• RTO / RPO — Recovery Time Objective (how fast you restore service) and Recovery Point Objective (how much data you can afford to lose); bounded here by Bigtable replication and the Pub/Sub retention window.
• Edge computing — filtering, aggregating, or running ML on the device or a local gateway before sending, to cut network and cloud volume for very large fleets.