Every cloud has a “hello world” serverless demo, and none survive contact with production. The demo is a function behind a URL. Production is a single-page application that must load fast from Chennai and Chicago, an API that must authenticate every request without a session store, a payment webhook that must never be processed twice, a nightly report that must run though no server exists at 02:00, and a bill finance expects to fall on quiet weekends. The gap between demo and system is architecture, and on Google Cloud it has a well-worn shape: Cloud Storage + Cloud CDN for the static shell, Cloud Run for the APIs, Identity Platform for tokens, Firestore for state, Pub/Sub and Cloud Tasks for everything that should not happen inside a user’s request, with Secret Manager and Cloud Trace/Logging/Monitoring wrapped around it.
This article builds that system end to end, decision by decision. Not “use Cloud Run” but which concurrency, which min-instances setting and what each idle instance costs; not “use Firestore” but when its 1 MiB document cap and per-document write limits should push you to Cloud SQL; not “use Pub/Sub” but why push subscriptions 403 on day one and why a rate-limited Cloud Tasks queue — not a topic — belongs in front of your email provider. Every component gets both the gcloud command and the Terraform resource.
By the end you can draw the full request path from a browser to a Firestore document and back, defend every hop in a design review, price the system within a few dollars a month, and — the part that matters — say precisely when this architecture wins and when a boring VM or GKE cluster beats it.
What problem this solves
The traditional way to host a web app is to rent computers: VMs (or Kubernetes nodes) running 24×7 whether they serve one request or one million. That model has five production pains this architecture removes structurally, not by tuning:
| Serverful pain | In production | The serverless answer here |
|---|---|---|
| Paying for idle | An e2-standard-2 at ~$49/mo serves 40 req/day on an internal tool |
Cloud Run bills vCPU-seconds and GiB-seconds only while requests run; scale to zero at night |
| Capacity guessing | Campaign spikes saturate the 3-VM group at 09:03; the autoscaler adds a VM in 4–6 min — too late | Cloud Run adds instances in seconds, request-driven; max-instances caps the blast radius |
| Undifferentiated ops | OS patching, SSH keys, nginx configs, cert renewals, image rebuilds | No OS to patch; managed TLS on the LB; deploy = new immutable revision |
| Synchronous coupling | Checkout calls email + analytics inline; a slow SMTP provider takes down checkout | Pub/Sub and Cloud Tasks absorb the work; the request returns in 80 ms; retries happen off the hot path |
| State on the box | Sessions in local memory; scale-in logs users out; disk fills with uploads | JWTs make instances stateless; files go to Cloud Storage; data goes to Firestore |
Who hits this: teams shipping SaaS dashboards, internal tools, mobile/web backends and event-driven integrations — spiky traffic, small platform team. Who should pause: steady 24×7 high load (always-busy compute can cost more on Cloud Run than committed-use GKE), sub-10 ms latency budgets, massive long-lived connection fan-out; the break-even is quantified under Cost & sizing.
Learning objectives
By the end of this article you can:
- Serve an SPA from a Cloud Storage backend bucket behind the global load balancer with Cloud CDN, handling deep-link 404s and cache invalidation.
- Configure Cloud Run deliberately — concurrency, min/max instances, CPU allocation, execution environment, timeout — and explain what each does to latency and the bill.
- Explain the cold start lifecycle and apply the mitigations in the right order.
- Verify Identity Platform ID tokens statelessly in middleware; choose Identity Platform vs Firebase Auth vs IAP.
- Choose Firestore native vs Datastore mode vs Cloud SQL on data-shape and limit criteria; model documents around per-document write limits.
- Wire Pub/Sub push to Cloud Run with OIDC auth and dead-letter topics; put Cloud Tasks in front of rate-limited downstreams; schedule cron with Cloud Scheduler.
- Deliver configuration with Secret Manager (env vs volume, version pinning, rotation).
- Instrument with Cloud Trace, Logging and Monitoring; build a monthly cost estimate from unit prices and argue when serverless loses.
Prerequisites & where this fits
You should be comfortable with HTTP, containers (build an image from a Dockerfile), one backend language, and gcloud fundamentals at the level of our gcloud CLI quickstart. Know what a service account is — if not, read GCP IAM & service accounts with least privilege first, because every arrow in this architecture is authorized by one.
This is the integration layer over two component deep-dives: Cloud Run explained for the compute runtime, Pub/Sub and event-driven architecture for messaging semantics. Here we make the system-level decisions: what goes where, how pieces authenticate to each other, what the assembly costs. Still choosing between Cloud Run, GKE and Compute Engine? That fork is its own article.
Core concepts
The mental model: a serverless web app is a set of stateless request handlers glued together by managed queues, with all state pushed into managed services. Every box in the diagram either holds no state and can be created/destroyed freely, or holds state and is someone else’s operational problem. Nothing gets SSH’d into, patched, or paged on for disk space.
Every moving part, its job, and — critically — its scaling and pricing unit:
| Component | Role | Scaling unit | You pay per | Fails how |
|---|---|---|---|---|
| Global external ALB | One anycast IP; TLS; routes / → bucket, /api/* → Cloud Run |
Global, automatic | Forwarding rule ($/hr) + data processed | Misrouted URL map, cert not provisioned |
| Cloud CDN | Caches SPA assets at ~200 edge PoPs | Per cache hit | Cache egress GiB + lookups | Stale index.html, misses on unversioned assets |
| Cloud Storage | Origin for SPA build artifacts | N/A (object store) | GiB stored + ops | Deep links 404, wrong Cache-Control |
| Identity Platform | OIDC provider: sign-in, tokens, MFA, tenants | Per MAU | MAU tiers (free <50k) | Misvalidated JWTs, wrong aud |
| Cloud Run (API) | Stateless handlers in containers | Instance (many req each) | vCPU-s + GiB-s + requests | Cold starts, concurrency saturation, 429 at max |
| Firestore | Document DB, strong consistency | Automatic | Doc reads/writes/deletes + GiB | Hot documents, 1 MiB cap |
| Pub/Sub | Event fan-out, decoupling | Automatic | GiB of throughput | Push auth 403, redelivery storms |
| Cloud Tasks | Rate-limited, de-duplicated queues | Per queue config | Per million operations | Wrong dispatch rate, task >1 MB |
| Cloud Scheduler | Cron without a server | Per job | Per job/mo (3 free) | Wrong time zone, missing OIDC token |
| Secret Manager | Versioned secrets, IAM-gated | N/A | Version-months + accesses | latest drift, missing accessor role |
| Trace / Logging / Monitoring | Waterfalls, logs, alerts | Automatic | GiB ingested, spans | Log-cost surprise, uncorrelated logs↔traces |
The second load-bearing concept is the synchronous/asynchronous split: everything a user is actively waiting for stays on the request path and completes in tens to hundreds of milliseconds; everything else leaves through a topic or queue. Getting this split right is 80% of the architecture:
| Work | Path | Mechanism | Why |
|---|---|---|---|
| Read a dashboard, submit a form | Synchronous | Browser → LB → Cloud Run → Firestore | User is waiting |
| Email/SMS, a 50 req/s third-party API | Async, rate-limited | Cloud Run → Cloud Tasks → worker | Downstream has a rate contract; needs retry + de-dup |
| “Order placed” → analytics, inventory, CRM | Async, fan-out | Cloud Run → Pub/Sub → N push subs | Many consumers; producer must not know them |
| Thumbnail every upload | Async, event-driven | GCS event → Eventarc → Cloud Run | React to infrastructure events |
| Nightly aggregation, 02:00 IST | Scheduled | Cloud Scheduler → Cloud Run | Cron with no server |
Keep the taxonomy straight: Pub/Sub is one-to-many event distribution, Cloud Tasks is one-to-one dispatch under sender-controlled rate limits, Eventarc is Google-infrastructure events as CloudEvents (Pub/Sub under the hood), Cloud Scheduler is time as a trigger. Use Pub/Sub for all four and you will reimplement rate limiting badly inside subscribers.
The edge: SPA on Cloud Storage behind the global load balancer and Cloud CDN
The static shell — index.html, hashed bundles, images — should never touch compute. The canonical pattern is a backend bucket: the global external ALB routes / to a Cloud Storage bucket with Cloud CDN, and /api/* to a serverless NEG pointing at Cloud Run. One domain, one anycast IP, one certificate — and no CORS, because SPA and API share an origin; split-domain setups burn weeks on preflights and cookie scoping that path routing deletes.
Know the alternatives — this is the first real decision:
| Option | CDN | SPA deep-link fallback | Custom domain + TLS | CORS | Cost floor | Choose when |
|---|---|---|---|---|---|---|
| GCS backend bucket + ALB + Cloud CDN | Cloud CDN | 404→index.html hack (status stays 404) |
Managed certs on LB | No | LB ≈ $18/mo | You need the LB anyway (Cloud Run, Cloud Armor, one domain) |
| Firebase Hosting | Built-in | Native rewrites (clean 200) |
Free managed certs | Only if API elsewhere (can proxy Cloud Run) | ~$0 small | SPA-first; clean rewrites, previews |
| Cloud Run serving static | Only via LB+CDN | Framework routing | LB or run.app |
No | Instance time | SSR (Next.js/Nuxt) — the shell is rendered |
| GCS website endpoint, no LB | No | 404 page only | No HTTPS custom domain | Yes | ~$0 | Demos only |
The classic trap in row 1: a backend bucket serves objects, and /orders/42 is not an object. Setting the bucket’s not-found page to index.html loads the shell and client-side routing takes over — but the status is still 404, which poisons uptime checks and SEO; the URL map cannot rewrite arbitrary paths to /index.html either (rewrites handle prefixes only). If deep-link 404s matter, host the shell on Firebase Hosting (first-class rewrites) and keep the LB for /api/*; otherwise accept 404-with-correct-body and exclude those routes from uptime checks. Decide in week one, not after the SEO audit.
Cache behaviour is the second decision — Cloud CDN’s cache mode decides who controls TTLs:
| Setting | Values | Default | Production guidance | Gotcha |
|---|---|---|---|---|
| Cache mode | CACHE_ALL_STATIC, USE_ORIGIN_HEADERS, FORCE_CACHE_ALL |
CACHE_ALL_STATIC |
USE_ORIGIN_HEADERS — set explicit Cache-Control per object |
FORCE_CACHE_ALL caches index.html and strands deploys |
| Default TTL | 0s–1yr | 3600s | Irrelevant once origin headers are set | Applies to static types missing Cache-Control |
| Hashed assets | any | none | public, max-age=31536000, immutable |
Safe only because filenames change per build |
index.html |
any | none | no-cache (or max-age=60) |
This one file controls deploy propagation |
| Negative caching | on/off + TTLs | off | On; cache 404/410 for 60s | Otherwise bots scanning paths hammer the origin |
| Invalidation | invalidate-cdn-cache --path |
— | Emergency lever; rate-limited, takes minutes | Versioned filenames make it a non-event |
Build it with gcloud (project PROJECT_ID, region asia-south1, domain app.example.com):
# 1. Bucket + build upload with explicit cache headers
gcloud storage buckets create gs://app-example-spa --location=asia-south1 \
--uniform-bucket-level-access
gcloud storage cp -r dist/assets gs://app-example-spa/assets \
--cache-control="public, max-age=31536000, immutable"
gcloud storage cp dist/index.html gs://app-example-spa/ --cache-control="no-cache"
gcloud storage buckets update gs://app-example-spa \
--web-main-page-suffix=index.html --web-error-page=index.html
gcloud storage buckets add-iam-policy-binding gs://app-example-spa \
--member=allUsers --role=roles/storage.objectViewer
# 2. Backend bucket with CDN + serverless NEG for the API
gcloud compute backend-buckets create spa-backend \
--gcs-bucket-name=app-example-spa --enable-cdn --cache-mode=USE_ORIGIN_HEADERS
gcloud compute network-endpoint-groups create api-neg --region=asia-south1 \
--network-endpoint-type=serverless --cloud-run-service=api
gcloud compute backend-services create api-backend --global \
--load-balancing-scheme=EXTERNAL_MANAGED --protocol=HTTPS
gcloud compute backend-services add-backend api-backend --global \
--network-endpoint-group=api-neg --network-endpoint-group-region=asia-south1
# 3. URL map, managed cert, HTTPS frontend
gcloud compute url-maps create web-map --default-backend-bucket=spa-backend
gcloud compute url-maps add-path-matcher web-map --path-matcher-name=api \
--default-backend-bucket=spa-backend --backend-service-path-rules="/api/*=api-backend"
gcloud compute ssl-certificates create app-cert --domains=app.example.com --global
gcloud compute target-https-proxies create web-proxy --url-map=web-map \
--ssl-certificates=app-cert
gcloud compute addresses create web-ip --global
gcloud compute forwarding-rules create web-fr --global --address=web-ip \
--target-https-proxy=web-proxy --ports=443
Point the domain’s A record at the reserved IP; the Google-managed certificate only flips from PROVISIONING to ACTIVE ~15–60 minutes after DNS resolves to the LB — teams reliably do this in the wrong order, then debug it as an LB failure. In Terraform:
resource "google_compute_backend_bucket" "spa" {
name = "spa-backend"
bucket_name = google_storage_bucket.spa.name
enable_cdn = true
cdn_policy {
cache_mode = "USE_ORIGIN_HEADERS"
negative_caching = true
}
}
resource "google_compute_region_network_endpoint_group" "api" {
name = "api-neg"
region = "asia-south1"
network_endpoint_type = "SERVERLESS"
cloud_run { service = google_cloud_run_v2_service.api.name }
}
resource "google_compute_url_map" "web" {
name = "web-map"
default_service = google_compute_backend_bucket.spa.id
host_rule { hosts = ["app.example.com"] path_matcher = "main" }
path_matcher {
name = "main"
default_service = google_compute_backend_bucket.spa.id
path_rule {
paths = ["/api/*"]
service = google_compute_backend_service.api.id
}
}
}
The API tier: Cloud Run, knob by knob
Cloud Run runs your container image and gives each revision an autoscaled fleet of instances. These knobs are the entire performance/cost surface of the tier — the defaults are tuned for “generic demo”, not your workload:
| Setting | Values | Default | When to change | Trade-off / gotcha |
|---|---|---|---|---|
--concurrency |
1–1000 | 80 | I/O-bound APIs: 80–250; CPU-bound: 1–4 | Higher = fewer instances = cheaper, but CPU is shared across in-flight requests; too high → latency collapse |
--min-instances |
0–N | 0 | ≥1 on user-facing APIs to kill cold starts | Idle instances bill 24×7 — often the biggest line item at low traffic |
--max-instances |
1–1000 (default cap 100/revision) | 100 | Lower to protect downstreams and bound spend | Too low → 429s; each instance may hold DB connections |
--cpu |
0.08–8 (concurrency >1 needs ≥1) | 1 | 2–4 for CPU-heavy handlers | 4+ vCPU requires ≥2 GiB memory |
--memory |
128 MiB–32 GiB | 512 MiB | JVM/.NET: 1–2 GiB floor | OOM kills the instance mid-request → 503 |
| CPU allocation | request-only vs --no-cpu-throttling |
Request-only | Always-on for background threads | Always-on switches to instance-lifetime billing |
--timeout |
1–3600s | 300s | 30–60s for user APIs | Long work belongs in Tasks/Pub/Sub, not a 1h request |
| Execution env | gen1, gen2 | Service-dependent | gen2 for full Linux syscalls, volume mounts | gen1 cold-starts slightly faster |
--cpu-boost |
on/off | on (new services) | Keep on — extra startup CPU | — |
--ingress |
all, internal, internal-and-cloud-load-balancing |
all |
LB-only for the API; internal for workers |
all lets users bypass LB, CDN, Cloud Armor via run.app |
--session-affinity |
on/off | off | Per-instance cache optimizations | Best-effort — never for correctness |
| Sidecars / volumes | sidecars; GCS/NFS/in-memory volumes | none | Cloud SQL Auth Proxy as sidecar | Volumes need gen2 |
Autoscaling and concurrency, quantitatively
The autoscaler targets your concurrency setting: at concurrency 80, roughly ceil(concurrent_requests / 80) instances stay warm. The arithmetic that matters: 1,000 concurrent requests at concurrency 80 needs 13 instances; at concurrency 1 it needs 1,000 — max-instances throttling (HTTP 429) and a 12× bill. Set it from a load test: raise until p99 at target load degrades, back off one step. A Node/Go JSON API on Firestore sustains 100–250; CPU-heavy Python is honest at 4–10.
Cold starts: anatomy and mitigation
A cold start is the latency of scale-from-zero (or scale-out): Cloud Run schedules an instance, pulls the image, boots your process, and waits for it to listen on $PORT. Typical: ~100–500 ms for a slim Go/Node image; 2–10 s for fat JVM/.NET images with eager init. Mitigations in order:
| # | Lever | What it does | Cost | When |
|---|---|---|---|---|
| 1 | Slim image (distroless, multi-stage) | Less to pull and extract | Free | Always |
| 2 | Lazy-init heavy clients | Listens on $PORT sooner |
Free | Always |
| 3 | --cpu-boost |
Extra CPU during startup (big for JIT runtimes) | ~Free | Always |
| 4 | --min-instances 1..N |
Warm instances; cold starts vanish for baseline traffic | Idle-rate billing 24×7 | User-facing APIs |
| 5 | Raise concurrency | Fewer scale-out events | Free | I/O-bound services |
| 6 | gen1 exec environment | Slightly faster starts | No volume mounts | Only if measured |
Do not skip to #4: min-instances at 1 vCPU/512 MiB runs roughly $45–55/month per always-warm instance at Tier 1 idle rates — often more than the entire request-driven bill. Slim images and lazy init are free and frequently cut cold starts 5–10×.
Deploy it
gcloud run deploy api \
--image=asia-south1-docker.pkg.dev/PROJECT_ID/app/api:1.4.2 \
--region=asia-south1 \
--ingress=internal-and-cloud-load-balancing \
--allow-unauthenticated \
--service-account=api-sa@PROJECT_ID.iam.gserviceaccount.com \
--concurrency=120 --cpu=1 --memory=512Mi --timeout=60 \
--min-instances=1 --max-instances=40 --cpu-boost \
--set-env-vars=FIRESTORE_DB=app \
--set-secrets=STRIPE_KEY=stripe-api-key:3
Two deliberate choices: the API is --allow-unauthenticated at the platform layer because browsers cannot attach IAM tokens — end-user auth is JWT middleware (next section) — while --ingress pins traffic to the LB. Internal workers invert it: --no-allow-unauthenticated, invoked only by Pub/Sub, Tasks and Scheduler with OIDC tokens. In Terraform:
resource "google_cloud_run_v2_service" "api" {
name = "api"
location = "asia-south1"
ingress = "INGRESS_TRAFFIC_INTERNAL_LOAD_BALANCER"
template {
service_account = google_service_account.api.email
max_instance_request_concurrency = 120
timeout = "60s"
scaling { min_instance_count = 1 max_instance_count = 40 }
containers {
image = "asia-south1-docker.pkg.dev/${var.project}/app/api:1.4.2"
resources {
limits = { cpu = "1", memory = "512Mi" }
cpu_idle = true # request-based billing
startup_cpu_boost = true
}
env { name = "FIRESTORE_DB" value = "app" }
env {
name = "STRIPE_KEY"
value_source {
secret_key_ref { secret = google_secret_manager_secret.stripe.secret_id version = "3" }
}
}
}
}
}
Platform limits you design around:
| Limit | Value | Design consequence |
|---|---|---|
| Request timeout max | 3600 s | Longer → Cloud Run Jobs, Tasks chains, Workflows |
| Max concurrency | 1000/instance | Above that, add instances |
| Max instances (default quota) | 100/revision, raisable | Quota-request before launch, not during |
| Request/response size (buffered) | 32 MiB | Uploads go browser→GCS via signed URLs, never through the API |
| Max per instance | 8 vCPU / 32 GiB | Bigger units → GKE or Compute Engine |
| Startup | Must listen on $PORT (8080) within the startup probe window |
The #1 first-deploy failure |
| Instance lifetime | Recycled at any time; ~10s after SIGTERM | No local state, graceful shutdown handler |
Identity: Identity Platform tokens, verified statelessly
Stateless compute forbids server-side sessions — instance N must authenticate a user whose login ran on an instance that no longer exists. The pattern is OIDC ID tokens (JWTs): Identity Platform (productized Firebase Auth, a.k.a. GCIP) handles sign-up, sign-in, password reset, OAuth and SAML/OIDC federation, MFA and multi-tenancy; the SPA sends its signed ID token as Authorization: Bearer <jwt> on every call; middleware verifies the signature locally against Google’s JWKS keys — no network call per request, no session store.
Place Identity Platform among its alternatives first:
| Option | What it is | SAML/OIDC federation | Multi-tenancy | MFA | Free tier | Choose when |
|---|---|---|---|---|---|---|
| Identity Platform | Enterprise Firebase Auth with SLA | Yes | Yes | SMS/TOTP | 50k MAU (email/social) | Default for customer-facing apps on GCP |
| Firebase Auth | Same SDKs, consumer packaging | No | No | Limited | Generous | Hobby/mobile-first, never needs SAML |
| IAP | Google-account gate at the LB | Workforce identity | N/A | Google MFA | Included | Internal tools, all users on org accounts |
| DIY / rented IdP | You run it | Varies | Varies | Varies | Varies | Existing IdP mandate; otherwise don’t build auth |
The ID token is a JWT with claims your middleware must actually check — memorize the bold four:
| Claim | Example | Verify |
|---|---|---|
iss |
https://securetoken.google.com/PROJECT_ID |
Exact match |
aud |
PROJECT_ID |
Must be your project — foreign Firebase tokens verify cryptographically but are not yours |
exp / iat |
1-hour lifetime | Reject expired; SDKs auto-refresh client-side |
sub / uid |
9X8... stable user ID |
Your foreign key in Firestore — never email, which changes |
email, email_verified |
user@x.com, true |
Gate sensitive actions on email_verified |
firebase.sign_in_provider |
google.com, password |
Policy (e.g., require MFA for password) |
| Custom claims | {"role":"admin"} |
Set via Admin SDK; ≤1000 bytes; propagate on next refresh (≤1h lag!) |
Verification middleware in Node with the Admin SDK (it caches Google’s public keys in memory and rotates them automatically):
import { initializeApp } from "firebase-admin/app";
import { getAuth } from "firebase-admin/auth";
initializeApp(); // uses the Cloud Run service account — no key file, ever
export async function requireUser(req, res, next) {
const m = (req.headers.authorization || "").match(/^Bearer (.+)$/);
if (!m) return res.status(401).json({ error: "missing bearer token" });
try {
const token = await getAuth().verifyIdToken(m[1]); // local sig check, cached JWKS
req.user = { uid: token.uid, email: token.email, role: token.role ?? "user" };
next();
} catch {
res.status(401).json({ error: "invalid or expired token" });
}
}
Two production notes. First, custom-claim lag: claims ride the token and tokens live an hour — a revoked role takes effect on the next refresh unless you force getIdToken(true) or check a “revoked” flag on genuinely sensitive routes. Second, authorization is yours: Identity Platform authenticates; whether uid may read orders/42 is enforced in your API — and mirrored in Firestore Security Rules if the SPA also reads Firestore directly, a legitimate hybrid that takes the API off pure-read paths.
Data: Firestore by design, Cloud SQL by exception
Firestore is the default state store here because it matches the compute model: per-operation pricing, automatic scaling, no connection pools to exhaust when Cloud Run jumps from 2 to 200 instances, strong consistency, realtime listeners. But “Firestore” is two products at one console entry, and the mode is permanent per database:
| Dimension | Native mode | Datastore mode |
|---|---|---|
| Data model | Documents in collections/subcollections | Entities in kinds |
| Clients | Web/mobile SDKs + server SDKs | Server SDKs only |
| Realtime listeners / offline sync | Yes | No |
| Security Rules (direct client access) | Yes | No (IAM only) |
| Query language | SDK query builder | GQL + SDKs |
| Consistency | Strong everywhere | Strong (post-migration backend) |
| Best for | New web/mobile apps — this article | Legacy App Engine Datastore workloads |
Pick native mode for a new web app. A project can now hold multiple named databases (e.g., (default) in nam5, an app database in asia-south1), making per-environment and data-residency splits clean.
Model documents around the limits — these numbers shape schemas:
| Limit | Value | Schema consequence |
|---|---|---|
| Max document size | 1 MiB | Unbounded lists (comments, events) become subcollections, not embedded arrays |
| Sustained writes per document | ~1/s (guidance) | Counters need sharded counters (N shard docs, sum on read) |
| Writes per transaction/batch | 500 | Chunk bulk imports; multi-doc invariants ≤500 docs |
| Index entries per document | 40,000 | Every field indexes by default — exempt big maps you never query |
| Composite indexes per database | 200 (default quota) | Each where X == && order by Y combo consumes one |
| Subcollection depth | 100 | Practically keep 2–3 levels |
| Monotonic document IDs | Hotspots | Auto-IDs; never timestamp-prefixed IDs on hot collections |
| Query model | Index-backed only; cost ∝ documents returned; count()/sum()/avg() exist |
No accidental table scans — but no ad-hoc joins either |
That last row is the philosophical one: queries must be index-backed and you are billed per document read, so you model for your screens. “My 20 most recent orders” should read 20 documents — denormalize the product name into the order rather than “joining” 20 product reads. Duplication is a feature, paid for with Pub/Sub-driven backfill when the source changes.
| Pattern | Use for | Watch out |
|---|---|---|
Embedded map (user.address) |
1:1 data read together | — |
Embedded array (order.items[]) |
Small bounded lists | Unbounded growth hits 1 MiB; element updates rewrite the doc |
Subcollection (orders/{id}/events) |
Unbounded 1:N owned data | Collection-group index to query across parents |
Root collection + FK (reviews.productId) |
N:M, queried both ways | You manage the “join” with a second read |
Denormalized copy (order.productName) |
Read-path speed/cost | Needs update fan-out (Pub/Sub) on source change |
Sharded counter (counters/x/shards/{0..9}) |
>1 write/s counters | Read = sum shards; shard count ≥ peak writes/s |
When does Cloud SQL (PostgreSQL) beat Firestore in this same architecture? When the data is relational even though the compute is serverless:
| If your workload has… | Firestore | Cloud SQL (PostgreSQL) |
|---|---|---|
| Screens-first access, per-entity fetches, realtime UI | ✅ Designed for it | Works, more code |
| Ad-hoc reporting, JOINs, BI tools | ❌ Export to BigQuery instead | ✅ SQL is the point |
| Multi-row invariants (ledgers, reservations) | Transactions ≤500 docs, careful modeling | ✅ ACID, constraints, FKs |
| Spiky 0→200 instance scale-out | ✅ No connections to exhaust | ⚠️ Cap max-instances × pool under max_connections |
| True pay-per-use to zero | ✅ Per-op pricing | ❌ Instance bills 24×7 |
| ORM/SQL team skills | Learning curve | ✅ Boring, in the best way |
The honest hybrid is common: Firestore for user-facing state, a small Cloud SQL for the relational core, BigQuery for analytics fed by Pub/Sub. Cloud SQL reintroduces a 24×7 instance cost and a connection ceiling — keep max-instances × pool size under Postgres max_connections.
gcloud firestore databases create --database=app \
--location=asia-south1 --type=firestore-native
# Composite index for: where userId == X order by createdAt desc
gcloud firestore indexes composite create --database=app \
--collection-group=orders \
--field-config=field-path=userId,order=ascending \
--field-config=field-path=createdAt,order=descending
resource "google_firestore_database" "app" {
project = var.project
name = "app"
location_id = "asia-south1"
type = "FIRESTORE_NATIVE"
point_in_time_recovery_enablement = "POINT_IN_TIME_RECOVERY_ENABLED"
delete_protection_state = "DELETE_PROTECTION_ENABLED"
}
Turn on point-in-time recovery (7-day rewind) and scheduled backups from day one; “serverless” does not mean immune to your own bad deploy writing garbage for six hours.
Async: Pub/Sub push, Cloud Tasks, Eventarc and Scheduler
The request path stays short; everything else flows through one of four services. This table is the decision — pin it to the wall:
| Pub/Sub | Cloud Tasks | Eventarc | Cloud Scheduler | |
|---|---|---|---|---|
| Shape | 1 event → N subscribers | 1 task → 1 handler | GCP event → handler | Time → handler |
| Pace controlled by | Subscriber scaling | Sender-configured queue rate | Subscriber | Cron |
| Rate limiting | No (that’s the point) | ✅ dispatch rate + concurrency caps | No | N/A |
| Delayed delivery | No | ✅ per-task schedule_time (≤30 days) |
No | ✅ recurring |
| De-duplication | At-least-once; exactly-once on pull only | ✅ named tasks (windowed) | At-least-once | Per tick |
| Ordering | Ordering keys (1 MB/s per key) | No | No | N/A |
| Retry | Per-subscription backoff + DLQ | Per-queue attempts/backoff | Via Pub/Sub | Per-job config |
| Payload cap | 10 MB | 1 MB (HTTP tasks) | Event-dependent | Your body |
| Use for | Domain-event fan-out | Emails, third-party APIs, per-user jobs | “Object landed in GCS” | Nightly reports |
Pub/Sub push to Cloud Run, with auth done right
For serverless consumers, use push subscriptions: Pub/Sub POSTs each message to the worker’s URL, so the worker scales with deliveries and to zero without them. (Pull needs an always-on puller — right for GKE consumers and for exactly-once, which push does not support.) Delivery authenticates with an OIDC token minted as a push service account; the worker — --no-allow-unauthenticated, --ingress=internal — admits only callers with roles/run.invoker.
| Setting | Values | Default | Guidance |
|---|---|---|---|
--push-endpoint |
HTTPS URL | — | Dedicated route, e.g. /pubsub/orders |
--push-auth-service-account |
SA email | none | Always set; without it the worker must be public |
--ack-deadline |
10–600 s | 10 s | ≥ p99 handler time; the 2xx response is the ack |
--min-retry-delay / --max-retry-delay |
0–600 s | ~10 s / 600 s | Exponential backoff between redeliveries |
--dead-letter-topic + --max-delivery-attempts |
5–100 | none | Always set (5–10); a poison message otherwise retries for 7 days |
| Retention | 7 days (31 max) | 7 d | Enough to survive a weekend outage |
| Ack semantics | 102/200/201/202/204 = ack; else redeliver | — | Return 200 only after durable side effects |
gcloud pubsub topics create orders
gcloud pubsub topics create orders-dlq
gcloud iam service-accounts create pubsub-push
gcloud run services add-iam-policy-binding worker --region=asia-south1 \
--member=serviceAccount:pubsub-push@PROJECT_ID.iam.gserviceaccount.com \
--role=roles/run.invoker
gcloud pubsub subscriptions create orders-worker --topic=orders \
--push-endpoint=https://worker-xxxxx-el.a.run.app/pubsub/orders \
--push-auth-service-account=pubsub-push@PROJECT_ID.iam.gserviceaccount.com \
--ack-deadline=60 --min-retry-delay=10s --max-retry-delay=600s \
--dead-letter-topic=orders-dlq --max-delivery-attempts=5
Two IAM grants hide here: the Pub/Sub service agent (service-PROJECT_NUMBER@gcp-sa-pubsub.iam.gserviceaccount.com) needs roles/iam.serviceAccountTokenCreator to mint OIDC tokens as your push SA, plus publisher on the DLQ topic and subscriber on the source subscription to dead-letter. Miss the first and every push 403s; miss the second and dead-lettering silently fails.
Because delivery is at-least-once, handlers must be idempotent: key side effects on the immutable messageId or a business key — a create-if-absent on deliveries/{messageId} in Firestore is the cheapest idempotency ledger you will ever build.
resource "google_pubsub_subscription" "orders_worker" {
name = "orders-worker"
topic = google_pubsub_topic.orders.id
ack_deadline_seconds = 60
push_config {
push_endpoint = "${google_cloud_run_v2_service.worker.uri}/pubsub/orders"
oidc_token { service_account_email = google_service_account.pubsub_push.email }
}
retry_policy {
minimum_backoff = "10s"
maximum_backoff = "600s"
}
dead_letter_policy {
dead_letter_topic = google_pubsub_topic.orders_dlq.id
max_delivery_attempts = 5
}
}
Cloud Tasks: the rate limiter and the scheduler-of-one
Pub/Sub delivers as fast as your workers scale — exactly wrong when the downstream is a third-party API with a 10 req/s contract. Cloud Tasks inverts control: the queue enforces dispatch rate and concurrency, retries each task on its own schedule, de-duplicates named tasks, and can hold a task up to 30 days before delivery (“send follow-up in 48 h”).
| Setting | Values | Default | Guidance |
|---|---|---|---|
--max-dispatches-per-second |
≤500/queue | 500 | Set to the downstream’s contract (e.g., 10) |
--max-concurrent-dispatches |
≤5000 | 1000 | ≤ downstream’s parallelism tolerance |
--max-attempts |
-1 (unlimited) or N | 100 | 5–10 with alerting; unlimited hides poison tasks |
--min-backoff / --max-backoff |
0.1 s–1 h | 0.1 s / 1 h | Raise min to 5s+ for flaky SaaS APIs |
--max-doublings |
0–16 | 16 | Length of the exponential phase |
| Task name | Optional, unique | auto | Deterministic names (email-{orderId}) = enqueue-side de-dup window |
schedule_time |
now → +30 days | now | Delayed delivery without timer infrastructure |
gcloud tasks queues create email-queue --location=asia-south1 \
--max-dispatches-per-second=10 --max-concurrent-dispatches=20 \
--max-attempts=8 --min-backoff=5s --max-backoff=300s
# Tasks are enqueued from the API with an OIDC token (service account with
# run.invoker), so the worker stays private — same pattern as Pub/Sub push.
Eventarc and Cloud Scheduler round it out
Eventarc turns Google Cloud events into CloudEvents POSTed to Cloud Run — canonically google.cloud.storage.object.v1.finalized on an uploads bucket triggering a thumbnailer, no polling, no app-level publish. Under the hood it is Pub/Sub, so the same push-auth and idempotency rules apply. Cloud Scheduler is cron-as-a-service: minute granularity, IANA time zones, OIDC-authenticated HTTP targets.
gcloud scheduler jobs create http nightly-rollup --location=asia-south1 \
--schedule="0 2 * * *" --time-zone="Asia/Kolkata" \
--uri=https://worker-xxxxx-el.a.run.app/jobs/rollup \
--oidc-service-account-email=scheduler-invoker@PROJECT_ID.iam.gserviceaccount.com \
--attempt-deadline=180s
Scheduler fires at-least-once per tick and overlapping runs are possible after retries — the rollup handler needs the same idempotency discipline (e.g., lease a jobs/rollup-2026-07-07 doc before working).
Secrets and configuration
Non-secret configuration rides env vars on the revision — fine, because revisions are immutable and diffable. Anything sensitive belongs in Secret Manager: versioned, IAM-gated (roles/secretmanager.secretAccessor to the service’s service account, never a user), audit-logged, CMEK-capable:
| Delivery | How | Updates when | Best for | Gotcha |
|---|---|---|---|---|
Env var (--set-secrets=KEY=name:version) |
Resolved at instance start | New instances only | Almost everything | With :latest, two live instances can hold different values mid-rotation |
| Volume mount | File in an in-memory volume, fetched at start | New instances | Libraries wanting key files; large values | Still not live-reload |
| Runtime fetch (client library) | AccessSecretVersion on demand |
Whenever called | Rotation-sensitive creds | Latency + failure mode on the hot path; cache it |
| ❌ Plaintext in Terraform/YAML env | — | — | Never | Persists in state files, repos, services describe output |
The operational rule: pin explicit versions (stripe-api-key:3, not :latest) and rotate by adding a version + deploying a revision. Rotation becomes a reviewable, rollbackable deploy instead of an invisible mutation — why the earlier deploy said :3.
gcloud secrets create stripe-api-key --replication-policy=automatic
printf '%s' "$STRIPE_KEY" | gcloud secrets versions add stripe-api-key --data-file=-
gcloud secrets add-iam-policy-binding stripe-api-key \
--member=serviceAccount:api-sa@PROJECT_ID.iam.gserviceaccount.com \
--role=roles/secretmanager.secretAccessor
Observability: Trace, Logging, Monitoring
Serverless observability starts ahead of VMs — request logs, traces and metrics exist with zero agents — but correlation is on you:
| Tool | Question it answers | Serverless specifics | Alert on |
|---|---|---|---|
| Cloud Logging | “What happened in request X?” | stdout JSON becomes structured logs; per-request platform logs |
Log-based metric on severity>=ERROR rate |
| Cloud Trace | “Where did the 900 ms go?” | LB + Cloud Run auto-spans; propagate traceparent for Firestore/Pub/Sub hops |
p95 per route (via Monitoring) |
| Cloud Monitoring | “Is the system healthy?” | Built-ins: request count/latency by status, instance count, CPU/mem, Pub/Sub oldest-unacked age | The four below |
| Error Reporting | “Which new exception is this?” | Auto-groups stack traces from logs | New error group in prod |
| Uptime checks | “Is it up from outside?” | Hit /healthz through the LB from 4 regions |
2+ regions failing |
The single highest-leverage habit: log structured JSON carrying the trace ID, so one click pivots from a slow trace to its exact logs:
// Correlate logs with traces on Cloud Run (Node)
app.use((req, res, next) => {
const trace = (req.get("x-cloud-trace-context") || "").split("/")[0];
req.log = (severity, message, extra = {}) =>
console.log(JSON.stringify({
severity, message, ...extra,
"logging.googleapis.com/trace": `projects/${process.env.GOOGLE_CLOUD_PROJECT}/traces/${trace}`,
}));
next();
});
Four alerts cover 90% of what goes wrong here: (1) 5xx ratio > 1% over 5 min; (2) p95 latency over SLO per route; (3) oldest_unacked_message_age > 10 min on any subscription — the universal “consumer is broken” signal; (4) DLQ depth > 0 — every dead-lettered message is a bug report. Add a billing budget alert at 1.5× expected spend: in pay-per-use, a cost spike is an incident (retry storm, crawler loop, runaway fan-out). For burn rates, channels and dashboards as code, see Cloud Monitoring & the operations suite.
Architecture at a glance
Read the diagram left to right, following one user’s click. The browser loads the SPA from the edge: the global ALB answers on one anycast IP, Cloud CDN serves hashed assets from cache, and the GCS backend bucket is touched only on misses (badge 1 marks the deep-link 404 trap). The SPA signs in against Identity Platform; every /api/* call then carries a one-hour JWT that middleware verifies locally against cached JWKS keys (badge 2) — no session store anywhere. The Cloud Run API (concurrency 120, min-instances 1 against cold starts — badge 3) does the synchronous work: Firestore reads and writes, secrets from pinned versions.
Everything slow exits right, off the hot path: domain events publish to Pub/Sub, which pushes — OIDC-authenticated, run.invoker-gated (badge 4) — into an internal worker; rate-limited work goes through a Cloud Tasks queue dispatching at exactly the pace the downstream tolerates (badge 5); Cloud Scheduler fires nightly jobs into the same worker. The worker writes back to Firestore, watching the per-document write limits badge 6 flags. Every hop is a managed service scaling independently — and every hop is authorized by a dedicated service account, not a shared key.
Real-world scenario
Meridian Expense, a fictional but representative 9-person Pune startup, ships expense-report SaaS for Indian SMBs: React SPA, REST API, receipt OCR, GST reports, email notifications. Traffic is violently diurnal — 09:00–19:00 IST weekdays, near-zero otherwise, a 20× spike on the 30th and 31st as finance teams close books.
Their first architecture — two e2-standard-2 VMs behind a regional LB, PostgreSQL on a third — cost ~₹13,000/month and fell over every month-end: the autoscaler took six minutes to add a VM, and OCR ran inside the API process, so upload bursts starved interactive requests. The rebuild followed this article’s shape: SPA to a backend bucket with Cloud CDN; API to Cloud Run at concurrency 150, min-instances 1, max 60; auth to Identity Platform (12,400 MAU — free tier); expenses to Firestore native in asia-south1; receipt uploads via signed URLs straight to GCS, whose object.finalized events flow through Eventarc to an OCR worker (2 vCPU, concurrency 4, internal ingress); expense.approved events fan out over Pub/Sub to email, webhook and analytics subscribers; the email path runs through a Cloud Tasks queue capped at 14 dispatches/s — the email provider’s contract.
Month-end now costs money and nothing else: Cloud Run peaks at 41 instances for ~3 hours, then drains to 1. A representative month billed ≈ $96 (~₹8,300): $18 LB, $52 compute (mostly the min-instance floor plus OCR), $14 Firestore, $6 CDN/egress, the rest in tailing decimals — Pub/Sub, Tasks, Scheduler and Secret Manager all rode free tiers.
Two incidents taught the sharp edges. Push-auth: the worker went out --allow-unauthenticated “temporarily” and a pentest found it; locking it down then broke deliveries for an hour because nobody had granted the push SA run.invoker — both fixes belong in one Terraform commit. And a Firestore hotspot: a per-org expenseCount document throttled with ABORTED errors at month-end (every approval incremented it); a 10-shard counter fixed it in an afternoon. The CTO’s board summary: cost down 35%, month-end incidents zero, and — the unadvertised win — staging now costs ₹200/month because it scales to zero at night.
Advantages and disadvantages
| Advantages | Disadvantages |
|---|---|
| True pay-per-use: idle ≈ zero compute cost; dev/staging cost pocket change | Fixed floors remain: the LB ($18/mo) and any min-instances bill 24×7 |
| 0 → hundreds of instances in seconds, no capacity planning | Cold starts exist and must be engineered away |
| No OS/patching/node ops; immutable revisions, instant rollback | Less control: no kernel tuning, 8 vCPU/32 GiB per-instance ceiling |
| Stateless by construction; regional HA automatic | Statelessness is forced: ephemeral disk, time-bounded WebSockets, best-effort affinity |
| Components scale and bill independently; async absorbs spikes | Distributed-systems tax: at-least-once, idempotency, DLQs, eventual consistency between copies |
| Security posture: no SSH, per-service SAs, OIDC everywhere, IAM-gated secrets | Vendor coupling: Firestore models, Pub/Sub semantics, Identity Platform tokens don’t lift-and-shift |
| Observability built in: logs, traces, metrics with no agents | Debugging spans five consoles unless you wire trace↔log correlation early |
| Unit economics knowable per request | At steady high load, per-request pricing can exceed committed-use VMs/GKE by 2–4× |
When serverless wins: spiky or unpredictable traffic, small teams, many small services, dev/staging fleets, event-driven integrations. When it loses: sustained near-100% utilization (break-even below), hard-realtime latency budgets, heavy long-lived connections, portability mandates, or workloads one apt install away from a ₹4,000/month VM you already run.
Hands-on lab
Deployable in a fresh project, free-tier-friendly (skip the LB section and there is almost no cost; the LB runs ~$0.60/day). Prereqs: gcloud, a billing-enabled project.
# 0. Setup
export PROJECT_ID=your-project REGION=asia-south1
gcloud config set project $PROJECT_ID
gcloud services enable run.googleapis.com firestore.googleapis.com \
pubsub.googleapis.com cloudtasks.googleapis.com cloudscheduler.googleapis.com \
secretmanager.googleapis.com compute.googleapis.com
# 1. Firestore (native) + a demo secret
gcloud firestore databases create --location=$REGION --type=firestore-native
gcloud secrets create demo-key --replication-policy=automatic
printf 'shh-not-really-secret' | gcloud secrets versions add demo-key --data-file=-
# 2. API service (sample image; swap for your build)
gcloud iam service-accounts create api-sa
gcloud secrets add-iam-policy-binding demo-key \
--member=serviceAccount:api-sa@$PROJECT_ID.iam.gserviceaccount.com \
--role=roles/secretmanager.secretAccessor
gcloud projects add-iam-policy-binding $PROJECT_ID \
--member=serviceAccount:api-sa@$PROJECT_ID.iam.gserviceaccount.com \
--role=roles/datastore.user
gcloud run deploy api --image=us-docker.pkg.dev/cloudrun/container/hello \
--region=$REGION --allow-unauthenticated \
--service-account=api-sa@$PROJECT_ID.iam.gserviceaccount.com \
--concurrency=120 --min-instances=0 --max-instances=5 \
--set-secrets=DEMO_KEY=demo-key:1
# Expected: "Service [api] ... deployed"; note the URL.
# 3. Feel a cold start, then kill it
time curl -s $(gcloud run services describe api --region=$REGION --format='value(status.url)') >/dev/null
# First call after idle: ~1-3s. Repeat immediately: ~50-150ms.
gcloud run services update api --region=$REGION --min-instances=1
# Re-test after a minute: cold start gone.
# 4. Private worker + authenticated Pub/Sub push + DLQ
gcloud run deploy worker --image=us-docker.pkg.dev/cloudrun/container/hello \
--region=$REGION --no-allow-unauthenticated --ingress=internal
gcloud iam service-accounts create pubsub-push
gcloud run services add-iam-policy-binding worker --region=$REGION \
--member=serviceAccount:pubsub-push@$PROJECT_ID.iam.gserviceaccount.com \
--role=roles/run.invoker
gcloud pubsub topics create orders && gcloud pubsub topics create orders-dlq
gcloud pubsub subscriptions create orders-worker --topic=orders \
--push-endpoint=$(gcloud run services describe worker --region=$REGION --format='value(status.url)') \
--push-auth-service-account=pubsub-push@$PROJECT_ID.iam.gserviceaccount.com \
--ack-deadline=30 --dead-letter-topic=orders-dlq --max-delivery-attempts=5
gcloud pubsub topics publish orders --message='{"orderId":"o_1"}'
gcloud logging read 'resource.labels.service_name="worker"' --limit=3 \
--format='value(httpRequest.status, timestamp)'
# Expected: a 200 within seconds. Remove the run.invoker binding, republish → 403s + retries.
# 5. Cron against the private worker (reusing the push SA)
gcloud scheduler jobs create http tick --location=$REGION \
--schedule="*/5 * * * *" --time-zone="Asia/Kolkata" \
--uri=$(gcloud run services describe worker --region=$REGION --format='value(status.url)') \
--oidc-service-account-email=pubsub-push@$PROJECT_ID.iam.gserviceaccount.com
# 6. Teardown
gcloud scheduler jobs delete tick --location=$REGION -q
gcloud pubsub subscriptions delete orders-worker -q
gcloud pubsub topics delete orders orders-dlq -q
gcloud run services delete api worker --region=$REGION -q
gcloud secrets delete demo-key -q
The lab reproduces the two failure modes you will meet first in production: the cold start you can feel (step 3) and the push-auth 403 you can induce (step 4).
Common mistakes & troubleshooting
The playbook, ordered by how often each bites:
| # | Symptom | Root cause | Confirm | Fix |
|---|---|---|---|---|
| 1 | Pub/Sub push 403s, messages pile up | Push SA lacks roles/run.invoker, or no --push-auth-service-account |
gcloud logging read '... httpRequest.status=403' on the worker; subscription shows retries |
Grant run.invoker to the push SA; set OIDC on the subscription |
| 2 | First request after idle takes 2–10 s | Cold start: fat image + eager init | Trace shows a gap before the handler; container start line in logs just before the request | Slim image, lazy init, --cpu-boost, then --min-instances=1 |
| 3 | 429 Too Many Requests during spike |
max-instances reached |
container/instance_count flat at max while requests climb |
Raise max (and quota); confirm the downstream can take it first |
| 4 | Intermittent 503, instance restarted |
OOM | Log: Memory limit of 512 MiB exceeded; memory utilization ≈ 1.0 |
Raise --memory; fix leaks; lower concurrency (memory is shared per instance) |
| 5 | / works; F5 on /orders/42 shows 404 |
Backend bucket serves objects, not routes | curl -I https://app.example.com/orders/42 → 404 |
Bucket error page = index.html (status stays 404) or move shell to Firebase Hosting |
| 6 | Deploy shipped; users see the old app | index.html cached |
`curl -sI …/index.html | grep -iE 'cache | age’` |
| 7 | Firestore ABORTED: too much contention |
Hot document or monotonic IDs | Error string in logs; Key Visualizer hot range | Sharded counters; auto-IDs; stop batching onto one doc |
| 8 | Every message retries to DLQ though handler “works” | Handler exceeds ack deadline, or returns non-2xx on success | Handler p99 vs --ack-deadline; oldest_unacked_message_age climbing |
Raise deadline; return 200 only after durable work |
| 9 | Same event processed twice | At-least-once + no idempotency key | Two log entries, same messageId |
Idempotency ledger on messageId; conditional Firestore create |
| 10 | PERMISSION_DENIED on a secret at start |
Runtime SA lacks secretAccessor on that secret |
Revision fails; error names secret + SA | Grant on the secret (not project-wide); redeploy |
| 11 | Cert error / default 404 right after LB setup | Managed cert PROVISIONING (DNS not pointed) or wrong default backend |
gcloud compute ssl-certificates describe app-cert --global |
Point DNS first; wait; verify path matcher |
| 12 | Bill spiked 8× overnight | Retry storm (no DLQ), crawler on uncached paths, or log flood | Billing by SKU; request count by URL; log volume by service | DLQ + max attempts; negative caching; log exclusions; budget alert |
Status codes by layer, and who actually emitted them:
| Code | Emitted by | Meaning here | First move |
|---|---|---|---|
404 (HTML) |
Backend bucket | Object not found — often an SPA deep link | Bucket error-page config |
429 |
Cloud Run | max-instances saturated (or quota) | Instance-count graph vs limit |
403 |
Cloud Run platform | Caller lacks run.invoker (push/Tasks/Scheduler → private service) |
IAM binding on the service |
401 |
Your middleware | Missing/expired/foreign JWT | Decode: aud, iss, exp |
500 |
Your app | Unhandled exception | Error Reporting group + trace |
503 |
Cloud Run | Instance crashed/OOM, or no healthy instance | Memory graph; container start logs |
504 |
LB / Cloud Run | Handler exceeded --timeout or LB backend timeout |
Trace waterfall; move work async |
Best practices
- One service account per service (
api-sa,worker-sa,pubsub-push), minimum roles each — never the over-privileged default compute SA. - Restrict ingress everywhere:
internal-and-cloud-load-balancingfor the API,internalfor workers; an openrun.appURL bypasses Cloud Armor and CDN. - Version everything immutable: hashed assets, pinned secret versions, tagged images, named revisions — rollback is
gcloud run services update-traffic api --to-revisions=api-00041-abc=100. - Make every async handler idempotent before scaling it; at-least-once is a promise Pub/Sub, Tasks, Eventarc and Scheduler all keep.
- DLQ + max attempts on every subscription/queue, alert on depth > 0. A retry loop without a floor is an outage and a bill.
- Set
max-instancesdeliberately on every service — your circuit breaker for downstreams, quotas and spend. - Signed URLs for uploads/downloads — bytes go browser↔GCS; the API only signs. Dodges the 32 MiB cap and byte-shoveling compute.
- Split traffic on deploy (
--to-revisions LATEST=10) and promote when the error graph agrees. - Model Firestore for your screens, denormalize with update fan-out, audit composite indexes quarterly.
- Dev/staging in separate projects with the same Terraform: project boundary = IAM + quota boundary = a readable bill.
Security notes
Identity between services, not shared secrets. Every internal hop authenticates with OIDC tokens minted for dedicated service accounts, enforced by roles/run.invoker on --no-allow-unauthenticated services. No API keys exist between components — nothing to rotate or leak on that plane.
Edge: attach Cloud Armor to the API backend service (preconfigured OWASP CRS rules, rate-based bans, geo controls); it only protects traffic that traverses the LB — exactly why ingress restriction is a security control, not tidiness. Add an HTTP→HTTPS redirect forwarding rule.
Data: Firestore and GCS encrypt at rest by default (CMEK if mandated). Security Rules are the only enforcement on any direct SPA→Firestore path — test them with the emulator like code. The SPA bucket gets allUsers: objectViewer and nothing else; enable public access prevention on every other bucket.
Secrets: IAM-gate per secret, pin versions, enable data-access audit logs for AccessSecretVersion. Never pass secrets through plain Terraform variables — they persist in state.
Supply chain: build in Cloud Build to Artifact Registry with vulnerability scanning; consider Binary Authorization so only attested images deploy. And when IAM says no, work the permission-denied decision tree instead of sprinkling roles/editor.
Cost & sizing
Pay-per-use rewards arithmetic. Unit prices (list, USD; Cloud Run Tier 1 region — Mumbai asia-south1 is Tier 2 at ~1.4× those rates; Firestore shown for nam5):
| Service | Unit price (list) | Always-free monthly allowance |
|---|---|---|
| Cloud Run (request-based) | $0.000024/vCPU-s · $0.0000025/GiB-s · $0.40/M requests | 180k vCPU-s · 360k GiB-s · 2M requests |
| Cloud Run idle (min-instances) | ~$0.000018/vCPU-s · ~$0.000002/GiB-s | Counts against the same free vCPU-s/GiB-s |
| Firestore (nam5) | $0.06/100k reads · $0.18/100k writes · $0.02/100k deletes · $0.18/GiB-mo | 50k reads · 20k writes · 20k deletes per day · 1 GiB |
| Pub/Sub | $40/TiB throughput (≥1 KB counted per publish and per delivery) | First 10 GiB/mo |
| Cloud Tasks | $0.40/M operations | 1M operations |
| Cloud Scheduler | $0.10/job/mo | 3 jobs |
| Secret Manager | $0.06/version/location-mo · $0.03/10k accesses | 6 versions · 10k accesses |
| Global ALB | ~$0.025/hr first 5 forwarding rules (≈$18.25/mo) + data processing | — |
| Cloud CDN | Cache egress by geo (order of $0.02–0.09/GiB) + $0.0075/10k lookups | — |
| Cloud Storage (standard) | ~$0.020–0.023/GiB-mo + operations | 5 GiB (US regions) |
| Identity Platform | Free < 50k MAU, then from $0.0055/MAU | 50k MAU |
Worked example — a real SaaS at moderate scale: 10M API requests/month (avg 150 ms at 1 vCPU/512 MiB, effective concurrency 10), min-instances 1, 15M Firestore reads, 3M writes, 5 GiB stored, 2M Pub/Sub messages (~2 KB), 500k Cloud Tasks, 50 GiB CDN egress, 4 secrets, 20k MAU:
| Line item | Arithmetic | USD/mo |
|---|---|---|
| Cloud Run requests | (10M − 2M free) × $0.40/M | $3.20 |
| Cloud Run busy compute | 10M × 0.15 s ÷ 10 conc = 150k vCPU-s < 180k free | ~$0 |
| Cloud Run idle min-instance | ~2.63M s × ($0.000018 + 0.5 × $0.000002) − remaining free | ~$47 |
| Firestore reads | (15M − 1.5M free) × $0.06/100k | $8.10 |
| Firestore writes | (3M − 0.6M free) × $0.18/100k | $4.32 |
| Firestore storage | (5 − 1) GiB × $0.18 | $0.72 |
| Pub/Sub | 2M × 2 KB × 2 ≈ 8 GiB < 10 GiB free | $0 |
| Tasks / Scheduler / Secrets | 500k < 1M free · 3 jobs free · 4 × $0.06 + accesses | ~$0.30 |
| Global LB + data processing | $18.25 + ~$2 | ~$20 |
| CDN + GCS | 50 GiB × ~$0.08 + lookups + storage | ~$4.50 |
| Identity Platform | 20k MAU < 50k | $0 |
| Total (Tier 1 list) | ≈ $88/mo ≈ ₹7,700 |
Three lessons fall out. Variable costs are nearly free at this scale — 10M requests of compute rode the free tier; the money is the fixed floor: one warm instance ($47) and one LB ($20). Drop min-instances and serve the SPA from Firebase Hosting and this workload runs ≈ $16/month — why hobby projects report absurdly low bills. Everything above the floor scales linearly, so unit economics are knowable per request. And the serverful break-even: a 24×7 fully-busy vCPU at request-based rates costs ~$63/month (2.63M s × $0.000024) before memory — an e2-standard-2 (2 vCPU) on a 3-year CUD is ~$30. A service genuinely >40–50% busy around the clock belongs on committed VMs or GKE; serverless keeps winning everything idle, spiky, or operationally expensive. That is “when serverless loses” in one number.
Sizing rules: start every JSON API at 1 vCPU / 512 MiB / concurrency 100 and adjust from p95 + memory graphs; give OCR/render workers their own service at 2–4 vCPU, concurrency 1–4; set max-instances to floor(downstream capacity ÷ concurrency); revisit min-instances monthly — the knob with the highest ₹-per-decision in the stack.
Interview & exam questions
Relevant to Professional Cloud Architect and Professional Cloud Developer; cost and async questions also appear in Associate Cloud Engineer.
-
Why front Cloud Run with a global external ALB instead of the default
run.appURL? One domain and anycast IP, path-routing to bucket + API (no CORS), Cloud CDN, Cloud Armor, managed certs — and ingress restricted tointernal-and-cloud-load-balancingso none of it can be bypassed. -
How does Cloud Run concurrency differ from Lambda’s model, and why does it matter for cost? Lambda runs one request per sandbox; Cloud Run multiplexes up to 1000 (default 80) per instance. I/O-bound APIs need an order of magnitude fewer instances — but they share the instance’s CPU/memory, so CPU-bound work wants concurrency near 1.
-
A user-facing API shows 3 s latency on the first request after quiet periods. Fix order? Cold start: slim the image, lazy-init clients, keep
--cpu-boost, raise concurrency to reduce scale-outs — then--min-instances=1if p99 still matters, accepting ~$45–55/mo per warm 1 vCPU instance. -
When would you pick Cloud SQL over Firestore in an otherwise serverless stack? Relational invariants (ledgers, FKs), ad-hoc SQL/BI, transactions beyond 500-doc limits, ORM-heavy teams. You re-accept a 24×7 instance and must cap Cloud Run scale-out against
max_connections. -
Pub/Sub push vs pull for Cloud Run consumers — and where does exactly-once fit? Push: no always-on puller, worker scales with deliveries. Exactly-once exists only on pull subscriptions, so push consumers must be idempotent (at-least-once), e.g. a messageId-keyed ledger.
-
What two IAM grants make authenticated Pub/Sub push work? The push service account needs
roles/run.invokeron the worker; the Pub/Sub service agent needsroles/iam.serviceAccountTokenCreatorto mint OIDC tokens as that SA (plus publisher/subscriber grants for dead-lettering). -
Why Cloud Tasks — not Pub/Sub — in front of a third-party email API? Tasks enforces sender-side rate (
max-dispatches-per-second) and concurrency caps, per-task retry, named-task de-dup and scheduled delivery. Pub/Sub delivers as fast as subscribers scale — no rate governor. -
How do you verify Identity Platform JWTs without adding latency? Verify signatures locally against Google’s JWKS (cached by the Admin SDK), checking
iss,aud,exp— no per-request network call. Custom claims lag up to an hour, so critical revocations need a server-side check. -
Why do SPA deep links 404 on a GCS backend bucket, and what are the options? The bucket serves objects;
/orders/42isn’t one. Set the not-found page toindex.html(right body, still 404) or host the shell on Firebase Hosting for real rewrites; URL maps can’t rewrite arbitrary paths to a file. -
What limits Firestore write throughput in practice? Per-document sustained writes (~1/s guidance) and hotspotting on monotonic keys — not database-wide caps. Contention surfaces as
ABORTED; fix with sharded counters and random IDs. -
Your serverless bill tripled overnight. Three likely causes and the telltale metric for each? Retry storm from a failing handler with no DLQ (
oldest_unacked_message_ageclimbing); crawler on uncached/404 paths (request count by URL, CDN hit ratio down); log flood (Logging ingestion GiB by service). Budget alerts at 1.5× catch all three in hours.
Quick check
- Which single Cloud Run flag most directly kills cold starts for a user-facing API, and what is its cost mechanism?
- Your push subscription retries every message five times, then they vanish. Which two configurations explain the pattern?
- A per-organization counter document throws
ABORTED: too much contentionduring peak. Fix? - Why must the API’s ingress be
internal-and-cloud-load-balancingfor Cloud Armor to mean anything? - In the worked cost model, which two line items dominate, and what does each buy?
Answers
--min-instances=1+: warm instances skip pull/boot/listen; you pay the idle rate 24×7 — roughly $45–55/month per 1 vCPU / 512 MiB instance at Tier 1 list.--max-delivery-attempts=5with a--dead-letter-topic: messages are dead-lettered, but the DLQ has no subscription/alerting — or the Pub/Sub service agent lacks the publisher grant, so dead-lettering fails unobserved.- Sharded counter: N shard documents (N ≥ peak writes/s), increment one at random, sum on read — the ~1 sustained write/s per document guidance is the constraint.
- Cloud Armor attaches to the LB’s backend service and only filters traffic traversing the LB; with ingress
all, anyone who finds therun.appURL bypasses WAF, CDN and domain controls. - The warm min-instance (~$47) buys zero cold starts; the LB (~$20) buys one domain/IP, CDN, managed TLS and Cloud Armor attachment. Nearly everything usage-based rode free tiers.
Glossary
- Serverless NEG — network endpoint group letting a global LB route to Cloud Run/Functions/App Engine in a region.
- Revision — an immutable deployed version of a Cloud Run service; traffic splits enable canary and rollback.
- Cold start — latency of creating an instance from zero: schedule → pull image → boot → listen on
$PORT. - ID token — the 1-hour OIDC JWT Identity Platform issues; verified statelessly via Google’s JWKS.
- Sharded counter — N shard documents summed on read, to exceed Firestore’s ~1 sustained write/s per document.
- Push subscription — Pub/Sub delivery that POSTs messages to an endpoint with an OIDC token; at-least-once.
- Dead-letter topic (DLQ) — where a message lands after
max-delivery-attempts; depth > 0 should page someone. - Eventarc — routes Google Cloud events (GCS, Audit Logs) as CloudEvents to Cloud Run over managed Pub/Sub.
- Signed URL — time-limited URL for direct browser↔GCS transfer, keeping large bytes off the API tier.
Next steps
- Go deeper on revisions, traffic management and the runtime in Cloud Run explained: serverless containers that scale to zero.
- Master ordering keys, exactly-once pull, schemas and DLQ patterns in Pub/Sub and event-driven architecture on GCP.
- Tighten every service account this article created with GCP IAM and service accounts: least privilege in practice.
- Build the dashboards and burn-rate alerts for this stack via Cloud Monitoring and the operations suite.
- Feed the analytics events you now publish into BigQuery as your analytics warehouse.