Private Service Connect on GCP: Publishing and Consuming Services End-to-End

VPC Peering is the wrong default for exposing a service across organizational boundaries: it forces non-overlapping CIDRs on parties who never coordinated and leaks the entire route table of both sides. Private Service Connect (PSC) inverts the model: a producer publishes exactly one service behind a load balancer, and a consumer reaches it through a single private IP in their own VPC, with no route exchange, no peering, and no shared address space. This is how managed services on GCP (Cloud SQL, Memorystore, Confluent, MongoDB Atlas) reach you, and it is how your platform team should publish internal services to tenant projects.

This walkthrough builds the full seam: the producer-side service attachment, the consumer-side endpoint and backend, NAT subnet sizing, PSC for Google APIs, DNS automation, security, and a troubleshooting playbook for the connection states you will actually hit.

1. The PSC connectivity models

PSC has three consumer-side constructs and one producer-side construct. Knowing which is which prevents most of the confusion.

The distinction that matters: an endpoint is a flat 1:1 reach (one IP -> one service). A backend puts PSC behind your own load balancer, which you need for a global anycast IP, your own TLS termination, Cloud Armor, or health-checked failover across regional producers. Start with endpoints; graduate to backends when you need an LB feature.

The connection is unidirectional. The consumer initiates; the producer never reaches back into the consumer VPC. Traffic from the producer to the consumer’s client appears to originate from the producer’s NAT subnet, which is why that subnet exists and why its sizing is a real capacity decision, not a formality.

2. Producer side: internal load balancer and service attachment

The producer publishes a service that already sits behind an internal passthrough or internal Application Load Balancer. PSC wraps the load balancer’s forwarding rule. You cannot publish a bare VM or a public LB.

First, the load balancer. Assume an internal Application Load Balancer already exists with a forwarding rule named producer-ilb-fr in us-central1. The new requirement is a dedicated NAT subnet with purpose=PRIVATE_SERVICE_CONNECT. This subnet is not for VMs; it is the address pool from which PSC source-NATs consumer traffic.

# Dedicated PSC NAT subnet in the producer VPC (see section 4 for sizing)
gcloud compute networks subnets create psc-nat-subnet \
  --project=producer-prj \
  --network=producer-vpc \
  --region=us-central1 \
  --range=10.100.0.0/24 \
  --purpose=PRIVATE_SERVICE_CONNECT

Now publish the service attachment. The critical decision is the connection-acceptance policy. --connection-preference=ACCEPT_MANUAL requires you to approve each consumer project explicitly; ACCEPT_AUTOMATIC accepts anyone who can guess the attachment URI. For anything multi-tenant or external, use ACCEPT_MANUAL.

gcloud compute service-attachments create producer-sa \
  --project=producer-prj \
  --region=us-central1 \
  --producer-forwarding-rule=producer-ilb-fr \
  --connection-preference=ACCEPT_MANUAL \
  --nat-subnets=psc-nat-subnet \
  --consumer-accept-list=consumer-prj-a=10 \
  --consumer-reject-list=blocked-prj \
  --enable-proxy-protocol

A few things earn their place here:

• --consumer-accept-list=consumer-prj-a=10 both allows project consumer-prj-a and caps it at 10 endpoints (the connection limit). This is your per-tenant quota and your blast-radius control.
• --enable-proxy-protocol prepends a PROXY protocol header so the backend can recover the consumer’s original PSC connection ID. Without it, every consumer’s traffic arrives sourced from the NAT subnet and you lose all attribution. Enable it only if your backend parses PROXY protocol; otherwise it corrupts the stream.

In Terraform the same attachment is reviewable and idempotent:

resource "google_compute_service_attachment" "producer_sa" {
  name                  = "producer-sa"
  project               = "producer-prj"
  region                = "us-central1"
  enable_proxy_protocol = true
  connection_preference = "ACCEPT_MANUAL"
  nat_subnets           = [google_compute_subnetwork.psc_nat.id]
  target_service        = google_compute_forwarding_rule.producer_ilb_fr.id

  consumer_accept_lists {
    project_id_or_num = "consumer-prj-a"
    connection_limit  = 10
  }
}

Capture the attachment URI; the consumer needs it verbatim:

gcloud compute service-attachments describe producer-sa \
  --project=producer-prj --region=us-central1 \
  --format="value(selfLink)"
# projects/producer-prj/regions/us-central1/serviceAttachments/producer-sa

3. Consumer side: endpoints and backends

3a. The simple case: a PSC endpoint

The consumer creates a global or regional internal address, then a forwarding rule that targets the producer’s attachment. The forwarding rule is the endpoint.

# Reserve an internal IP in the consumer subnet for the endpoint
gcloud compute addresses create psc-endpoint-ip \
  --project=consumer-prj-a \
  --region=us-central1 \
  --subnet=consumer-subnet \
  --addresses=10.20.0.50

# Create the PSC endpoint (a forwarding rule targeting the attachment)
gcloud compute forwarding-rules create psc-endpoint-fr \
  --project=consumer-prj-a \
  --region=us-central1 \
  --network=consumer-vpc \
  --address=psc-endpoint-ip \
  --target-service-attachment=projects/producer-prj/regions/us-central1/serviceAttachments/producer-sa

That IP, 10.20.0.50, is now the service from the consumer’s perspective. Clients in consumer-vpc connect to it; PSC carries the packets to the producer’s load balancer. No peering, no routes, no overlap concern even if both VPCs use 10.20.0.0/16 internally.

3b. The powerful case: a PSC backend behind your own LB

When you need a consumer-owned anycast frontend, your own Cloud Armor policy, or failover across two regional producers, target the attachment with a PRIVATE_SERVICE_CONNECT NEG and put it behind a consumer LB.

# A NEG that points at the producer's service attachment
gcloud compute network-endpoint-groups create psc-neg \
  --project=consumer-prj-a \
  --region=us-central1 \
  --network-endpoint-type=PRIVATE_SERVICE_CONNECT \
  --psc-target-service=projects/producer-prj/regions/us-central1/serviceAttachments/producer-sa

# Wire it into a backend service used by your own (e.g. global ALB) LB
gcloud compute backend-services add-backend my-consumer-bes \
  --project=consumer-prj-a \
  --global \
  --network-endpoint-group=psc-neg \
  --network-endpoint-group-region=us-central1

PSC NEGs do not take a health check; the producer’s load balancer owns health. For multi-region resilience you create one PSC NEG per regional service attachment and add both as backends, letting your LB shift traffic if one region’s attachment stops accepting connections.

3c. Explicit connection approval

With ACCEPT_MANUAL, the endpoint comes up PENDING until the producer approves. The consumer’s project ID must already be on the accept list (section 2) or the producer cannot approve it. The producer accepts by connection identity:

# Producer lists who is knocking
gcloud compute service-attachments describe producer-sa \
  --project=producer-prj --region=us-central1 \
  --format="value(connectedEndpoints[].status,connectedEndpoints[].pscConnectionId)"

# Producer accepts a specific consumer project (and sets its endpoint cap)
gcloud compute service-attachments update producer-sa \
  --project=producer-prj --region=us-central1 \
  --update-consumer-accept-list=consumer-prj-a=10

The consumer watches the forwarding rule’s pscConnectionStatus flip from PENDING to ACCEPTED.

4. Sizing and isolating the PSC NAT subnet

This is where teams self-inflict outages. The NAT subnet supplies source addresses for all consumer traffic flowing through the attachment. Each PSC connection consumes source-port space from one NAT IP. A /29 looks fine in a demo and exhausts under real fan-in.

How to size it:

• PSC allocates source ports from the NAT subnet IPs. The usable host count is total addresses - 4 (GCP reserves the first two, the last, and the network/broadcast equivalents). A /24 yields ~252 usable NAT IPs.
• Each NAT IP offers tens of thousands of source ports, but the practical ceiling is connection concurrency and churn, not a clean per-IP number. Treat the subnet as a pool and over-provision: a high-fan-in data service (think a shared Kafka or a database proxy with thousands of consumer connections) should start at a /24, not a /28.
• The subnet purpose=PRIVATE_SERVICE_CONNECT is single-use. It cannot host VMs, it cannot be shared with another attachment’s NAT role, and you cannot resize it down once allocated. You can attach multiple NAT subnets to one service attachment to grow the pool, which is the supported way to scale without recreating the attachment.

# Grow capacity by adding a second NAT subnet to the existing attachment
gcloud compute networks subnets create psc-nat-subnet-2 \
  --project=producer-prj --network=producer-vpc --region=us-central1 \
  --range=10.100.1.0/24 --purpose=PRIVATE_SERVICE_CONNECT

gcloud compute service-attachments update producer-sa \
  --project=producer-prj --region=us-central1 \
  --nat-subnets=psc-nat-subnet,psc-nat-subnet-2

The isolation rule: give every service attachment its own NAT subnet, carved from a CIDR block you reserve specifically for PSC NAT. Do not let it overlap your VM subnets, GKE pod/service ranges, or anything you might peer. The NAT range only appears as a source inside the producer VPC, so it need not be globally unique, but a dedicated supernet (e.g. 10.100.0.0/16 for “all PSC NAT”) makes firewall rules and audits trivial.

5. PSC for Google APIs: custom endpoints instead of Private Google Access

Private Google Access routes API traffic to Google over the default 199.36.153.8/30 (restricted) or .4/30 (private) ranges. PSC for Google APIs replaces that with an endpoint you own, inside your address space, which is what unlocks consistent on-prem reach over Interconnect and per-VPC API control.

You create a global internal address and a global forwarding rule whose target is a Google API bundle, not a service attachment. Use all-apis for the general bundle or vpc-sc when you require VPC Service Controls enforcement on the path.

# Reserve a global internal IP for the API endpoint
gcloud compute addresses create psc-googleapis-ip \
  --project=consumer-prj-a \
  --global \
  --purpose=PRIVATE_SERVICE_CONNECT \
  --addresses=10.250.0.5 \
  --network=consumer-vpc

# Point a global forwarding rule at the Google APIs bundle
gcloud compute forwarding-rules create psc-googleapis-fr \
  --project=consumer-prj-a \
  --global \
  --network=consumer-vpc \
  --address=psc-googleapis-ip \
  --target-google-apis-bundle=all-apis

Now any request to that IP reaches Google APIs. The next section makes clients use it transparently.

6. DNS automation: PSC zones and the p.googleapis.com pattern

An IP nobody resolves to is useless. PSC for Google APIs has a designated hostname pattern: <endpoint>-p.googleapis.com and the wildcard *.p.googleapis.com, which Google publishes as PSC-routable. You create a private DNS zone that maps these names to your endpoint IP so existing SDKs and gcloud keep using normal hostnames.

# Private zone for the PSC Google APIs domain, attached to the consumer VPC
gcloud dns managed-zones create psc-googleapis-zone \
  --project=consumer-prj-a \
  --dns-name="p.googleapis.com." \
  --visibility=private \
  --networks=consumer-vpc \
  --description="PSC endpoint for Google APIs"

# A wildcard A record so every *.p.googleapis.com resolves to the endpoint
gcloud dns record-sets create "*.p.googleapis.com." \
  --project=consumer-prj-a \
  --zone=psc-googleapis-zone \
  --type=A --ttl=300 \
  --rrdatas=10.250.0.5

For the standard service hostnames clients already use (storage.googleapis.com, bigquery.googleapis.com), add CNAMEs into the googleapis.com private zone pointing at the corresponding p.googleapis.com name, so a storage.googleapis.com lookup ultimately resolves to your PSC IP without touching application config. For a producer-published service attachment (not Google APIs), the producer can supply a --domain-names value on the attachment and you front it with your own private zone mapping that domain to the endpoint IP.

Set a modest TTL (300s here). When you migrate the endpoint IP or fail over regions, you do not want clients pinned to a stale record for an hour.

7. Securing and observing PSC

PSC is private but not automatically safe. Three controls matter.

Firewall the NAT source on the producer. Producer-side traffic arrives sourced from the NAT subnet. Lock the backend so only PSC-originated traffic on the service port is allowed, denying lateral movement from elsewhere in the producer VPC.

gcloud compute firewall-rules create allow-psc-ingress \
  --project=producer-prj \
  --network=producer-vpc \
  --direction=INGRESS \
  --action=ALLOW \
  --rules=tcp:443 \
  --source-ranges=10.100.0.0/16 \
  --target-tags=psc-backend

Cap connections per consumer. The connection_limit on the accept list (sections 2 and 3c) is the enforcement point. Set it per tenant; an unbounded limit means one consumer can exhaust the NAT pool for everyone.

Turn on flow logs and check capacity. Enable VPC Flow Logs on the NAT subnet and the consumer subnet to get the 5-tuple for every PSC flow, then watch the attachment’s connection count against its limit.

gcloud compute networks subnets update psc-nat-subnet \
  --project=producer-prj --region=us-central1 \
  --enable-flow-logs \
  --logging-aggregation-interval=interval-30-sec \
  --logging-flow-sampling=0.5

In Cloud Logging, the producer can confirm which consumer connection IDs are live and whether any are being dropped near the limit:

resource.type="gce_subnetwork"
logName=~"compute.googleapis.com%2Fvpc_flows"
jsonPayload.connection.dest_port="443"
jsonPayload.src_vpc.subnetwork_name="psc-nat-subnet"

Verify

Run these end to end before declaring victory.

# 1) Producer: attachment is up and lists the expected consumers as ACCEPTED
gcloud compute service-attachments describe producer-sa \
  --project=producer-prj --region=us-central1 \
  --format="table(connectedEndpoints[].pscConnectionId, connectedEndpoints[].status)"

# 2) Consumer: the endpoint forwarding rule shows ACCEPTED, not PENDING/REJECTED
gcloud compute forwarding-rules describe psc-endpoint-fr \
  --project=consumer-prj-a --region=us-central1 \
  --format="value(pscConnectionStatus)"

# 3) Consumer VM: the endpoint IP actually answers on the service port
curl -sS -o /dev/null -w "%{http_code}\n" https://10.20.0.50/healthz

# 4) PSC for Google APIs: the hostname resolves to YOUR endpoint IP
nslookup storage.googleapis.com    # expect 10.250.0.5, not a public Google IP

# 5) From the consumer, confirm an actual API call traverses the endpoint
gcloud storage ls --project=consumer-prj-a   # succeeds with no public egress

A green run means: attachment ACCEPTED, endpoint ACCEPTED, the service IP returns 200, Google API hostnames resolve into your PSC range, and a real API call works with no Private Google Access route in play.

Enterprise scenario

A payments platform team ran a shared transaction-fraud scoring service in a central producer-prj, published over PSC with ACCEPT_MANUAL to roughly 40 tenant projects. They sized the NAT subnet at /26 (~60 usable IPs) because “we only have 40 consumers.” During a regional promotion event, three high-volume tenants opened thousands of concurrent gRPC streams each. New connections began failing intermittently while existing ones held; the producer LB looked healthy, and CPU was flat. Flow logs on the NAT subnet showed source-port pressure on the /26 pool and a rising count of dropped SYNs that never reached the backend.

The constraint: PSC source-NATs every consumer connection through the NAT subnet’s IPs, and a /26 simply did not have the source-address headroom for that connection churn. You cannot resize a PSC NAT subnet down, and they did not want to recreate the attachment (which would have bounced all 40 tenants).

The fix used the supported scale path: add NAT subnets to the existing attachment, non-disruptively.

gcloud compute networks subnets create psc-nat-2 \
  --project=producer-prj --network=producer-vpc --region=us-central1 \
  --range=10.100.2.0/24 --purpose=PRIVATE_SERVICE_CONNECT

gcloud compute networks subnets create psc-nat-3 \
  --project=producer-prj --network=producer-vpc --region=us-central1 \
  --range=10.100.3.0/24 --purpose=PRIVATE_SERVICE_CONNECT

# Attach both alongside the original; no consumer reconnection required
gcloud compute service-attachments update producer-sa \
  --project=producer-prj --region=us-central1 \
  --nat-subnets=psc-nat-subnet,psc-nat-2,psc-nat-3

They also tightened connection_limit per tenant so a single noisy consumer could no longer monopolize the pool, and moved their PSC NAT allocations into a documented 10.100.0.0/16 supernet so the next capacity bump is a one-line CIDR pull, not an archaeology project. Lesson: NAT subnet sizing is a concurrency-and-churn decision, not a headcount of consumers.

Checklist

• Decided endpoint vs backend per consumer (flat reach vs needing your own LB/Cloud Armor/multi-region)
• Producer service sits behind a supported internal load balancer (passthrough or internal ALB)
• Dedicated NAT subnet with purpose=PRIVATE_SERVICE_CONNECT, sized for concurrency, from a reserved PSC supernet
• ACCEPT_MANUAL plus an explicit accept list with per-consumer connection_limit for any multi-tenant or external attachment
• --enable-proxy-protocol set only if the backend parses PROXY protocol (else attribution and stream both break)
• Consumer endpoint/backend created and confirmed ACCEPTED, not stuck PENDING/REJECTED
• PSC for Google APIs endpoint plus *.p.googleapis.com private zone and CNAMEs so SDKs need no config change
• Producer firewall restricts ingress to the NAT source range on the service port only
• VPC Flow Logs enabled on the NAT subnet; connection count watched against the limit
• DNS TTLs modest (e.g. 300s) so endpoint/region migration is not blocked by stale records
• Capacity scaling path validated: add NAT subnets to the existing attachment rather than recreating it