Network isolation in us-central1-b and us-central1-f after router maintenance error

Summary

On Tuesday, 1 September 2026, some customers in us-central1 experienced network service degradation and instance isolation for a duration of 4 hours and 11 minutes. To our customers whose businesses were impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.

Root Cause

The event was triggered by the inadvertent physical disconnection of network fiber-optic cables during routine hardware maintenance. A technician was performing a scheduled capacity upgrade on data center routers that support a fraction of capacity in the us-central1-b zone and a small fraction of capacity in the us-central1-f zone.

During the hands-on maintenance, optical fibers connecting the data center routers to the network fabric were unplugged and the optical transceivers were replaced with a transceiver that supports higher-density connectivity. This transceiver was not compatible with the transceivers still in place in the data center fabric, causing a loss of connectivity for each fiber.

The network is designed with redundant routers in separate data center rooms within the same building so that failure of one router does not interrupt service. The maintenance was intended to upgrade one router at a time over several days, with traffic diversions at each step and verification between steps that the network has returned to a fully-connected state.

However, a procedural error in the manually orchestrated upgrade process caused the complete list of transceiver replacements across all routers to be issued to the technician without instructions to sequence the work one router at a time. The maintenance workflow also did not include the expected human and software verification steps for detecting unintended disruption.

As a result, the technician sequentially unplugged all fiber paths across the affected devices within 13 minutes. The speed and nature of the error prevented warnings of incorrect action from reaching the engineer before connectivity was lost. Additionally, a standing procedure to halt if light is detected on any fiber-optic cable after unplug was not followed.

This resulted in compute capacity in the impacted zone being isolated from the network. Customers were unable to reach their virtual machines, and those virtual machines could not establish connections outside their zone.

Remediation and Prevention

The issue was detected immediately by automated network loss monitoring systems as well as proactive probes, which rapidly engaged the network engineering and incident response teams.

To mitigate the immediate customer impact, engineering teams actively moved traffic away from the impacted infrastructure.

Concurrently, hardware operations technicians on site identified the disconnected optical links and physically re-inserted the original optical transceivers. Once the physical links were fully restored, traffic flow rates normalized and the traffic was redirected back to return the capacity to service.

Google is committed to preventing a repeat of this issue in the future and is completing the following actions:

• Reinforce training in us-central1 region and globally around safe maintenance techniques, including the mandatory requirement to verify whether light is on every unplugged fiber.
• Finish the migration of network upgrade workflows to a fully automatically sequenced orchestration system. Most common regional network workflows were completed in 2025, and zonal workflows are in progress.
• Complete the deployment of work stop alerting for actions resulting in the unintended disconnection of live fiber cables in Google data centers.
• Complete the deployment of the automated system that will move regional traffic away from faulty zones, reducing the time from approximately 19 minutes in this instance to around 5 minutes.

Detailed Description of Impact

On Tuesday, 1 September, from 07:41 to 11:52 US/Pacific, a portion of the us-central1-b and us-central1-f zones experienced severe network degradation and resource isolation.

1. Multiple products affected: The network disruption to/from a portion of us-central1-b and us-central1-f zones impacted all zonal GCP Products for some customers in those zones.
2. Regional impact: Regional products using capacity in the affected data center were affected until traffic diversions were fully in place at 08:00 US/Pacific. Google Kubernetes Engine and Cloud Run / Google App Engine had longer impacts, discussed below.
3. Error rate: Traffic flow drop rates for resources hosted in the affected area reached 100% during the peak of the incident, resulting in unreachable virtual machines and elevated packet loss. Affected Services and Features

• Google Compute Engine: Inability for users in the affected portion of us-central1-b and us-central1-f zones to access virtual machines externally, and inability for VMs to reach remote resources.
• Cloud SQL, AlloyDB for PostgreSQL, Google Cloud Bigtable, Cloud Filestore: Data access and connectivity severed for instances localized strictly to the impacted infrastructure.
• Cloud Spanner, Cloud Firestore: Elevated Remote Procedure Call (RPC) error rates, write timeouts, and latency spikes for the nam5 multi-region instances due to severed connectivity to components running in the impacted zone.
• Virtual Private Cloud (VPC), Cloud NAT, Apigee, Google BigQuery, Google Cloud Dataflow, Looker, Google SecOps SOAR, Hybrid Connectivity, Cloud Interconnect: Elevated network packet loss, connectivity drops, and API timeouts in the impacted zone.
• Google Kubernetes Engine (GKE): Between 07:40 and 09:15 US/Pacific on 2026-09-01, customers experienced errors and timeouts on requests to their GKE cluster control planes (Kubernetes API servers) in us-central1. This could have caused new or updated workloads to fail scheduling and Kubernetes API operations to fail.
• Cloud Run / Google App Engine: Temporary latency spikes and pending queue aborts as backend workloads automatically evacuated and shifted to healthy capacity. A subset of workloads in the specifically impacted area experienced degradation until 11:52 US/Pacific. Customer Impact

Customers with resources hosted in the specific affected clusters within us-central1-b and us-central1-f experienced a complete loss of network connectivity starting at 07:41 US/Pacific.

During this time, virtual machines and associated services became unreachable from the internet and from other Google Cloud regions, and those internal resources could not initiate outbound connections.

The issue was detected almost immediately and engineers took traffic diversion actions at 07:45 and 08:00 US/Pacific to reduce impact to regional products.

By 08:50 US/Pacific, the majority of physical connections were restored and traffic began recovering, and the traffic diversion actions were removed at 09:19 US/Pacific.

Full recovery across nearly all affected services and long-running operations was confirmed by 11:52 US/Pacific.

Because the network isolation was strictly limited to specific clusters within us-central1-b and us-central1-f, multi-zonal deployments correctly utilizing redundancy across other zones in us-central1 were largely able to bypass the physical hardware failure and continue serving traffic.

Customers with projects in both us-central1-b and us-central1-f would not have seen multi-zonal impact - at most one of their zones would have capacity in the affected data center.

Addendum to Incident Report

This addendum extends the Detailed Description of Impact from the previous message. Though only two clusters in a datacenter in us-central1-b and us-central1-f experienced network isolation, a few regional products with a dependency on the affected datacenter were also affected. Between 07:41 and 11:52 US/Pacific on Tuesday, September 1, 2026, the following products were affected:

• Compute Engine VMs: VMs running in the affected data center lost network connectivity to resources located outside of their datacenter. From a Compute Engine perspective, this incident only affected a subset of VMs in us-central1-b and us-central1-f.
• Zonal and Regional Products Based on Compute Engine Products with dependencies on VMs in the affected datacenter experienced symptoms like connection timeouts or service unavailability because the underlying VMs lost network connectivity.

Examples of products that are based on Compute Engine VMs include: Apigee, Cloud SQL, AlloyDB for PostgreSQL, Dataflow, Cloud Filestore, and Looker.

• Google Kubernetes Engine (GKE): GKE clusters whose control plane had dependencies on Compute Engine VMs in the affected clusters experienced errors and timeouts when administrators or nodes attempted to connect to the Kubernetes API server. Consequently, deployments might have been delayed, Kubernetes API operations might have failed, and new nodes might not have registered with the control plane (causing node pool operational delays).
• Cloud Run and App Engine: Workloads in the us-central1 region experienced elevated serving latency, 5xx aborted request errors, and outbound connectivity degradation. To prevent severe request failures, traffic was diverted to other clusters. Diverting requests to other clusters temporarily caused elevated latency or errors in those clusters.Once the impacted cluster was restored, workload demand for these new backends created a ‘thundering herd’ of concurrent instance initializations. This sudden surge bypassed warm caches and overwhelmed internal file-serving tiers, extending symptoms until emergency capacity was provisioned.
• Other Google APIs and Services, in us-central1 or a multi-region that includes us-central1, were affected if they had dependencies on software tasks in the affected clusters and weren’t able to use software tasks in an unaffected cluster. Symptoms included connection timeouts or service unavailability. Examples of other Google APIs and services are: Google Cloud Bigtable, BigQuery, Google SecOps SOAR, Cloud Spanner, and Cloud Firestore. For some services, symptoms were limited to temporary latency spikes and service unavailability until workloads shifted to unaffected clusters.
• Private Service Connect: Clients (in any region) experienced connectivity issues involving PSC endpoints with dependencies in us-central1, in specific circumstances: - PSC endpoints for Google APIs were affected in the same way as noted in the Google APIs and services section. - Producer/consumer PSC endpoints were affected if the producer had load balancers or backend VMs in the affected clusters of us-central1.
• Hybrid Connectivity: Some Cloud Router, Cloud VPN, and Cloud Interconnect VLAN attachments in us-central1 were affected in specific circumstances: - Cloud Router BGP software tasks for some Cloud Routers in us-central1 experienced a Cloud Router maintenance event if their BGP software tasks were running in the affected clusters. - Cloud VPN tunnel software tasks running in the affected clusters lost network connectivity. This caused some Cloud VPN tunnels in us-central1 to go down until a replacement tunnel task in a different cluster was assigned. - Resources (in any region) sending packets through Cloud Interconnect VLAN attachments in us-central1 might have experienced intermittent connectivity issues in certain situations where network flows weren’t already programmed.
• Envoy-based load balancers and other products: Load balancers and other products that depended on managed Envoy tasks of a proxy-only subnet in us-central1 experienced increased latency or connectivity issues if enough Envoy tasks were in the affected clusters. The following products were affected: - Regional external application load balancers in us-central1 - Regional internal application load balancers in us-central1 - Cross-region internal application load balancers with proxies in us-central1 - Regional external proxy network load balancers in us-central1 - Regional internal proxy network load balancers in us-central1 - Cross-region internal proxy network load balancers with proxies in us-central1 - Secure Web Proxy in us-central1