Google Cloud Storage elevated write latency and error rates in us-central1

Incident Report.

Date/Time of the Issue (All time US/Pacific)

Incident First Occurrence Incident Start: 08 October 2026 04:00 Incident End: 08 October 2026 16:30 Duration: 12 hours, 30 minutes Incident Recurrence Recurrence Start: 09 October 2026 04:05 Recurrence End: 09 October 2026 09:46 Duration: 5 hours, 41 minutes Total Duration: 18 hours, 11 minutes

Summary

On Thursday, 08 October 2026, Google Cloud Storage experienced elevated write latency and write error rates while performing upload or write operations in the us-central1 region for a duration of 12 hours, 30 minutes. This was followed by a recurrence on Friday, 09 October 2026 in the us-central1-b zone for a duration of 5 hours, 41 minutes. We are taking immediate steps to ensure this doesn’t happen again.

Root Cause

Google Cloud Storage relies on a highly distributed backend architecture to process and store incoming data. During the incident window, specific storage backend nodes within the us-central1 region experienced a localized capacity shortfall due to increased traffic. The sequence of events leading to customer impact was as follows:

• A traffic sorting optimization configuration update to the Google Cloud Storage load balancing layer intended to address localized capacity constraints caused severe resource exhaustion on specific storage backend nodes in the affected region by inadvertently bypassing the standard capacity-aware routing logic, resulting in a severe load imbalance.
• Consequently, a small fraction of storage backend nodes in the us-central1 region received an overwhelming proportion of incoming write traffic despite having low available serving capacity. This concentration of traffic led to resource exhaustion on the affected backend servers causing backend services handling object data ingestion to begin timing out.
• Because the underlying instances could not process the high volume of requests quickly enough, the incoming upload and write requests routed to these nodes experienced elevated latency and failed to complete, resulting in errors surfaced to customers.
• Concurrently, a separate issue with the underlying database service contributed to additional metadata processing errors during a portion of the incident window.
• The recurrence of the issue on October 9 was caused by an incomplete rollback of an upload planner feature, which broke the load balancing mechanism and overloaded a few smaller storage backend nodes.

Remediation and Prevention

The issue was initially detected on 08 October 2026 at 04:50 by Google engineering teams actively monitoring system telemetry early in the incident window.  To mitigate the issue, Google engineers executed the following step-by-step strategy:

• Engineers initially attempted to dynamically allocate additional emergency capacity to the affected region, but this was blocked by systemic regional capacity limits.
• To relieve immediate pressure, engineers selectively drained traffic from the most overloaded storage backend nodes.
• A configuration change was deployed to disable the traffic sorting optimization, significantly reducing resource overhead on the storage backend.
• For the concurrent metadata issue, engineers rolled back a recent update to the underlying metadata database service.
• While these steps provided temporary relief, a recurrence of the load imbalance required further traffic redirection and draining of affected storage components. The incident was permanently resolved when engineering fully reverted the problematic load balancing configuration change, restoring the previous capacity-aware routing behavior and allowing traffic to distribute evenly across the region. Once traffic levels stabilized and resource utilization returned to normal baseline levels, the previously drained nodes were restored to active service, successfully restoring normal operations. Preventative Actions: To ensure this issue does not recur, we are implementing the following improvements:

• Safe Configuration Rollouts: We are updating our deployment protocols to ensure that load balancing logic changes are rigorously validated in alternative, smaller regions before being rolled out to larger, highly utilized regions.
• Intelligent Operational Controls: We are enhancing the storage load balancing architecture to strictly enforce capacity constraints, ensuring that traffic distribution logic cannot bypass pre-spilling safeguards and route traffic to clusters lacking sufficient serving resources.
• Resilient Scaling Logic: We are implementing targeted improvements to accelerate internal data movement within highly utilized storage cells, enabling the system to process data more efficiently and handle temporary load spikes with greater resilience.

Description of Impact

On 08 October 2026, from 04:00 to 16:30 in us-central1 region and 09 October 2026, from 04:05 to 09:46 in us-central1-b zone, customers experienced elevated latency and error rates during Cloud Storage upload and write operations.

• Upload & Write Failures: Customers primarily observed HTTP 503 UNAVAILABLE errors on WriteObject operations and client-side timeouts.
• Increased Write Latency: Write operations that did succeed experienced significantly higher latencies than normal.
• Cloud Storage operations in all other regions, as well as all other Google Cloud services, remained unaffected by this specific event.