Today's outage for several Google services

Author: Ben Treynor, VP of Engineering

Earlier today, most Google users accessing logged-in services including Gmail, Google+, Calendar, and Documents experienced unavailability for approximately 25 minutes. About 10 percent of users faced extended disruptions lasting roughly 30 additional minutes. Google apologized for missing “the mark today” despite efforts to maintain service availability and speed.

The incident stemmed from a software bug in an internal configuration-generation system occurring at 10:55 a.m. PST. This system produced incorrect configuration data distributed to live services over 15 minutes, causing user requests to be disregarded and generating errors. Users began encountering these errors at 11:02 a.m., triggering alerts to Google’s Site Reliability Team. At 11:14 a.m., the same system automatically generated corrected configuration and distributed it. Service restoration accelerated immediately, with full restoration achieved by 11:30 a.m.

Google committed to three corrective measures: fixing the configuration generator bug and auditing similar systems; implementing additional validation checks preventing future bad configurations from disrupting services; and deploying targeted monitoring for faster detection and diagnosis of service failures.