Contents · 3 sections
In January, GitHub experienced three incidents causing degraded service performance.
January 9 Incident (01:26 UTC, lasting 31 minutes)
Between 01:26 and 01:56 UTC, GitHub experienced widespread disruption across services, with error rates averaging 6% and peaking at 6.85%.
Root Cause: A deployment introduced a query that saturated a primary database server.
Resolution: The team identified the source of the problematic query and rolled back the deployment, taking 14 minutes total from engagement to identification.
Preventive Measures: GitHub plans to invest in tooling to detect problematic queries prior to deployment.
January 13 Incident (23:35 UTC, lasting 49 minutes)
Between 23:35 and 00:24 UTC, all Git operations were disrupted.
Root Cause: A configuration change related to traffic routing and testing caused GitHub’s internal load balancer to drop requests.
Resolution: The configuration change was rolled back.
Preventive Measures: GitHub will improve monitoring and deployment practices to improve time to detection.
January 30 Incident (14:22 UTC, lasting 26 minutes)
Between 14:22 and 14:48 UTC, web requests to github.com were degraded, with a peak error rate of 44%.
Root Cause: A hardware failure in the caching layer that supports rate limiting, compounded by a lack of automated failover.
Resolution: The team performed a manual failover to trusted hardware.
Preventive Measures: GitHub will implement high availability cache configuration and add resilience to cache failures.