Contents · 3 sections
In May, GitHub experienced three incidents affecting service availability.
May 1 Incident (22:09 UTC, lasting 1 hour 4 minutes)
Between 22:09 and 23:13 UTC, users of the Issues service were unable to upload attachments; approximately 130,000 users were affected for roughly 45 minutes.
Root Cause: A new feature added a custom header to all client-side HTTP requests, causing CORS errors when uploading attachments.
Resolution: The feature flag was rolled back at 22:56 UTC.
Preventive Measures: GitHub is adding new metrics to monitor safe rollout of client-side request changes; an augmented feature version has since been deployed and is performing well in production.
May 28 Incident (09:45 UTC, lasting 5 hours)
Between 09:45 and 14:45 UTC, Ubuntu-24 standard hosted runners on public repositories experienced delayed job starts, with approximately 19.7% of jobs delayed.
Root Cause: A misconfiguration in backend caching behavior after a failover, which led to duplicate job assignments reducing overall capacity.
Resolution: The configuration was fixed by 12:45 UTC and pools were scaled up to process the backlog.
Preventive Measures: GitHub is improving failover resiliency and validation protocols.
May 30 Incident (08:10 UTC, lasting 7 hours 50 minutes)
Between 08:10 and 16:00 UTC, the Microsoft Teams GitHub integration service experienced a complete outage, with a 100% error rate across all functionality except link previews.
Root Cause: An authentication issue with GitHub’s downstream authentication provider.
Resolution: GitHub worked with the provider to restore authentication functionality.
Preventive Measures: GitHub is migrating to more durable authentication methods to reduce similar risks.