In May, GitHub experienced one incident that resulted in degraded performance across services.
Incident: May 21, 2024 (11:40 UTC, lasting 7 hours 26 minutes)
Root Cause: A scheduled operating system upgrade resulted in unintended and uneven distribution of traffic within the cluster.
Impact:
- GitHub Copilot Chat (p50 latency: 2.5s; p95 latency: 6s)
- GitHub Actions (20–60 minute delays for workflow run updates), with Actions runs displaying as stuck despite successful completion
- GitHub Enterprise Importer (longer migration times)
- Billing metrics and UI reporting showed outdated information
- No data loss occurred
Resolution: Short-term mitigation involved increasing the number of network routes between GitHub’s data centers and its cloud provider.
Preventive Measures: GitHub identified gaps in monitoring and alerting for load thresholds and committed to improving both detection and mitigation response times.