In November, GitHub experienced one incident that resulted in degraded performance across services.
Incident: November 3, 2023 (18:42–19:20 UTC, lasting 38 minutes)
Root Cause: The GitHub authorization service experienced excessive memory consumption. A performance optimization introduced a memory leak that testing did not expose the service to sufficient traffic to discover. This flaw reached production at 18:37 UTC and manifested under high load, causing pods to crash starting at 18:42 UTC.
Impact: Authorization requests failed, triggering 404 and error responses across most GitHub pages and API endpoints.
Resolution: The team rolled back the problematic authorization service change by 19:08 UTC. Recovery occurred once pods came back online.
Preventive Measures:
- Enhanced rollout strategy with automated monitoring and checks to block deployments with unsatisfactory metrics
- Removed dependencies between the authorization service and deployment rollback tools to speed future recovery