In March, GitHub experienced two incidents that resulted in degraded performance across services.
Incident: March 15, 2024 (19:42–20:24 UTC, lasting 42 minutes)
Root Cause: A regression in the permissions system, caused by a framework upgrade that introduced MySQL query syntax incompatible with the database proxy service used in some production clusters.
Impact: GitHub Codespaces, GitHub Actions, and GitHub Pages were affected.
Resolution: GitHub rolled back the deployment and corrected a misconfiguration in development and CI environments to prevent recurrence.
Incident: March 11, 2024 (22:45 UTC to 00:48 UTC the following day, lasting 2 hours 3 minutes)
Root Cause: An inadvertent deployment of network configuration to the wrong environment led to intermittent errors.
Impact: API requests, GitHub Copilot, GitHub secret scanning, and 2FA via GitHub Mobile were affected.
Resolution: The issue was detected within 4 minutes and a rollback was initiated immediately; most impact was mitigated by 22:54 UTC. However, the rollback failed in one data center because system-created configuration records were missing a required field, causing 0.4% of requests to continue failing. Manual intervention corrected the configuration data, enabling full service restoration by 00:48 UTC.
Preventive Measures: GitHub implemented measures for safer configuration changes, including prevention and automatic cleanup of obsolete configuration and faster issue detection.