Contents · 3 sections
In July, GitHub experienced one new incident affecting Codespaces, plus detailed follow-ups on two previously reported June incidents.
Incident 1: July 27 Codespaces Degradation (22:29 UTC, lasting 5 hours 55 minutes)
Alerting systems detected reduced availability affecting Codespaces services in US West and East regions. Root cause was under investigation at time of publication; GitHub stated they were “still investigating the contributing factors and will provide a more detailed update on cause and remediation in the August Availability Report.”
Incident 2: June 28 DNS Migration Issue (Follow-up) (17:16 UTC, lasting 26 minutes)
Codespaces became unavailable. Codespaces services rely on a global DNS record using a load balancer to route requests to regional backends. During an infrastructure migration, the existing record was deleted to replace it, but “adding the new replacement record failed.”
Root Cause: Failed DNS record replacement during load balancer migration; the subsequent rollback also initially failed.
Resolution: Disabled an endpoint in the old load balancer, enabling a successful rollback. Future plans include safer testing mechanisms and atomic DNS updates via mirrored testing zones.
Incident 3: June 29 Multi-Service Outage (Follow-up) (14:48 UTC, lasting 1 hour 27 minutes)
GitHub Actions, API requests, Codespaces, Git operations, Packages, and Pages experienced issues. Internal APIs returned 502 errors. A data migration service’s exception handler “captured all exceptions rather than just a subset,” triggering continuous requeuing. This generated “hundreds of thousands of jobs” overwhelming the database proxy server.
Root Cause: Overly broad exception handling in the migration service combined with retry logic caused cascading job requeuing.
Resolution: Paused new migrations (15:07 UTC) and pruned job queues. Updated exception handling to retry only specific error types with fixed retry limits.