Contents · 3 sections
In August, GitHub experienced three incidents affecting service availability.
August 5 Incident (15:33 UTC, primary issue lasting 32 minutes; secondary issue lasting until 19:45 UTC)
Between 15:33 and 16:13 UTC, pushes, webhooks, notifications, and pull requests were affected, along with approximately 4% of web and REST API traffic.
Root Cause: A production database migration dropped a column from a pull request functionality table. The ORM continued to reference the dropped column in a subset of pull request queries, causing elevated error rates.
Resolution: GitHub deployed a change instructing the ORM to ignore the removed column. However, the fix applied only to the largest production environment; custom and canary environments missed the update, triggering a secondary incident affecting approximately 0.1% of pull request traffic until 19:45 UTC.
Preventive Measures: GitHub is adding application monitoring to prevent continued rollout when impact is observed, and streamlining changes across environments to prevent similar secondary incidents.
August 12 Incident (13:30 UTC, lasting 3 hours 44 minutes)
Between 13:30 and 17:14 UTC, GitHub search, and the issues, pull requests, projects, and deployments pages, along with Actions workflow and label filters, were degraded. Up to 75% of search queries failed between 14:00-15:30 UTC, and search result updates were delayed up to 100 minutes.
Root Cause: Intermittent connectivity issues between load balancers and search hosts. Retry logic masked the problems initially, but retry queues eventually overwhelmed the load balancers, causing failure.
Resolution: GitHub throttled the search indexing pipeline at 15:30 UTC to reduce load; an automated reboot of a search host at 17:14 UTC resolved the connectivity failures.
Preventive Measures: GitHub improved internal monitors and playbooks, and tuned the search cluster load balancer configuration.
August 27 Incident (20:35 UTC, lasting 46 minutes)
Between 20:35 and 21:17 UTC, Copilot averaged a 36% request failure rate (peaking at 77%), and approximately 2% of non-Copilot web and REST API traffic failed.
Root Cause: A production database migration dropped a column from a Copilot functionality table. The ORM continued to reference the dropped column, generating numerous 5xx responses, similar to the August 5 incident.
Resolution: GitHub applied a fix to the production schema at 21:15 UTC, with full recovery by 21:17 UTC.
Preventive Measures: GitHub implemented a temporary block on all drop column operations, is adding safeguards to prevent future similar issues, and is implementing graceful degradation so Copilot issues won’t impact other product features.
Tags
Only on this entry: orm