In March, GitHub experienced a single incident, occurring in two related waves, that degraded specific services.
March 29-31 Incident (7:00 UTC March 29 to 17:00 UTC March 31, spanning approximately 58 hours)
Email subscription management (github.com/settings/emails) and resource registration forms (resources.github.com ebook and event registration) were degraded.
Root Cause: The March 29 incident occurred due to expired credentials used for an internal service. A similar credential expiry on March 31 resulted in users experiencing degradation again. The underlying problem was an issue in the automated alerting for monitoring upcoming credential expirations — a bug in alerting resulted in the invalid credentials being discovered only after they had expired.
Resolution: GitHub mitigated the incident by renewing the credentials and redeploying the affected services.
Preventive Measures: GitHub enhanced its credential expiry detection, alerting, and rotation processes, and is working on improving on-call observability.
Tags
Only on this entry: expiry