GitLab.com down (Severity 2)

Problem

GitLab.com experienced an availability outage with elevated 5xx errors and timeouts across web and API, and some Git operations appeared stalled.

Impact

GitLab.com had an availability outage with elevated 5xx errors and timeouts across web and API, and some Git operations appeared stalled. Main error rates returned to baseline by about 16:23 UTC, and Duo (ai-assisted) was heavily degraded for a period before recovering. By 17:09 UTC, web, API, Git, and ai-assisted traffic had returned to pre-incident levels.

Causes

A single project bot generated most of the merge request discussion-reply note creations in one very large thread, and each additional reply became more expensive to process. This drove the CPU and GVL contention spike in the note-creation path.

Response strategy

We confirmed the broad platform impact was driven by application-side saturation rather than a database or release regression. We confirmed the web pod restarts were caused by kubelet liveness probe failures, manually deleted the startup-hung ai-assisted pods so replacements could become Ready, and disabled the GitLab account associated with the high-volume activity that likely triggered the incident.