500 errors on GitLab SaaS when accessing groups and projects with secrets manager enabled (Severity 2)

Problem

Groups and projects with GitLab Secrets Manager enabled are returning 500 errors when users open GitLab UI.

Impact

All SaaS groups and projects with GitLab Secrets Manager enabled received 500 errors when opening work items, merge requests, and any page that includes the sidebar. The incident started at 03:57 UTC and ended at 07:20 UTC.

Causes

Kubernetes probes for the OpenBao pod were not properly configured. As a result, we entered a loop where the OpenBao active node was restarted before it could complete its startup sequence and become available. Additionally, GitLab UI was calling OpenBao when rendering the sidebar without safely handling unexpected non-JSON responses. Ultimately this resulted in GitLab returning 500 errors for groups and projects where the GitLab Secrets Manager is enabled.

Response strategy

We fixed the configuration of the probes on Kubernetes. We’re now implementing a fix for the sidebar so that it fails gracefully when OpenBao is not available.