ai-assisted rails_request error rate at 10.99% (Severity 2)

Problem

The ai-assisted service’s rails_request SLI showed elevated errors, reaching a 10.99% error rate in cny and spreading beyond canary.

Impact

The ai-assisted service’s rails_request error rate reached 10.99% in cny and spread beyond canary into main, affecting user-facing requests. Tristan also reproduced a workflow failure during the incident using Duo CLI.

Causes

We traced the issue to a deployed code change affecting Duo workflow checkpoint requests. Those requests failed with a Global ID error when a checkpoint was created without a model id.

Response strategy

We rolled back the ai-assisted rollout in gprd at 05:32 UTC, including draining gprd-cny. We also prepared and merged a revert to remove the suspected checkpoint primary key change in the canonical GitLab repo (MR !263139). The deployment later finished and the incident was resolved at 11:03:55 UTC; the reverted package was validated in canary and error spikes did not return before production deployment proceeded.