Customer Portal Returning 404 (Severity 2)

Problem

The Customer Portal is intermittently returning 404 errors and failing to load some page content.

Impact

Support and Fulfillment saw intermittent Customers Portal failures since around 21:30 UTC, including 404s on the Okta sign-in path, pages loading without the login form, and some subscription pages showing only headers. No customer tickets have come in.

Causes

The CustomersDot production deploy at 19:22 UTC failed partway when 3 of 6 hosts disconnected during asset precompilation. A later redeploy also hit a timeout while checking GitLab API for deployment blockers, which left one host on an older release until a subsequent deploy job retry completed.

Response strategy

We retried the production deployment using the documented redeployment process. We then retried it again at 23:27 UTC, and it completed on all 6 production hosts at 23:35 UTC with no errors. Since then, repeated checks have shown sign-in working and pages loading fully without JavaScript asset 404s.