Incident Postmortem - High error rates on 1Password.com when syncing data and provisioning users

Postmortem

Date of Incident: 16 October 2026
Time of Incident (UTC): 23:00 - 00:11 (Oct 17)
Service(s) Affected: Admin Console, Item Syncing, Provisioning
Impact Duration: 1hr10m

Summary

What happened: On Friday, October 16, 2026, from 4:00 to 5:10 p.m. Pacific (23:00–00:10 UTC), some requests to 1Password web services failed. Earlier that day, we deployed an infrastructure change meant to improve efficiency. The change caused one of our backend systems to hold on to resources longer than expected, and the system eventually ran out of capacity.

Impact: About 2% of requests to 1Password web services in the US failed during the 70-minute window. The issue happened during a period of low traffic, which limited the number of people affected.

Resolution: We increased the affected system’s capacity and applied configuration changes so resources are recycled more quickly.

Impact on Customers

• Accessing vault contents. Users experienced intermittent failures when their 1Password clients tried to sync vault contents.
• Provisioning. Certain public endpoints used to provision users from customer identity providers were unavailable, leading to delayed user provisioning.
• Number of Affected Customers (approximate): 15
• Geographic Regions Affected: United States

What Happened?

Timeline of Events (UTC):

• 23:00 (Oct 16): Error rates for 1Password web services begin climbing.
• ~23:20: Automated alerts notify on-call engineers.
• 23:27: Incident declared; engineers begin investigating.
• 23:45: Root cause identified; engineers begin preparing a fix.
• 00:10 (Oct 17): Affected systems recover, errors stop, and normal operations resume.
• 02:55 (Oct 17): Configuration changes applied to prevent the issue from recurring.
• 18:51 (Oct 20): Permanent fixes applied.

Root Cause Analysis:

On October 16, 2026, 1Password web services returned an elevated number of errors because certain data systems ran out of a type of memory used to process read-only requests. When this memory was full, those requests failed.

Each open connection to these systems could hold a small piece of this memory in reserve, even while sitting idle. Earlier that day, we rolled out a new tool that keeps connections open after use so they can be reused, which normally improves performance. Its settings allowed too many idle connections to accumulate. Over the afternoon, the number of open connections grew until it exceeded the available memory, at which point requests began to fail.

How Was It Resolved?

• Mitigation Steps: Engineers increased the amount of memory available to the affected systems, providing extra capacity to support the system’s projected traffic.
• Resolution Steps: After a period of observation, engineers applied further configuration adjustments and lowered the limit on the number of idle connections the connection management tool can keep open.
• Verification of Resolution: We improved monitoring to detect any recurrence of the issue. The fix remained in place and was observed throughout the weekend of October 17–18.

What We Are Doing to Prevent Future Incidents

• We improved monitoring to monitor the specific type of memory exhausted during the incident. The monitor pages engineers well before 1Password services begin exhibiting errors.
• We updated our response procedures for handling this type resource exhaustion.
• We set stricter limits on our backend systems so they no longer hold on to memory longer than necessary.

Next Steps and Communication

• No action is required from our customers at this time.

We are committed to providing a reliable and stable service, and we are taking the necessary steps to learn from this event and prevent it from happening again. Thank you for your understanding.

Sincerely,

The 1Password Team

Incident updates

Resolved (2026-10-20 15:48 EDT): This incident has been resolved.

Monitoring (2026-10-19 16:12 EDT): The deployed fix mitigated the issue causing increased error rates for users. We continue to monitor our services while we implement a permanent fix.

Monitoring (2026-10-16 23:13 EDT): A fix has been deployed and we are monitoring the results.

Investigating (2026-10-16 20:34 EDT): We are seeing some improvements to affected services but are continuing to investigate the issue.

Investigating (2026-10-16 19:45 EDT): We are currently investigating this issue.