A postmortem of three recent issues

Engineering at Anthropic — Published Sep 17, 2025

This is a technical report on three bugs that intermittently degraded responses from Claude. Below we explain what happened, why it took time to fix, and what we’re changing.

Between August and early September, three infrastructure bugs intermittently degraded Claude’s response quality. We’ve now resolved these issues and want to explain what happened.

To state it plainly: We never reduce model quality due to demand, time of day, or server load. The problems our users reported were due to infrastructure bugs alone.

We recognize users expect consistent quality from Claude, and we maintain an extremely high bar for ensuring infrastructure changes don’t affect model outputs. In these recent incidents, we didn’t meet that bar.

How we serve Claude at scale

We serve Claude to millions of users via our first-party API, Amazon Bedrock, and Google Cloud’s Vertex AI. We deploy Claude across multiple hardware platforms, namely AWS Trainium, NVIDIA GPUs, and Google TPUs. Each hardware platform has different characteristics and requires specific optimizations. Despite these variations, we have strict equivalence standards for model implementations. Our aim is that users should get the same quality responses regardless of which platform serves their request.

Timeline of events

The overlapping nature of these bugs made diagnosis particularly challenging. The first bug was introduced on August 5, affecting approximately 0.8% of requests made to Sonnet 4. Two more bugs arose from deployments on August 25 and 26.

Although initial impacts were limited, a load balancing change on August 29 started to increase affected traffic. This caused many more users to experience issues while others continued to see normal performance, creating confusing and contradictory reports.

Three overlapping issues

1. Context window routing error

On August 5, some Sonnet 4 requests were misrouted to servers configured for the upcoming 1M token context window. This bug initially affected 0.8% of requests. On August 29, a routine load balancing change unintentionally increased the number of short-context requests routed to the 1M context servers. At the worst impacted hour on August 31, 16% of Sonnet 4 requests were affected.

Approximately 30% of Claude Code users who made requests during this period had at least one message routed to the wrong server type, resulting in degraded responses. Some users were affected more severely, as our routing is “sticky”: once a request was served by the incorrect server, subsequent follow-ups were likely to be served by the same incorrect server.

Resolution: We fixed the routing logic to ensure short- and long-context requests were directed to the correct server pools. We deployed the fix on September 4. Rollout was completed by September 16, and to AWS Bedrock by September 18.

2. Output corruption

On August 25, we deployed a misconfiguration to the Claude API TPU servers that caused an error during token generation. An issue caused by a runtime performance optimization occasionally assigned a high probability to tokens that should rarely be produced given the context, for example producing Thai or Chinese characters in response to English prompts, or producing obvious syntax errors in code.

This corruption affected requests made to Opus 4.1 and Opus 4 on August 25-28, and requests to Sonnet 4 August 25–September 2. Third-party platforms were not affected by this issue.

Resolution: We identified the issue and rolled back the change on September 2. We’ve added detection tests for unexpected character outputs to our deployment process.

3. Approximate top-k XLA:TPU miscompilation

On August 25, we deployed code to improve how Claude selects tokens during text generation. This change inadvertently triggered a latent bug in the XLA:TPU compiler, which has been confirmed to affect requests to Claude Haiku 3.5. We also believe this could have impacted a subset of Sonnet 4 and Opus 3 on the Claude API.

Resolution: We first observed the bug affecting Haiku 3.5 and rolled it back on September 4. We later noticed user reports of problems with Opus 3 and rolled it back on September 12. We have been working with the XLA:TPU team on a fix for the compiler bug and rolled out a fix to use exact top-k with enhanced precision.

A closer look at the XLA compiler bug

When Claude generates text, it calculates probabilities for each possible next word, then randomly chooses a sample from this probability distribution. The root cause involved mixed precision arithmetic. Our models compute next-token probabilities in bf16 (16-bit floating point). However, the vector processor is fp32-native, so the TPU compiler (XLA) can optimize runtime by converting some operations to fp32 (32-bit). This optimization pass is guarded by the xla_allow_excess_precision flag which defaults to true.

This caused a mismatch: operations that should have agreed on the highest probability token were running at different precision levels. The precision mismatch meant they didn’t agree on which token had the highest probability. This caused the highest probability token to sometimes disappear from consideration entirely.

What we’re changing

• More sensitive evaluations: we’ve developed evaluations that can more reliably differentiate between working and broken implementations.
• Quality evaluations in more places: although we run regular evaluations on our systems, we will run them continuously on true production systems to catch issues such as the context window load balancing error.
• Faster debugging tooling: we’ll develop infrastructure and tooling to better debug community-sourced feedback without sacrificing user privacy.

Evals and monitoring are important. But these incidents have shown that we also need continuous signal from users when responses from Claude aren’t up to the usual standard.