Incident Report: Retry Breadcrumb Memory Growth Causing OOM on v1.100.0

LiteLLM incident report. Published 2026-09-12.

Date: September 7 to September 9, 2026 Affected versions: v1.100.0 Severity: Medium (production outage for one customer; low blast radius, required an unreachable Redis) Status: Resolved in v1.100.1

Note: If you are running v1.100.0, upgrade to v1.100.1 or later. v1.99.1 and earlier are not affected. The v1.100.0 dev and rc pre-releases, v1.101.0-dev.1, and v1.101.0-dev.2 carry the same defect; v1.101.0-rc.1 and later are not affected.

Summary

Between September 7 and September 9, 2026, an enterprise customer running litellm-database:v1.100.0 behind nginx experienced two waves of kernel OOM kills across their proxy fleet: dozens of kills by the customer’s count, one worker process reaching 61 GB RSS, and thousands of HTTP 502s and tens of thousands of HTTP 499s while the platform was down.

This needed two things to be true at once: the provider on the customer’s traffic kept failing, so the router kept retrying, and Redis was unreachable at the same time, so every completing request logged the growing result of those retries in full. The retries grew unbounded because of a one-line change in PR #38133 (August 24), which ran retry breadcrumbs, the record the router keeps of each failed LLM attempt, through an existing credential-masking helper before storing them. The helper was correct for its original inputs but assumed it was copying a plain tree; the breadcrumb was not a tree, it contained a pointer back to the same list it was about to be appended to. Copying it without preserving that shared reference turned a structure that had cost nothing in memory since December 2023 into one that roughly doubled in size with every retry, until a cost-tracking error handler turned it into a log string on every request for as long as the Redis outage lasted, and a single allocation reached 27 GB.

We fixed the underlying growth in PR #39491, merged September 3 and included in v1.101.0-rc.1. It was not backported to v1.100.0 before that version was tagged stable two days later. We backported it to stable/1.100.x on September 9 (PR #40455) and published v1.100.1.

We own this outcome entirely. The defect existed in a form we could have caught, a ticket filed three days before the stable release correctly identified a growth problem in the same code path, and our own fix PR turned an untested hedge in that ticket into a stated fact. The rest of this post explains how each of those things happened and what we’re changing so that a defect like this can’t reach a stable release the same way again.

Background

1. What a retry breadcrumb is. When an LLM call fails and the router retries or falls back, it writes a note about the failed attempt into the request’s metadata under previous_models, so logs can show what was tried and what succeeded. One breadcrumb is roughly 10 KB.
2. Why the breadcrumb needed masking. Because it copies request settings wholesale, a breadcrumb can carry a forwarded Authorization header or a provider key, and it flows into spend logs and logging callbacks. mask_credentials_in_payload is an existing helper that replaces values under credential-looking keys with asterisks; it was already used elsewhere for guardrail responses. Running new breadcrumbs through it was reviewed and merged as a one-line security fix.
3. A pointer loop that was already there, and had always been free. For every incoming request, the proxy builds one metadata dict and a request snapshot whose body is a shallow copy of that same dict. When a call fails, the router’s breadcrumb stores the snapshot by reference, appends itself to a list shared across the process, and the metadata’s previous_models field points at that same list. The breadcrumb, the snapshot, the metadata, and the list form a loop. This had existed since December 2023 and cost nothing: a pointer is 8 bytes, so the process held four breadcrumbs’ worth of memory, forever, regardless of how many failures occurred.