Incident Report: Prisma DB Reconnect Blocks the Event Loop and Kills Liveliness

LiteLLM incident report (dated by publication; the incidents occurred across customer deployments in April 2026). Published 2026-04-29.

Date: April 2026 Duration: Multiple incidents across customer deployments before fix landed Severity: High; surfaced as full proxy outages in Kubernetes Status: Resolved

Note: This fix is available starting from the release that contains PR #26225 (merged April 29, 2026).

Summary

When the upstream Postgres database became unreachable, the LiteLLM proxy’s Prisma reconnect path called await self.db.disconnect(). Under prisma-client-py that call invokes a synchronous subprocess.Popen.wait() on the Rust query-engine subprocess. Because wait() does not yield, the asyncio event loop froze for as long as the engine took to shut down, typically 30–120 seconds in production when the engine was stuck on TCP close operations against the unresponsive database.

While the loop was frozen, no coroutines ran, including /health/liveliness. Kubernetes liveness probes timed out and the kubelet SIGKILLed the pod. From the operator’s point of view the proxy looked dead even though the underlying issue was a transient DB outage that the reconnect logic was supposed to ride through.

Impact: Any customer whose Postgres briefly became unresponsive saw proxy pods get killed and restarted instead of degrading gracefully and reconnecting once the DB came back. Reported externally by FLock and reproduced internally.

Background

LiteLLM’s proxy keeps a single long-lived Prisma client to talk to its Postgres metadata store (keys, teams, spend logs). When that connection drops it has to reconnect, otherwise every authenticated request fails. The reconnect path lives in litellm/proxy/db/prisma_client.py’s recreate_prisma_client() (and a now-removed “direct reconnect” branch in litellm/proxy/utils.py).

The intended flow was:

1. Health watchdog sees the DB queries failing.
2. Call await self.db.disconnect() to release the old engine process cleanly.
3. Construct a fresh Prisma() client.
4. await new_client.connect().
5. Swap the proxy’s prisma_client.db reference to the new client; resume serving.

The /health/liveliness route is intentionally cheap and does not touch the database. The expectation was that even during a DB outage, liveliness would stay green and Kubernetes would leave the pod alone.

Root cause

prisma-client-py’s engine cleanup is internally synchronous. The library’s Engine.aclose() looks async from Python’s perspective, but the implementation that finally shuts down the Rust query-engine subprocess calls:

self.process.send_signal(signal.SIGTERM)
self.process.wait()   # <-- BLOCKING. Does not yield to the loop.

When the database is healthy the engine exits within milliseconds and the blocking call is invisible. When the database is unhealthy, the engine’s own outbound TCP close() calls hang waiting for FIN/ACK from the unresponsive Postgres host, and wait() blocks the whole event loop for the duration.

The reconnect path was wrapped in asyncio.wait_for() as a “safety timeout”, but wait_for can only cancel at await points. There is no await inside subprocess.wait(), so the timeout could not fire. The loop simply did not run any coroutines, including the cancellation coroutine, until wait() returned on its own.

As a result every Prisma reconnect during a DB outage froze the entire proxy, and Kubernetes consistently mistook the freeze for a liveness failure.

The Fix

PR #26225 replaces disconnect() in both reconnect paths with a direct, non-blocking kill of the engine subprocess. The new flow is:

1. Look up the engine PID via _get_engine_pid() (hardened to only return real integers, so unit-test mocks do not crash callers).
2. Send SIGTERM to the subprocess directly.
3. await asyncio.sleep(0.5) — this is a real await, so the loop keeps running and /health/liveliness continues to respond.
4. If the process is still alive, send SIGKILL.
5. Construct a fresh Prisma() client and await new_client.connect().
6. Swap the proxy’s reference to the new client.

Both reconnect call sites, recreate_prisma_client and the formerly-separate “direct reconnect” branch in litellm/proxy/utils.py, now go through recreate_prisma_client. The two engine-alive and engine-dead paths converge on the same kill-then-recreate flow, which removes a class of “what if the engine died between checks” bugs.

The relevant change (simplified):

- # Old: blocks event loop for as long as the engine takes to shut down
- await self.db.disconnect()
+ # New: signal the engine subprocess directly, yield via real await,
+ # then SIGKILL if it has not exited.
+ pid = self._get_engine_pid()
+ if pid is not None:
+     try:
+         os.kill(pid, signal.SIGTERM)
+     except ProcessLookupError:
+         pass
+ await asyncio.sleep(0.5)
+ if pid is not None:
+     try:
+         os.kill(pid, signal.SIGKILL)
+     except ProcessLookupError:
+         pass

The new Prisma() client and its connect() are kept as before; the only thing that changed is how the old engine is torn down.

Verification

Reproduced end-to-end against a local proxy + Postgres in Docker, using docker pause on the Postgres container to simulate an unresponsive database: