FAST VERDICT

Two AI gateways merged semantic-cache fixes on the same day, for two different causes

litellm's semantic cache dropped tool_result text from the embedded prompt, so turn 2 matched turn 1 and replayed its answer. OmniRoute's cache signature left out reasoning effort and max_tokens, so different requests got identical hits. Both fixed Oct 7.

2026-10-16

Two AI gateways merged semantic-cache fixes on October 7, for two different causes. Both bugs made the cache return the wrong answer with full confidence. Neither raised an error.

litellm: the tool result that never got embedded

The report, from BerriAI/litellm PR 43878 (merged Oct 7): on /v1/messages, the tool_result text “was dropped from the embedded prompt, so turn 2 embedded byte-identical to turn 1”, and “the cache replayed the turn 1 answer on every later turn.”

Read that twice. The cache key was computed from the prompt text, but the tool result, the thing that changed between turns, was never part of what got embedded. Every later turn matched the first turn’s cache entry and got the first turn’s answer back.

The merged fix adds cache_params.max_messages (default 4): a request with more messages “is neither looked up nor stored”, and tool_result text is now embedded. The PR’s own caveat: “A 3 or 4 message tool turn is still under the cap, so it is still looked up.”

OmniRoute: the effort setting the signature forgot

The report, from OmniRoute issue 15149 (closed Oct 7 with the merge of PR 15156): the cache signature left out reasoning.effort and max_tokens. At temperature 0, a request at effort none followed by one at effort max got a cache HIT in under 1 second with a byte-identical body. A maintainer confirmed the missing fields on Sep 30. Per PR 15156, both requests computed the exact same cache signature.

Merged is not released: as of Oct 15, OmniRoute’s latest release is v3.8.51 and the fix sits on release/v3.8.52. Per the maintainer, the per-key bypass (cache_default_mode: bypass) and the per-request X-OmniRoute-No-Cache: true header stay valid until it ships.

Our reading, untested

Change one input at a time, the tool result, the effort, max_tokens, and check that the answer changes. A cache hit is not proof the answer is right. We did not run either project; the findings are the projects’ own. No claim about other caches or how many users were hit. Dates UTC.

The pattern across both: the cache key did not cover everything that determined the answer. That is the whole bug, twice, in two codebases, on the same day.

This post started as an Instagram post →

Discussion

Talk it through

Argue with us on Instagram.