When a semantic cache helps, and when it serves the wrong answer
A cache for a language model sounds like free money. The same question comes in twice, you answer the second one from memory, and the provider never sees it.
The exact version of that is easy and safe. If two requests are identical byte for byte, the stored answer is as good as a new one.
The interesting version is the semantic cache. It answers from memory when a prompt means the same as an earlier one, even if the words are different. “How do I reset my password?” and “How can I reset my password?” should share an answer, and an exact cache will never see that.
Our gateway has had a semantic cache since its first day. A third of its image is the small embedding model behind it. Before relying on it, I wanted to know when it actually makes sense.
So we did what we did with the model router. Measure first, then decide.
The short version
| Kind of prompt | What the cache does at the default threshold |
|---|---|
| A short question, reworded | Finds the earlier answer 85% of the time |
| The same question with typos or no capitals | Finds it every time |
| A different question with the same wording around it | Never confuses them |
| “Zoom in” against “zoom out”, Java against JavaScript | Sometimes serves the wrong answer |
| The same words in an order that changes the meaning | Serves the wrong answer two times in three |
| A paragraph plus “summarise it in one sentence” against “in five bullet points” | Serves the wrong answer almost every time |
How it decides
A semantic cache turns each prompt into a vector, a list of 384 numbers here. Prompts that mean similar things get vectors that point in similar directions. The cache compares a new prompt with the stored ones and measures how close they are, from 0 to 1.
Above a threshold, it returns the stored answer. Ours is 0.97. It used to be 0.95, until two prompts that differed only in a number started sharing an answer.
The two ways it can be wrong are not equal, the same as with the router. A miss costs a model call, which is what you pay without a cache anyway. A wrong hit gives someone the answer to a question they did not ask, and it looks just as confident as a right one.
How I tested it
I used the same model the gateway ships, bge-small-en-v1.5, quantised, and built the text the same way the gateway does. For every pair of prompts, the question was simple. Would serving the first prompt’s answer to the second one be right?
The pairs came from three places:
- Public datasets labelled by people. MRPC is built from news. PAWS was built to trick models like this one.
- Around 700 pairs built from templates, each one probing a single risk.
- 3,600 pairs of a text with two different instructions.
That is around 75,000 pairs, and none of them comes from company data. For each threshold I counted two things: how many right reuses the cache would catch, and how many wrong answers it would serve.
Where it works
Short questions are the case semantic caching was made for, and it does well there.
| Pairs | 0.90 | 0.95 | 0.97 | 0.98 |
|---|---|---|---|---|
| Same question, reworded | 100% | 99% | 85% | 49% |
| Same question, typos and no capitals | 100% | 100% | 100% | 100% |
| Different question, same wording around it | 0% | 0% | 0% | 0% |
The first two rows are hits you want. The last one is a hit you do not want, and it did not happen once. Two different questions with the same wording around them never scored above 0.77.
At 0.97 the cache catches 85 rewordings in 100. Move the threshold to 0.98 and that falls to half. The default already sits close to the edge.
One word that changes the answer
The trouble starts when two prompts are about the same thing, but one word changes the right answer.
| Only this changes | 0.90 | 0.95 | 0.97 |
|---|---|---|---|
| A number | 46% | 4% | 0.3% |
| A year | 43% | 3% | 0% |
| The operating system | 37% | 0% | 0% |
| The country | 0% | 0% | 0% |
| The programming language | 77% | 7% | 2% |
| The action becomes its opposite | 69% | 14% | 3% |
Every hit in this table is a wrong answer served from the cache. At 0.97 the rate is low, but it is not zero. And the pairs that pass are exactly the ones you would worry about:
| Prompt | Cached prompt it matched | Similarity |
|---|---|---|
| What is 561% of 18? | What is 563% of 18? | 0.978 |
| Explain how to zoom out in Chrome. | Explain how to zoom in in Chrome. | 0.975 |
| Write a Java function that retries an HTTP request three times. | Write a JavaScript function that retries an HTTP request three times. | 0.971 |
The router post already ran into the reason. Embedding models are trained to keep the meaning of a text and ignore its surface. “Zoom in” and “zoom out” are both about zooming in Chrome, so to the model they are almost the same sentence.
Word order
PAWS takes this one step further. Its pairs use the same words in a different order, and more than half of them change the meaning.
“He played with the A-level Kane County Cougars in 1993 and the AA Portland Sea Dogs.” Swap the two teams and the facts are wrong, but the similarity is 0.982. Across 65,000 PAWS pairs at 0.97, the cache would serve the wrong answer for 67% of the pairs that mean different things.
To be fair, PAWS was built on purpose to fool models, and real prompts are kinder. But it shows how little this model sees of the order of the words.
The instruction that disappears
This is the result I did not expect, and the one that matters most.
A lot of what people send to a model is a text plus an instruction. Summarise this, fix the grammar in this, write a reply to this. Two of those prompts with the same text and different instructions need different answers.
The gateway already skips the semantic cache for prompts over 1,500 bytes. The model reads at most 512 tokens, so two long prompts with the same start look identical. In the test they scored exactly 1.000.
I wanted to know whether shorter prompts were safe. So I built texts of different lengths from news sentences, and put ten pairs of different instructions after each one.
| Text before the instruction | Wrong answers at 0.97 | Wrong answers at 0.99 |
|---|---|---|
| None | 0% | 0% |
| 100 bytes | 31% | 2% |
| 250 bytes | 81% | 43% |
| 500 bytes | 98% | 72% |
| 1,000 bytes | 100% | 74% |
| 1,400 bytes | 100% | 89% |
With 500 bytes of text in front, “Summarise the text in one sentence.” and “Summarise the text in five bullet points.” scored around 0.99. Almost every pair of instructions behaved like that: counting the words against counting the sentences, a headline against a reply, shorter against longer.
The reason is the same as before, only bigger. The vector says what a prompt is about, and a paragraph of news about the dollar is about the dollar. The instruction is a few words at the end, and the model treats it as a detail.
Putting the instruction first helps a little, but not enough. With 500 bytes of text, 78% of the pairs still matched at 0.97.
No threshold fixes it either. At 0.98 the cache already misses half the rewordings it was built for. With 500 bytes of text, it still serves the wrong answer 94% of the time.
Longer rewordings
My own rewordings were short and close, and real people rewrite more than that. MRPC has pairs of news sentences that say the same thing in different words. At 0.97 the cache catches only 10% of them.
At 0.90 it catches 60%, and serves a wrong answer for 24% of the pairs that differ. The threshold that is safe catches little, and the threshold that catches a lot is not safe.
So, does it make sense?
For some traffic, yes. A semantic cache fits short, self-contained questions that many people ask in slightly different words. Think of a help desk bot, an FAQ, or a search box with a model behind it.
It does not fit:
- Coding agents. A typical request is around 11 KB, so it never reaches the semantic cache at all.
- A prompt that carries a text to work on. That is where it fails worst.
- Questions about code. Java and JavaScript need different answers, and they sit right on the edge.
What real data would change
This test says when a hit is right. It does not say how often hits happen, so it does not say how much a semantic cache would save. That depends on how often our own users repeat themselves, and it is the next thing to measure.
Real data would also make the cache itself better, in two ways:
- A threshold per kind of prompt. A short question and a prompt with a text behave so differently that one number cannot serve both.
- Training the model on its own mistakes. The failures above are pairs that differ in one word or one number. Fine-tuning on real pairs like those, marked as different, is the obvious next step.
Not a free win
A semantic cache is not free money. It is a classifier that decides whether two prompts deserve the same answer. Like any classifier, it has to be tested on the cases that matter.
On short questions it earns its place. Once a prompt carries a paragraph, it cannot tell one instruction from another.