Everything we didn't see coming while building our LLM gateway
When you write a gateway, you make a list of the things that can go wrong. The provider goes down, a key leaks, someone spends too much. Those are the easy ones, because you see them coming.
This post is about the other list. These are the bugs nobody wrote down, in our LLM gateway.
Some came out of adversarial reviews in July, where the only job was to break it. The rest came out in the first week of the pilot, when Claude Code started sending it real traffic.
None of them was hard to fix. All of them were hard to imagine.
The bill that came out at 0.4%
The whole reason the gateway exists is to know what each request costs. So it was a bad surprise to find that, for most Claude Code traffic, we were billing almost nothing.
The pricing code follows the OpenAI way of counting. prompt_tokens is the whole input, and the cached part is a subset of it.
Anthropic does it differently. Its input_tokens leaves out the tokens read from the prompt cache and the tokens written to it.
A Claude Code turn is almost all cache. The conversation so far is cached, and only the last message is new. So the base input_tokens is tiny, and billing on it alone came to about 0.4% of the real cost.
Cache writes were never billed at all. Gemini had the same problem the other way round. It reports the reasoning tokens of its thinking models in a separate field, so every thinking answer was under-counted.
The fix was to stop letting each provider’s shape reach the pricing code. Each adapter now turns the provider’s usage into one shape first. Then the pricing splits the input into three parts with their own price: cache reads, cache writes and the rest.
The lesson for me was simple. A cost number that looks a bit low does not look wrong. Nobody complains when the bill is small.
Paying twice for one answer
A gateway has three ways to send a request again: retries, failover to another provider and the circuit breaker. That is great for availability. It is dangerous for billing.
A completion is not idempotent. If the provider already got the request, sending it again means generating it again, and paying for it again. There is no request id the provider uses to drop duplicates.
The bug was in one small case. A provider answered 200, but the body was not valid JSON.
The gateway turned that into a 502 that could be retried, so it sent the request again. The first answer had almost certainly been generated and billed already.
Two more cases hid the same problem. A timeout while a stream was starting was retried, although the model had probably started. An error in the middle of a stream was passed on as a clean end, so the client thought it had the full answer.
The fix was to put the one question that matters into the error type: did the request reach the provider? If it did, nothing retries it, fails it over or replays it. The client gets the error and decides.
The usage records had the same problem one level down. A batch could be written, lose its confirmation and be written again. Now each row carries an id, and a second copy is ignored.
A price that was not a number
Prices live in the configuration. One of the reviews tried a price that was not a number, NaN.
The gateway accepted it. Every request on that model was booked at $0, so it never counted against a hard budget. A negative price was even better: it took money off the recorded spend.
Nobody would type NaN on purpose, of course. But a broken template or a bad environment variable can do it for you.
The gateway now checks every price when it loads the configuration, and refuses to start if one is wrong. A bad config should stop the boot, not show up weeks later in a report.
Masking that broke the prompt cache
The gateway can swap personal data for numbered placeholders, like <EMAIL_ADDRESS_1>. It puts the real values back in the response. The numbers have to stay the same across the whole conversation, or the prompt changes.
They did not. The gateway visited all the message text first, and the tool-call arguments after it. So when new personal data showed up in a later turn, the placeholders in earlier tool calls got new numbers.
That changed the start of the prompt. The provider’s prompt cache works on the start of the prompt, so every turn after that was a cache miss. A cache read costs a tenth of normal input, so this bug made long sessions much more expensive.
Nothing failed and no answer was wrong. The only sign was the bill. The fix was to visit each message whole, text and tool calls, before moving to the next.
One full replica that blocked every team
The gateway keeps a limit on how much request and response body it holds in memory at once. When there is no room, it should refuse the request. And it did, but with the wrong error.
A buffered response with no room was reported as a provider failure after sending. Provider failures feed the circuit breaker, and the breakers are shared. So memory pressure on one replica could open the breaker for a provider, for every team.
In other words, a local problem looked like the provider was down. Everyone was cut off from a provider that was working fine.
Now it is its own error. It is never retried, the breaker is left alone, and the client gets a 503. It also has its own metric, so a full replica is visible as what it is.
The model that refused <BANNED_WORD_7>
This was my favourite, and it came in the first days of the pilot.
The gateway masks internal host names and some banned terms before a prompt leaves the building. A masked value became something like <BANNED_WORD_7>, and the gateway put the real value back in the answer. On paper, the model never needed to know.
But the model did see it, and it did not like it. A placeholder like that looks like something was blocked. So the model refused to pass it to a tool, even though the gateway would have restored it on the way back.
The fix was to make the placeholders look like the values they replace. A host becomes host-3.invalid, an email user-2@masked.invalid and an IP 192.0.2.7.
These come from ranges reserved for documentation, so they can never be real values. The model treats them as normal data, and passes them on.
The gateway only restores tokens it issued itself. It also never masks a value that is already in those ranges. So a token the model writes back always survives the trip.
The 64 KB line
Three five-minute Opus answers died near the end, with an api_error. The logs said the gateway had cut them.
The gateway reads a streamed answer line by line, so it can scan the output. It refuses any line longer than its limit. That limit was 64 KB, which looked huge for one line of server-sent events.
It is not huge for Anthropic. When a long thinking block ends, the stream sends its signature on a single line, and that line can pass 64 KB. So the longest, most expensive answers were the ones that got cut, at the very end.
Then it got worse. Claude Code retried each one without streaming.
A buffered call says nothing until the whole answer is ready, and our read timeout was 60 seconds. So the retry was cut too, this time with a 502.
Two fixes, both one line. A stream line can now be up to 1 MB.
The read timeout went from 60 to 300 seconds, to match the deadline that was already there for the whole request. Streams do not feel this timeout, because every token resets it.
Tool calls had the same kind of limit. A coding agent sometimes writes a whole file in one call.
The gateway kept up to 256 KB of tool-call arguments to scan them, and a big file could pass that and cut the stream. That limit is now 1 MB too.
A host masked one way
Host masking had one more surprise. Agents often get internal host names in their system prompt. The gateway does not scan system prompts, so those hosts reached the model as they were.
Then the model used one in a tool call. The gateway scans the answer, so it masked the host there.
But that token was never issued on the way in, so there was nothing to restore. The tool got a fake host, and the call failed.
The masking worked as it was designed. The two directions just did not agree on what was secret. For the pilot, we turned off the scan on model output, and a few days later stopped masking URLs.
The 404s that were not errors
The dashboard showed hundreds of client errors a day. None of them was a real problem.
Claude Code checks that its base URL works with HEAD /api/hello. The gateway did not know that route, so it answered 404, around 400 times a day.
It also calls /v1/messages/count_tokens to know how full its context is, about once for every four messages. That route did not exist either.
Neither one broke anything for the user. But they buried the real errors on the dashboard. Both routes now exist, and the error rate shows only errors again.
What I took from it
Looking back, I see two kinds of bugs here.
The first kind is about money. The 0.4% bill, the double payment, the NaN price and the broken prompt cache all failed in silence.
No request failed and no user complained. The only sign was a number that was a bit off, and nobody checks a number that is lower than expected.
The second kind is about real traffic. The placeholder, the 64 KB line, the one-way host and the 404s all passed every test we had.
Our tests sent the requests we thought an agent would send. A real agent thinks for five minutes, writes whole files in one go and reads a placeholder like a person would.
So if you build something like this, test the cost as hard as you test the code. Then put real agents on it as early as you can.