How our LLM gateway decides when a cheaper model is good enough

Most callers of a language model make the same choice every time. They name the strongest model they are allowed to use, because nothing tells them otherwise. So a one-line translation costs the same as a long analysis of a codebase. Multiply that by every team and every agent, and it adds up fast.

The gateway sees both prompts, knows the prices and records the spend. If anything is in the right place to notice that mismatch, it is the gateway.

In the first post about the gateway I said, in one line, that it can pick the model for you in about five microseconds. This is the story behind that line. It took three attempts and one bad dataset. The idea that worked best was the one that looked too simple to try.

The short version

	What the router ended up as
Time per decision	4.85 µs
Size of the model	8 KB of weights
Neural networks involved	None
Cost to gateway throughput	Smaller than the noise
Projected saving	42.7% of the spend
Worst case	Exactly the same as no router

The rest of this post is how it got there, including the part where it was confidently wrong.

The idea

A caller can ask for a model called auto instead of a real one. The gateway reads the request, decides whether the task is mechanical, and sends it to a cheap model or to the caller’s usual one. A caller that names a real model always gets that model.

Routers like this already exist, and they come with impressive numbers. I did not want to trust them. LLMRouterBench, a large independent benchmark, found that several routers, commercial ones included, fail to reliably beat a simple baseline.

So I measured everything before writing a line of the real thing. I took two public datasets of prompts written and labelled by people, one in English and one in English and Spanish. Every approach had to answer one question: is this a cheap task or not?

Attempt one: describe the routes

The most popular approach is to write a short description for each route, like “translation and formatting” or “complex reasoning”. You compare each prompt with the descriptions and pick the closest. It needs no training data, which is why people like it.

It picked the right category 40.4% of the time. Always answering the most common category gets 42.8%. A router that ignores the prompt did better than one that reads it.

That ended it in an afternoon, which is the best way for an idea to end.

Attempt two: learn from examples

Instead of descriptions, I gave the router labelled examples. A small embedding model turned each prompt into a vector, and a simple classifier learned where the line between cheap and expensive was. This time it got the tier right 91 to 93% of the time.

A few things came out of this round that shaped everything after it.

Two tiers, not ten. Picking the exact category was unreliable, between 47% and 75% depending on the dataset. Cheap or not cheap held between 88% and 94% on both.
Only the last message. Real requests are conversations, with a system prompt and a history. Classifying all of it dropped accuracy from 91.6% to 70.5%, so the router reads only the last thing the user wrote.
Spanish needs Spanish examples. Trained and tested in Spanish, it scored 93.6%. Trained in English and tested in Spanish, 84.5%.
About 200 examples per tier. Ten already beat the baseline. The curve goes flat around 200.

The number that killed it

Accuracy was fine. Speed was not.

An embedding model costs milliseconds, not microseconds. One copy served 77 prompts per second. Eight copies on eleven cores, with the input cut to 384 characters, served 904.

That is a ceiling of around 150 requests per second per core. The gateway does 1,940 on a single CPU. Putting this router in front of it would have made it thirteen times slower, to save money on some of the requests.

Attempt three: eight numbers

So I tried something much dumber. Instead of a neural model, I counted things. Length, word count, newlines, the share of digits and the share of capitals. Whether there is a question mark, a code block, or a colon at the end.

Those eight numbers alone scored 90.0%. The embedding model scored 92.0%, and cost about ten thousand times more to compute. Two points of accuracy for four orders of magnitude is not a good deal.

Then I added hashed fragments of words and characters. Each prompt becomes a fixed vector of 1,032 slots, filled by hashing short pieces of text into buckets. No model, no vocabulary, nothing to download, and it scored 95.5%. It beat the embedding model on accuracy and on speed at the same time.

The speed difference is hard to picture. The embedding model managed 77 prompts per second on one thread. The new classifier does about 181,000.

It makes sense once you see it. Embeddings are trained to ignore the surface of a text and keep the meaning. But whether a task is mechanical shows mostly on the surface. “Translate this:” followed by a paragraph looks nothing like “Why does this test only fail on Mondays?”.

I also tried the obvious combination. The cheap classifier decides when it is sure, and passes the unsure cases to the embedding model. On those cases the embedding model was right 65.8% of the time, against 68.4% for the cheap one, so I dropped it.

The final classifier is 8 KB of weights. Training happens offline, never inside the gateway.

Only moving down

The two ways a router can be wrong are not equal. Send an easy task to an expensive model and you lose a bit of money, and nobody notices. Send a hard task to a cheap model and the answer gets worse, and the user notices straight away.

So the router only moves down when it is confident, above a threshold of 0.70. Everything else goes to the caller’s usual model: an unsure classifier, missing weights, a request it cannot parse. A failure in the router is logged and never fails the request.

That gives it a property I like a lot. In the worst case, it behaves exactly like having no router at all.

Two more rules sit in front of the classifier. A request that declares tools is never downgraded, because it is agent traffic and its last message says little about the work. And the model that gets recorded, billed and cached is the one that answered, never auto. Otherwise a cheap answer could be served from cache to an expensive request.

The dataset that lied

At this point the numbers looked great. 95% accuracy, a third of the spend saved, and about 1% of requests sent to the cheap model by mistake. I shipped it.

Then I tested it on a set where easy and hard tasks came in every length. It scored 50.0%, a coin flip, and sent a quarter of the hard requests to the cheap model.

The cause was in the training data. In the English and Spanish dataset I had trained it on, a cheap prompt was 639 characters long on average, and an expensive one was 63. Summaries and extractions come with a passage to work on, and open questions are one line.

A rule that says nothing but “longer than 400 characters means cheap” already scored 83.5% on it. So the router had learned to measure length. Long prompts went to the cheap model and short ones to the expensive one, whatever they asked for.

The labels had a second problem. They came from the dataset’s own categories, which mark many one-line factual questions as hard. 458 of its 800 rows were short factual questions labelled expensive.

“What is the Trans Tahoe Relay?” was expensive. “Who is Rajinikanth?” was cheap.

I dropped the dataset, replaced the weights and withdrew every number that came from it. The commit message says it plainly: the old weights classified length rather than difficulty.

Training on data I can trust

The new training data is generated from templates kept in the repository, so anyone can rebuild the weights without downloading anything. The script checks two things every time it runs.

Both tiers cover the same range of lengths. A length-only rule scores exactly 50% on the set, so the classifier has to look at the task.
Training and test sets share no template. I learned this one the hard way too. Testing on the same wording it was trained on reported 99.9%, and meant nothing.

On 100 prompts worded differently from anything in training, the new router scores 85.0%. At the default threshold it projects 42.7% of the spend saved, with 6.0% of requests getting a weaker model than they needed. That 85% is a floor for a classifier that has never seen our traffic, not a promise.

What it costs

I ran the throughput benchmark from the first post again, on the current build, with routing on and off. One CPU, the same 10.8 KB Claude Code request. Routing off did 2,474 requests per second and routing on did 2,430, while two runs of the same build differed by about 5%.

The router disappears in the noise. Measured directly, one decision costs 4.85 microseconds, about 1.2% of the time the gateway spends on a request. Only requests that ask for auto pay it.

Where it is now

It is in use, and it works. Teams that want it ask for auto, and everybody else is untouched.

How much it saves depends more on the traffic than on the classifier. The same classifier saved 27.7% on one dataset and 3.0% on the other, because one had far more mechanical work in it. The next step is to retrain on our own prompts, because surface features do not carry over to words they have never seen.

What I took from it

The router that runs today has no neural network, fits in 8 KB and decides in under five microseconds. It beat an embedding model on its own benchmark, and it can never make things worse than not having it.

Every approach here was measured before it was kept, and that is what killed the first two quickly. But the measurement that mattered most was not about the router. It was the one that checked whether the dataset asked the right question.

If a classifier looks too good, look at the data before you look at the model. Mine was 95% accurate at measuring length.