An LLM gateway in Rust, 36 times faster than LiteLLM
Every request a coding agent makes at work goes through a gateway before it reaches a model provider. The gateway checks the key, masks personal data and secrets, counts the tokens, records what the request cost and forwards it. If you read the post about the PII detector, you have met it already. That gateway is LiteLLM, the open source Python proxy, with our configuration on top.
At the end of June I started writing a replacement in Rust. Three months and a bit over five hundred commits later, it does everything the old one does on the path Claude Code uses, and quite a lot more. This is the first post in a series about it. Before any of the details, this is what happens when you put the two side by side.
The numbers
| Rust gateway | LiteLLM | ||
|---|---|---|---|
| Image size | 109.5 MB | 1,329.6 MB | 12x smaller |
| First request after a restart | 1.1 s | 17.3 s | 16x faster |
| Memory at idle | 15.7 MiB | 762.7 MiB | 49x less |
| Memory at peak, under load | 84 MiB | 1,001 MiB | 12x less |
| Throughput on 4 CPUs | 4,957 req/s | 139 req/s | 36x more |
| First streamed token, 32 streams on 2 CPUs | 7.5 ms | 236 ms | 31x faster |
These are not the best cases picked out of a long run. The benchmark has 32 load cases, and every one of them points the same way. The smallest gap in the whole report is the full stack on disk, databases included, at 2.6 times. The biggest is 50 times, on the first streamed token with eight streams on four CPUs.
A gap of 36 times usually means one side is not doing the work. So before anything else, this is how I made sure both were.
Making it fair
A benchmark like this is only worth something if nobody can say it was rigged. The point was to measure the gateway and nothing else, so everything around it had to be identical.
- Same upstream. A small local server answers the Anthropic Messages API instantly. On its own it does 27,700 requests per second at 0.14 ms, so it is never the bottleneck.
- Same route. Both gateways serve
POST /v1/messages, the route Claude Code actually calls. - Same masking. I sent a request with an email, an internal IP, an AWS key id and two internal URLs through both. They came out the other side masked in exactly the same way. Neither gateway ran with its guardrail switched off.
- Same bookkeeping. Both exported Prometheus metrics and wrote one accounting row per request into Postgres. I checked that both were really doing it during the run.
- No caches. Both had their caches off. A counter on the upstream confirmed that every request reached it.
- Same CPU architecture. The LiteLLM image we deploy is amd64. Under emulation on my M3 it would cost about three times more CPU, which would make the Rust side look better than it is. So I rebuilt it native for arm64.
The request body is 10.8 KB and shaped like a real Claude Code request, because that is the traffic the gateway sees. The load came from oha, in a closed loop, 20 seconds per run after a 6 second warmup. Docker held each gateway to 1, 2 or 4 CPUs.
The flat line
The most interesting result was not a number. It was a line that did not move.
| CPUs | Rust gateway | LiteLLM, 1 worker | LiteLLM, 1 worker per CPU |
|---|---|---|---|
| 1 | 1,940 req/s | 133 req/s | 133 req/s |
| 2 | 3,430 req/s | 129 req/s | 213 req/s |
| 4 | 4,957 req/s | 139 req/s | 319 req/s |
Four times the CPU gave LiteLLM 5% more requests. The reason is simple once you see it. It runs as one uvicorn worker, and one Python worker uses one core. That is how it is deployed. The other cores sit there doing nothing.
The obvious fix is one worker per core, and it helps, up to 319 requests per second on four. But each worker is a full copy of the process with its own memory. Two workers peaked at 2,592 MiB and four at 4,474 MiB. The pod it runs in has a 2 GiB limit, so not even the two-worker version fits. What caps the throughput today is memory, not CPU.
The Rust gateway goes from 1,940 to 3,430 to 4,957 in a single process, and never goes above 84 MiB. On four CPUs it was not even at the limit. It used around 310% of the 400% it was allowed, because the load generator and the upstream were on the same laptop. The real number is a bit higher.
What a developer actually feels
Nobody is going to send five thousand requests per second through this. A real model provider takes hundreds of milliseconds to answer, often seconds. So the throughput is headroom. What it really tells you is how much CPU each request costs, and that is what shows up as latency.
Claude Code always streams, so the number a developer feels is the time until the first token appears. Here it is on two CPUs:
| Streams at once | Rust gateway | LiteLLM |
|---|---|---|
| 1 | 0.43 ms | 6.41 ms |
| 8 | 1.66 ms | 52.55 ms |
| 32 | 7.54 ms | 236.19 ms |
Both gateways stream properly. They send the headers early and pass each event through as it arrives, so neither is holding the stream back. The gap is only the cost of each request. A quarter of a second of extra wait on every turn, before the model has even started, is something you notice when you use an agent all day. Seven milliseconds is not.
Buffered requests look the same. At 32 concurrent requests on two CPUs the median is 8 ms against 216 ms, and the p99 is 33 ms against 838 ms. At 64 concurrent requests on four CPUs, LiteLLM’s p99 is over a second and a half. The Rust gateway’s is 49 ms.
Then there is startup. I measured the time from starting the container to the first /v1/messages that returns 200. Not a health check, because a proxy can report healthy before it can route anything. After a restart with the databases already up, it is 1.1 seconds against 17.3. From nothing, with every volume wiped, it is 6.9 against 30.7. Every rolling deploy, every node drain and every crash loop pays that cost again.
Where the size goes
The difference in image size is almost funny when you open the layers.
| Rust gateway, 109.5 MB | LiteLLM, 1,329.6 MB | ||
|---|---|---|---|
| Compiled binary | 36.6 MB | Python virtualenv | 698 MB |
| Bundled embedding model | 34.7 MB | Prisma engines | 245 MB |
| Distroless base, glibc and libssl | 38.2 MB | Node.js, Python and OpenSSL | 197 MB |
| onnxruntime | 70 MB | ||
| Layer rewrites and extras | 120 MB |
A third of the Rust image is the binary, a third is a small embedding model it carries for the semantic cache, and a third is the base image. There is no shell, no package manager and no interpreter in it. The LiteLLM image carries a full Python environment, the engines of its ORM and a whole Node.js runtime. Nothing in it is wrong. It is just big.
The whole stack tells the same story. The Rust side runs a newer Postgres and one extra service, Qdrant, for the semantic cache. It is still 637 MB against 1,641 MB. Without Qdrant it is 440 MB.
So what is it
An LLM gateway is not an exciting thing to describe. It sits between applications and model providers, and it owns everything each application should not have to do on its own. This one does quite a lot of that.
- One API in front of several providers. Most callers use the OpenAI chat format. Tools that speak a provider’s own API, like Claude Code with the Anthropic Messages API, get that API proxied as it is. When several providers serve the same model, it fails over between them, and it can order them by cost, by latency or in turn.
- Keys, budgets and rate limits. Every team gets its own keys, and each key can carry its own budget and its own limit. The cost of every request is recorded, down to the prompt cache.
- Guardrails in the process. It masks personal data and secrets, or swaps them for placeholders and puts the real values back in the response. It also has deny rules and prompt-injection detection, on the way out and on the way back. There is no sidecar and no network hop.
- Two caches. An exact match cache in Redis and a semantic cache in Qdrant, with the embedding model running locally. Identical requests that arrive at the same time are collapsed into one upstream call.
- An automatic model choice. Callers can ask for a virtual model instead of a real one. A small local classifier sends mechanical work to a cheap model and the rest to the default one, in about five microseconds.
- One door for tools and agents. MCP servers and A2A agents sit behind the same gateway, with the same keys, policies and guardrails as the model traffic.
- Metrics, traces, a Grafana dashboard and a web console to manage keys, limits and spend.
It is one binary. It is built on Tokio and axum, with sqlx for Postgres. It has around fifteen hundred tests, and a warning fails the build.
What comes next
None of the numbers above come from a clever trick. They come from a lot of small decisions, and most of those decisions are more interesting than the benchmark.
This post was the overview. I am going to keep writing about the gateway as the work goes on, the parts that went well and the parts that did not, the same way I did with Semantic Graph.