The PII detector that fits in half a megabyte
Every request from a coding agent at work goes through our AI gateway before it reaches a model provider. The gateway masks personal data and secrets on the way out. Most of that is regex work and it is boring in a good way: emails, IPs, card numbers with a Luhn check, IBANs with a mod-97 check, a list of internal URL patterns. Three things a regex cannot find: a person’s name, a place tied to that person, and a credential value that does not follow a known prefix. For those we ran Presidio, Microsoft’s open source PII toolkit, as two sidecar containers next to the gateway.
We had to take Presidio out. This post is about what we put in its place: a model I trained on my laptop in a week. It is 533 KB, two thousand times smaller than what it replaced. It adds under 10 MB of memory to a process that already exists, and it is about ten times faster on a real request. It took about fifty versions to get right.
Why Presidio had to go
Presidio works, and I want to be fair to it. The problem was what it cost us to run. Every masked request made two HTTP round trips, one to the analyzer and one to the anonymizer, and each hop could time out. The two images were 1.2 GB together. They held 785 MiB of resident memory before serving a single request, and under load they took gigabytes from a cluster that had other things to do.
Worse, they were slow in a way people could feel. A typical Claude Code request is around 11 KB. Presidio handled that in 140 ms at one request at a time. With eight concurrent requests the throughput did not move, 6.4 to 6.5 requests per second, and the median latency went from 140 ms to 1,120 ms. It was using 71% of one core the whole time. The extra load was not processed. It was queued. A handful of people using agents at the same time was enough to make everyone wait.
Body size was worse. A 320 KB body took 22 seconds. A 1 MB body came back as HTTP 500. The analyzer log gave the reason: spaCy refuses inputs above a million characters, because the parser needs roughly a gigabyte of temporary memory per 100,000 characters. Our gateway accepts bodies up to 8 MB.
So we removed it. For a while the gateway masked only what the regexes could find, and names and places went through. That was not a state I wanted to stay in, but it was better than a guardrail that hurt the cluster and the people using it.
The idea
The gap Presidio left was small and specific: names, places tied to a person, and credential values with no known prefix. Everything else was already covered. I did not need a general NER pipeline. I needed a very small classifier that knew those three things and nothing else, ran in the same process as the regexes, and cost nothing when idle.
There was one more reason to build it rather than download it. Presidio’s names came from a spaCy pipeline we did not train, could not tune and could not fully explain. When someone asks where the weights came from and what data they saw, “we downloaded them” is a weak answer. We wanted to be able to answer that question fully, byte by byte.
The shape of the model
The replacement is a dilated convolutional network that reads raw bytes and emits one label per byte: O, PERSON, SECRET or LOCATION. No tokenizer, no vocabulary, no pretrained weights. Bytes go in, labels come out, for any length of input.
The trunk runs at a quarter of byte resolution. A strided stem downsamples by four, six dilated blocks run on that, and a small head at byte resolution restores exact span boundaries. The receptive field is 1,012 bytes, so a name is judged by roughly a kilobyte of context around it. The byte-resolution version of the same network was 2.5 times slower for the same accuracy, so stride four stayed.
Hidden width 96, 410k parameters, exported to ONNX and quantised to static int8. The file is 533 KB. Every int8 build of this architecture lands within four bytes of that.
A decode step turns the per-byte probabilities into spans. It snaps regions to token boundaries, picks the label by probability mass, and applies a threshold per label: 0.6 for PERSON, 0.8 for SECRET and LOCATION. The thresholds were chosen against false positives on public code, not against a headline F1.
Training without any real data
The rule I set on day one: no company repositories and no real traffic. Everything the model has seen is synthetic or public.
The generator writes code, chat, logs and config in English and Spanish, and plants synthetic people, places and credentials in them. The names come from public downloads: a random-name list, a world cities list, surnames per country. The background text is windows of public open source code, the Python standard library, sympy, torch. A separate set of public repositories, numpy, networkx, onnxruntime, setuptools, jinja2 and onnx, is never used for training. It exists only to count false positives, 14.1 MB of it.
The generator also plants hard negatives on purpose. Cloud regions, timezones and offices are places but they are labelled O, because masking eu-west-1 breaks a developer’s request. A bare Canada with nobody attached is O too. Only a place tied to a person or a customer is a LOCATION.
The whole thing took a week. Each full iteration, generate the data, train, export, quantise and score the eval sets, took between two and four hours depending on how much data I gave it. Most days that meant two or three complete versions, and a lot of evenings reading false positives. The deployed run was 400k synthetic documents, 12 epochs, about two hours forty on an M3 Pro.
The evaluation sets nobody generated
The synthetic validation set turned out to be useless for choosing a model. Two versions scored within 0.002 of each other on it while differing by six points of recall on real conversation. So I wrote evaluation sets by hand, independent of the generator, and scored each one separately because they fail differently.
| Set | What it is | Blocks |
|---|---|---|
| heldout | code, config and log shaped text | ~370 |
| chat | short conversational prose, which is what an agent actually receives | 117 |
| loc | a place tied to a person versus a place standing alone | 99 |
| address | a person addressed, then the sentence continues | 54 |
Scoring is per block, not on one concatenated body. A block is one message, which is what the gateway masks. Whole-body scores run two to four points higher on every set and hide the loss of cross-block context.
Here is the version in production, scored at the gateway’s thresholds.
| Set | Label | Precision | Recall |
|---|---|---|---|
| heldout | PERSON | 0.97 | 0.97 |
| heldout | SECRET | 0.97 | 0.95 |
| heldout | LOCATION | 0.93 | 0.95 |
| chat | PERSON | 1.00 | 0.75 |
| chat | LOCATION | 1.00 | 0.71 |
| loc | PERSON | 1.00 | 0.95 |
| loc | LOCATION | 0.98 | 0.85 |
| address | PERSON | 0.90 | 0.60 |
On code, config and logs, which is most of what the gateway sees, it finds 97 names in a hundred and 97 of what it finds are names. Precision on conversation is 1.00: in the chat set it never masked something that was not a person.
Then the other half of the rule: false positives on the 14.1 MB of held-out public code, at deployed thresholds. I read every hit. 41 PERSON spans were real author names in copyright headers and moduleauthor lines, and masking them is correct. The rest were nine wrong spans in 14 MB of code, all of them inside docstrings and argparse descriptions: Neural Network and Fortran and SmoothQuant and the like. Not one was a variable, a function or a line of code a developer would need back.
The bug that was a preposition
The most instructive failure came from probing v11 by hand. The same name moved from 0.81 to 0.03 on one word:
| Line | PERSON probability |
|---|---|
ask Petrov about the trace id |
0.81 |
ask Petrov for the trace id |
0.03 |
Petrov signed it off |
0.97 |
tell hannah the invoice bounced |
0.03 |
The model could not read the position of a name, only the wording around it. The reason was in the generator. It only ever put a name at the end of a request, followed by one of nine punctuation endings. “Verb, name, then the sentence continues” had never been seen. I added that frame, and paired each line with the same verb and continuation holding a role, a library or a product, so the frame itself carries no label.
While probing I also found that Eduardo was in none of the name lists, including the 4,913-entry public one. Unaccented spellings like Sanchez and Garcia were missing entirely. And a real surname that no list contains has to work anyway. So one name in five is now invented from syllables, and the invented ones are fed to the negative side too, so “unfamiliar capitalised token” cannot become the cue.
That last point bit me once more. v15 read can you repeat this: Eduardo? as O at 0.028, because the generator drew 45% of its places from a long tail of 34k villages. The model had learnt the opposite rule: an unfamiliar capitalised token in a neutral frame is a place. v16 draws only places the decode step can veto by name, and the case passes.
Lists remove, they never propose
Decode used to carry an 85k-entry name lexicon that added PERSON spans the model had scored as O. It felt like a safety net. I measured it:
| with lexicon | without | |
|---|---|---|
| heldout PERSON | P .86 / R .91 | P .91 / R .90 |
| loc PERSON | P .90 / R .90 | P .95 / R .90 |
| address PERSON | P .68 / R .63 | P .71 / R .50 |
| code corpus PERSON hits | 60 | 42 |
The lexicon bought four names on one set and two on another, at the price of five points of precision on two sets. It is gone. The model detects. Three lists remain, and every one of them can only delete a span the model already proposed: 504 countries and well-known cities are never a person, a bare city that is not also a first name is never a person, and months, weekdays and demonyms are never a location. Places that are also given names, Paris, Austin, Florence, Jordan, are left to the model.
What I tried that did not work
Wider frames in the generator raised recall and put junk inside source code: Visual, Neural Network, Bayes, Dijkstra as people. Twice.
A wider network, hidden 128 with 686k parameters, recovered 12 of the 34 misses on the hard sets. It also produced team at 0.96, yesterday at 0.78 and Hello at 0.64 as PERSON, now inside prose rather than code. Latency went from 252 to 350 ms on 320 KB. Width moved along the same trade-off curve. The defect is in what the generator labels, not in what the model can hold.
Lowering the PERSON threshold from 0.6 to 0.3 changed no recall figure on any set. When the model prefers O, the threshold is not the knob. Every one of the 34 missed names scores 0.84 to 0.97 when placed in X signed it off. The misses are sentence positions, not vocabulary.
Against Presidio, measured
Once the model was in the gateway I wanted a fair comparison, not the memory of a bad week. I brought both Presidio services back up from the gateway’s own compose file, warmed them so the spaCy model was loaded, and drove them with the same bodies I gave the model. Presidio had all eleven cores. The model was held to two threads, which is how the gateway runs it.
| Body | Presidio | Model | Faster by |
|---|---|---|---|
| 2 KB, a chat turn | 30.2 ms | 2.3 ms | 13x |
| 11 KB, a Claude Code request | 135.5 ms | 16.0 ms | 8.5x |
| 64 KB | 1,479.6 ms | 70.7 ms | 21x |
| 320 KB | 22,661.4 ms | 283.2 ms | 80x |
| 1 MB | HTTP 500 | 1,596 ms | |
| 5 MB | HTTP 500 | 6,416 ms |
The gap widens with body size because Presidio’s cost per byte rises and the model’s stays flat. Inference is chunked at 64 KB, so a 5 MB body costs no more memory than a 320 KB one. Presidio could not process that body at all.
The footprint is the part I keep coming back to. The model adds 9.8 MB of resident memory to a Python process the gateway was already running. Presidio’s 785 MiB was on top of everything, in two containers of its own, eighty times more before serving a single request. On disk it is 533 KB against 1,205 MB of images, a ratio of about 2,263 to one. The whole model is smaller than the favicon of most websites. And there is no network hop any more: the guardrail cannot be taken down by a sidecar, because there is no sidecar.
One machine, an M3 Pro under Docker Desktop. The ratios hold. The absolute numbers will not.
What it still gets wrong
Short chat is the hard case. The misses are almost all the same shape: an unusual verb of address followed by a name, pester Petrov, huddle with Haddad, collar Solberg. The names themselves score above 0.9 in any frame the generator knows, so this is a generator gap, and the next versions go there.
Spans sometimes take the wrong label. Account holder Rachel Whitaker, 14 Ashcombe Road, Bristol. masks the address as <PERSON> rather than <LOCATION>. The data is redacted either way, the tag is wrong.
Looking inside
The thing I enjoyed most was the last week. I built a small local page that runs the PyTorch checkpoint with forward hooks. You type any text and it shows, aligned to the same byte axis, the per-byte probabilities, the spans decode keeps, and every layer of the network as a heatmap of 96 channels by position. A name appears as a vertical band that grows deeper into the network. Click a masked span and it shows which input bytes decided it.
It is not a debugging tool in the usual sense. But when you have to explain to someone why a model with no company data in it is safe to trust with theirs, showing them the bytes that fired is a better answer than a precision table.