The indexer that ate five gigabytes

This is the post in the series about Semantic Graph where something breaks. In early August the indexer for one of our products got stuck in a crash loop. Fixing it properly turned out to touch a design decision I had not questioned since the beginning, and that fix is still not made. I want to write down why.

The crash loop

The indexer pod died about seventy seconds after every start. Eight times in under half an hour. Exit code 137, reason OOMKilled, against a 4 GiB memory limit.

Every restart followed the same path. Crash recovery found a job that had been running when the process died and reset it to pending. The worker picked it up, started indexing the same repository, and the pod was killed before it finished. The job never completed and never failed, so it was there again on the next start.

We raised the limit to 6 GiB. The job finished in under four minutes and the queue drained. That unblocked everyone. It fixed nothing.

Why a durable queue loops

The indexer has one worker. It takes jobs from a queue and runs them one at a time, sequentially. That was a resource decision. Indexing is CPU-heavy and memory-heavy, and running two repositories in parallel on one pod was more expensive than the time it saved.

The queue is persisted in Memgraph. Each job is a :JobQueue node with a kind, its arguments as JSON, and a status. This started as a practical thing. Early on, indexing failed a lot, and I was losing track of what had run and what had not. Writing the queue into the same graph the indexer already had a connection to was the smallest possible fix. Then it became a feature. A queue that survives restarts is what makes a single sequential worker safe: you can redeploy in the middle of a backlog and nothing is lost.

On startup the indexer sweeps for damage:

MATCH (run:IndexRun {status: 'running'})
SET run.status = 'abandoned', run.finished_at = $now;

MATCH (j:JobQueue {status: 'running'})
SET j.status = 'pending';

That second statement is the loop. The queue is durable, the process is not, and a job that kills the process is put straight back at the front. There is no attempt counter on the node. Nothing says “this one has died three times, stop”. I knew that when I wrote it. I did not think a single repository could take the pod down.

What we measured

With the limit raised and the queue empty, the pod sat at about 5.5 GiB. The page cache was 2 MB, so this was real process memory, not something the kernel would reclaim.

There were three processes in the container: the parent, and two parse workers.

Process	Resident	Private dirty
parent	4.4 GiB	1.1 GiB
worker 1	4.3 GiB	12 MB
worker 2	4.3 GiB	13 MB

The workers looked huge and owned almost nothing. They had been forked about three minutes into the run, and they inherited the parent’s heap through copy-on-write. Shared dirty memory was 3.2 GiB.

Two things follow. The parent holds the memory. The workers pin it for as long as they stay alive, because pages shared with a live child cannot be returned.

The root cause

The parse phase of an indexing run looks like this, slightly simplified:

def _parse_all():
    out = []
    for result in pool.map(_parse_file, work):
        out.append(result)
    return out

Every parsed file in the repository is alive at the same time. Two more lists downstream, one for the graph write and one for the vector write, point at the same objects. Peak memory is a function of repository size, with no ceiling.

The code already dropped those buffers by hand before the resolve phase, with a comment noting the cost on large repositories. But that release happens after the peak. And freed heap in a Python process does not go back to the operating system by default. glibc keeps it, in case you want it again.

I also ruled out the thing I suspected first. The embedding runtime has an arena allocator that is known to hold memory. But another product’s indexer ran two jobs of similar length on the same image and the same configuration, and sat at 0.23 GiB afterwards. One caveat I wrote down at the time: job duration does not measure how much text a job embeds. Strong evidence, not proof.

Three fixes, two made

Give the memory back. Three lines:

def trim_heap():
    ctypes.CDLL("libc.so.6").malloc_trim(0)

This asks glibc to return free heap to the OS. It also works as a diagnostic. If memory drops after a run, the allocator was holding freed pages and the problem is the peak, not a leak. If it does not drop, something still references the data and there is a real leak to find.

Close the parse pool after every job. The pool used to live for the life of the process. The docstring justified it: each worker spends about a hundred milliseconds importing tree-sitter grammars, and on a fleet of small webhook-driven reindexes that spin-up was the dominant cost. Reasonable, when it was written.

It stops being reasonable when the workers pin gigabytes between jobs. Closing the pool costs about two hundred milliseconds per job and releases the inherited pages. It has to cancel queued work as it goes, because pool.map submits every file up front, and a cancelled run can leave thousands of futures nobody will ever read. Waiting for them would block the worker for the rest of the parse.

Both run in the dispatcher’s finally, so they cover jobs that succeed, fail, time out or are cancelled by an operator:

finally:
    await _persist_delete(state, node_id)
    await _release_run_memory(state)   # close pool, then trim_heap()

They are cheap, and they are deployed. If the pod now idles low after a large job, glibc was the problem. If it does not, we are back to hunting.

Parse in chunks. This is the one that removes the ceiling, and it is the one we have not done. pool.map already returns an iterator. Stop draining it into a list, parse a chunk, index the chunk, throw it away, repeat. Peak memory becomes a function of chunk size instead of repository size.

The obvious version breaks a guarantee I rely on.

The guarantee

A full reindex of a repository is atomic, on purpose. The order is: embed everything into a staging Elasticsearch index, then write everything into Memgraph in one transaction, then flip the alias so the staging index becomes live. If the graph commits and the alias flip fails, the code names that state “diverged”, raises an alert, and asks for a retrigger. It is the one failure mode we cannot hide, so it is loud.

The phases are separated for a reason. Embedding is slow, and you do not want it happening inside an open graph transaction. But both phases need the same parsed objects. Phase one embeds them. Phase two writes them to the graph. So you cannot discard a chunk after phase one, and indexing chunk by chunk turns one atomic reindex into a series of partial ones.

There are three ways out, and none of them is free:

Option	What it costs
Spill parsed output to disk between the two phases	Disk I/O, and a serialisation format to maintain
Parse twice, once per phase	Roughly doubles parse time
Hold the graph transaction open while embedding	Long transactions, which the phase split exists to avoid

My leaning is the first one. Disk is cheap, the parsed model is already a plain dataclass, and it keeps both the atomicity and the phase separation. But it is a change to the shape of the indexer, not a patch, and it needs the team to agree before anyone writes it. The cheap fixes went in first precisely so that we would know which problem we have before we choose.

What I still do not know

Why this surfaced now. The repository that triggered it has been indexed many times before. Either it grew, or someone queued a mass reindex, or an image update changed the memory profile. All three are checkable and none is checked yet.

What that repository does that no other does. A run under tracemalloc would answer that in an afternoon.

And whether the current state is safe. If the pod idles low after the next large job, the two fixes did their part. Even then, a large webhook-driven reindex can push it over until chunked parsing lands. A restart clears the held memory. That is not a fix. It is a runbook entry.

Where this leaves things

Six posts so far, and the thread through all of them is the same. A tool that exists because grep does not scale to a few hundred repositories. A graph and a vector index because neither is enough alone. Two databases chosen by measuring, including a failed spike that made the winners faster. Call edges built without a compiler, with an honest count of what could not be resolved. An API shaped for a caller that guesses, stops early and pays per hop. And a memory problem that traces straight back to a design decision that was correct when it was made.

None of it is finished. The unresolved count is not zero, the chunked parser is not written, and the next repository will find the next assumption. That is the job. There will be more posts as it goes.