Sandboxes on demand, without the laptop
The previous post ended with a problem. The sandbox tool works well on a Mac, but the sandbox lives on the Mac. Close the lid and the agent stops. Once agent jobs started running for hours, I wanted two things. A machine that stays on, and a browser page where each person requests their own sandbox and comes back to it later. This post is about the server version.
The first attempt was Rust
The obvious move was to keep the language. Over two days in late July I rebuilt the Linux path as a small server. A control plane driving Docker on a Linux VM, an HTTP API, a WebSocket bridge for the terminal, and Postgres for state. The same terminal UI became a remote client. It worked, and it taught me what the actual problem was.
The server does very little heavy work. It coordinates. It supervises many long-lived sandboxes, each of which can fail on its own without taking the others down. Each one needs its own lifecycle state machine. People want to watch them change state in real time, from a page, without a separate frontend. And at some point it should run on more than one node. That is a description of Erlang’s OTP, so the second version is Elixir with Phoenix and LiveView. The decision record says it plainly: Rust gives control and performance a coordinator does not need, and the isolation boundary is the VM, not the language. The Rust version is still there. It was two days well spent.
Isolation: a VM with its own kernel
The threat model is different from the laptop tool. On my Mac I am the only user and the agent is semi-trusted. On a shared host the workload is treated as hostile. It may try to escape, exhaust the host, or reach the network in ways it should not. A plain container shares the kernel with the host and with every other tenant, so it was rejected. gVisor intercepts system calls in user space and is stronger, but it has compatibility gaps and its own large trusted surface. Full VMs per sandbox are the strongest option and too heavy to boot and operate when sandboxes are created and destroyed all day.
The choice is Kata Containers on Cloud Hypervisor. Each sandbox is a standard OCI container that Kata boots inside its own microVM with its own guest kernel. The host needs KVM, which is a hard constraint on where this can run. Standard images keep working, and the boundary is real. In the spike, the sandbox reported guest kernel 6.18 while the host ran 6.8, with one cloud-hypervisor process on the host per sandbox.
The orchestrator drives Kata through nerdctl against the system containerd, where Kata is registered as a runtime handler. The entire backend is about 250 lines. The mapping is thin on purpose:
| Callback | Command |
|---|---|
| start | nerdctl run --detach --runtime io.containerd.kata.v2 --cpus … --memory … <image> |
| stop | nerdctl stop <ref> |
| restart | nerdctl start <ref> |
| status | nerdctl inspect --format '{{.State.Status}}' <ref> |
| destroy | nerdctl rm --force <ref> |
Everything sits behind one small interface, a behaviour in Elixir terms, resolved per sandbox from config. The test suite binds the same key to a mock, so the whole lifecycle is tested with no KVM on a Mac. CPU and memory limits become cgroup limits inside the guest, and the tests that need KVM check the exact values.
One process per sandbox
Each sandbox is one supervised Erlang process under a dynamic supervisor, found by id through a registry. It drives the backend and holds the live state. That state is a cache. The record of truth is one Postgres row per sandbox: status, mode, image, backend, the runtime reference, and the spec as JSON.
Status changes follow an explicit transition table, and the changeset rejects illegal moves rather than trusting callers to behave.
pending → starting → running → stopping → stopped → starting
↓ ↓ ↓ ↓
failed ←─────┴──────────┴──────────┘Every persisted transition is broadcast over PubSub. Admins subscribe to a global topic. A regular user subscribes to a topic for their own sandboxes and never receives another user’s event. The dashboard is a LiveView page and it is the only client. There is no separate frontend and no public API. Launch, stop, start, delete and open a terminal, all from the list.
There is no job queue. The only periodic work is a reconciler, a plain process that wakes up every minute. On boot it re-adopts workloads that are still running, so restarting the control plane does not kill anyone’s session. It also cleans up orphans: a record stuck in pending or stopping with no live process. It never touches a sandbox that still has one.
Who sees what
Authentication is local email and password. The generated scaffold came with magic links, confirmation emails and password resets, and I removed all of them, because the control plane will never send email. Single sign-on can come later, and the code has a place for it.
Ownership is enforced in the context and the engine, never in the pages. A user sees and launches only what they own. An admin supervises the whole fleet but is refused when they try to launch. A sandbox you may not see returns not found, not forbidden, so its existence never leaks. The owner is stamped server-side from the caller’s session and never read from a form. Registration cannot create an admin. The first one comes from a mix task on the server. Each user is capped at a number of active sandboxes, and the launch is refused before anything is created when the cap is hit.
State that survives a disconnect
This was the whole point, and the answer is less clever than I expected. Stop does not destroy. It halts the container and keeps it. Start is nerdctl start on the same id, so the same writable layer boots again and the work on disk is still there. The record moves from stopped to starting, and the process boots in restart mode instead of provision mode.
What does not survive a stop is memory. PID 1 of the image recreates a fresh tmux session on boot, so the shell comes back empty. That distinction is written down in the code, next to the restart function.
The disconnect case is separate and it is a tmux story. PID 1 owns the tmux server. Attaching from the browser runs tmux new-session -d, a no-op if the session exists, then tmux attach-session -d. That second -d evicts the stale client, so a reopened page repaints correctly. Reconnecting is just opening the terminal page again. The agent kept running while you were gone.
That only works because PID 1 owns tmux. Kata tears down an exec’s whole process tree when the exec ends, so a tmux server started by the attach dies the first time you press Back. On a clean detach the bridge writes tmux’s detach sequence and waits 150 milliseconds before closing the port, because closing it straight away kills the chain.
The terminal in the browser
The terminal is xterm.js in the browser talking over the LiveView socket that is already authenticated. On the server, a process opens an Erlang port on the attach command and bridges its stdio. Keystrokes arrive as input events. Output is base64 encoded before it is pushed, so raw bytes and ANSI escapes survive the JSON hop. The port is spawned with an executable path and an argument list, so there is no shell to inject into.
Two things took longer than they should have. containerd refuses an interactive exec unless the client’s own stdin is a console, and an Erlang port’s stdin is always a pipe. The fix is to run the exec under script, the util-linux tool that allocates a pseudo-terminal. And the shell is sized to the browser at attach time, but a mid-session resize never reaches the guest. An Erlang port cannot send the resize signal, and pushing stty into the live shell’s stdin would corrupt what you are typing. That needs a side channel and is still open.
Policy: the same rules as the laptop
The two things I cared most about in the laptop tool carried over unchanged. Egress is default-deny. Every sandbox gets an allowlist, and anything not on it is dropped inside the guest. It is the same nftables approach as before, with hostnames resolved at boot and pinned to their ports. Credentials are injected at launch as environment variables, scoped to that sandbox, and never written into the image. Both are running in production and have worked without problems since the first deployment.
One detail about where they get tested. The Mac development setup runs nested virtualisation inside a Lima VM. There Cloud Hypervisor cannot hot-plug the network device, so a sandbox on the laptop has no network at all. Egress and credentials are tested on the real host, and the local suite covers everything else through the mock backend.
Numbers and what is still open
The first version took three days. It is about 4,400 lines of Elixir in 39 modules with roughly 190 tests, four of which boot real microVMs and only run on a host with KVM. A 2 GiB microVM boots in about 17 seconds on the nested development setup, and bigger guests time out there. The production host does not have that problem.
What is still open is smaller than I expected. Users cannot pick an image. Every sandbox boots one curated base with git, tmux, Node and Claude Code, and the form says so. There are no idle timeouts and no cost tracking, only the per-user cap on active sandboxes. Mid-session terminal resize needs that side channel. And single sign-on is not there yet. None of those stopped people using it. That is the point of the whole thing: you ask for a sandbox from a browser, give the agent a long job, close the laptop and come back to it.