Supervisors, all the way down
The BEAM (Elixir, Gleam, OTP) and Kubernetes, explained for engineers who use neither, with mycelia as the example.
1 · The stack, outside in
This is where one piece of mycelia runs. Each box sits inside the box around it. The right-hand side says who brings it back when it dies.
/healthRead it bottom-up as an escalation chain. When a process fails, its supervisor restarts it. If that supervisor fails too often, its parent takes over, and so on up the tree. If the top supervisor gives up, the BEAM exits. Kubernetes then sees a dead container and starts a new one. Purple is the BEAM's job. Blue is Kubernetes'.
2 · The BEAM in two minutes
The BEAM is a virtual machine, like the JVM. Erlang was built on it at Ericsson in the '80s for telephone switches, which had to stay up for years. Elixir and Gleam are newer languages that compile to the same VM and get all of its behaviour.
A "process" is not an OS process
It's a lightweight thread managed by the VM. It starts in about a microsecond with a few KB of memory, and one VM can run millions of them. Think goroutine, or an actor.
Nothing is shared
Each process has its own memory and its own garbage collection. The only way to talk is to send a message to another process's mailbox. So one process crashing cannot corrupt another's state.
"Let it crash"
Don't write defensive code for every unexpected case. Write the happy path. When something weird happens, the process dies, and its supervisor restarts it in a clean state, which usually fixes the problem.
OTP is the standard library
OTP provides the building blocks: GenServer (a process holding state that answers messages), Supervisor, Registry (look up a process by name), Task. Everyone uses these, so every Elixir app has the same shape.
| Name | What it is | In mycelia |
|---|---|---|
| Erlang | The original language and VM, and OTP's home. | The runtime under everything. |
| Elixir | A Ruby-like language on the BEAM. Dynamically typed. | The whole server: mycelia/lib. |
| Phoenix | Elixir's web framework (think Rails). LiveView renders pages on the server and pushes changes over a WebSocket. | HTTP, the API, and the WebSocket channels to the browser. |
| Gleam | A statically typed language that compiles to the BEAM or to JavaScript. | The browser UI (with the Lustre framework), compiled to JavaScript. Here Gleam doesn't run on the BEAM at all. petrichor is where it does. |
3 · Supervisors: try it
A supervisor is a process whose only job is to start its children in order, watch them, and restart them when they die. Supervisors can supervise other supervisors, which makes a supervision tree. Each supervisor has a strategy for what to restart:
| Strategy | When a child dies… | Use it when |
|---|---|---|
one_for_one | restart only that child | the children are independent |
one_for_all | stop and restart every child | they only make sense together |
rest_for_one | restart that child and every child started after it | later children depend on earlier ones |
Each supervisor also has a restart limit, by default 3 restarts in 5 seconds. Past that, it assumes restarting isn't helping, stops all its children, and dies itself. That hands the problem to its own parent.
A toy app
Tap a box to crash it. Switch the strategy and crash the same box again. Crash things 4 times in a row to watch it escalate. The real default limit is 3 restarts in 5 s; the demo stretches that to 15 s so you can tap fast enough.
mycelia's real tree (trimmed)
These are the actual module names and strategies from mycelia/lib. Each agent the owner talks to is one Spores.Tree: a Task supervisor that reads its event stream, a Host that owns its pod, and an Agent that holds the conversation.
Try these: crash Agent in spore A, and only it restarts, reloading the conversation from the database. Crash Host, and Agent restarts too, because a new pod connection means a new agent (rest_for_one). Crash Registry, and every spore's tree stops, because their names lived in it. TreeSupervisor comes back empty, and each tree starts again the next time its spore is used (tap a grey tree to do that). Spore B never notices anything that happens to spore A.
4 · Kubernetes in two minutes
Kubernetes runs containers across a group of machines (a cluster). You don't tell it what to do, you tell it what should exist, for example "3 copies of this image, reachable at this name". Controllers run in a loop, comparing what exists with what should, and fix the difference. Every Kubernetes feature works this way.
| Term | Plain meaning | Rough analogy |
|---|---|---|
| Cluster | A group of machines plus the control plane (API server, scheduler, controllers) that manages them. | A datacenter with an autopilot |
| Node | One machine in the cluster. Its kubelet starts and watches the containers placed on it. | A server, with a local babysitter |
| Pod | One or more containers that share a network address. The smallest thing Kubernetes schedules. Disposable: a replacement gets a new name and IP. | A process |
| Deployment | "Keep N pods of this image running." It replaces missing pods and rolls out new versions gradually. | A supervisor |
| Service / Ingress | A stable name (and address) that routes to whichever pods are currently ready. Ingress exposes it to the internet. | A load balancer / a registry |
| Probes | Liveness: "is it alive? if not, restart it." Readiness: "should it get traffic yet?" mycelia's both hit /health. | A health check |
| Namespace | A folder for resources, with its own permissions (RBAC), quotas and network rules. | A tenant / a project |
| Manifest | A YAML file describing what should exist. The owner's live in the mirage repo. | Infrastructure as code |
A Deployment keeping 3 pods alive
Tap a pod to kill it. Tap a node's name to power it off (or on). "Roll out v2" replaces pods one at a time and never drops below 3 ready, the same maxSurge: 1, maxUnavailable: 0 settings mycelia's own Deployment uses.
Timing is sped up. A real pod starts in seconds, and Kubernetes waits about 5 minutes before it gives up on a powered-off node.
5 · Side by side
Once you know one, you mostly know the other. The ideas are the same at different sizes:
| Idea | BEAM / OTP | Kubernetes |
|---|---|---|
| Unit of work | process (KB, µs to start) | pod / container (MB, seconds to start) |
| Isolation | own memory, messages only | own filesystem and IP, network only |
| Who restarts it | its supervisor | the kubelet (container), the Deployment (pod) |
| What to restart | strategy: one_for_one / one_for_all / rest_for_one | only the failed pod (always one_for_one) |
| Giving up | restart limit → the supervisor dies → its parent decides | CrashLoopBackOff: keep retrying, waiting longer each time |
| Finding it by name | Registry, named processes | Service DNS name |
| Noticing it died | monitors and links (instant, built into the VM) | probes and heartbeats (polled every few seconds) |
| Desired state lives in | code: the child list in init/1 | YAML manifests in the API server |
| Scope | one program on one machine (or a few, clustered) | many programs on many machines |
Why use both? The BEAM can't bring back a dead machine or ship a new version of the image. Kubernetes is far too coarse and slow to restart a single conversation in a few microseconds without dropping the others. Each covers the other's blind spot.
6 · How mycelia uses both
mycelia is a web app that starts and watches AI coding agents ("spores"). Each spore is two things at once:
Inside mycelia: a process tree
Spores.Tree → Tasks, Host, Agent. It's cheap and fast, and it holds the state the browser sees. Up to 16 trees run in each mycelia pod.
In the cluster: a pod
A pod in the spores namespace that runs Claude Code plus a small Python bridge. It's heavy (2 CPU, 3 GiB), and it's where the actual work happens.
The Host process links the two. It calls the Kubernetes API to create the pod, watches it, and deletes it once the spore has been idle for 30 minutes. In other words, a supervisor tree in mycelia acts as a Kubernetes controller for agent pods. Kubernetes people call that pattern an operator.
The two layers are deliberately independent:
- An agent process crashes: OTP restarts it in microseconds. It reloads the conversation from Postgres and reconnects to the same pod. The agent's work isn't interrupted.
- mycelia itself is redeployed: Kubernetes starts the new mycelia pod before stopping the old one. The two BEAMs connect into a cluster, and the old one hands its trees over. Spore pods keep running throughout. At boot, a
reattachtask rebuilds a tree for every spore pod it finds. - A spore pod dies (out of memory, evicted): the Agent loses its connection and asks the Host to check. The Host sees the pod has gone and creates a new one. The spore's files live on a volume that outlives pods, so the conversation picks up where it left off.
7 · Glossary
| GenServer | An OTP process that holds state and answers messages one at a time. Most of mycelia's processes (Agent, Host) are GenServers. |
| DynamicSupervisor | A supervisor whose children are added at runtime rather than listed up front. TreeSupervisor adds a tree each time a spore starts. |
| Link / monitor | The VM-level "tell me when that process dies" signals that supervisors are built on. |
| PubSub | Broadcast messages to every process subscribed to a topic. It's how a spore's update reaches every open browser tab. |
| Phoenix Channel | A WebSocket topic between a browser and a server process. |
| k3s | A small, single-binary Kubernetes distribution. All the owner's clusters run it. |
| RBAC | Kubernetes permissions: which accounts may do what in which namespace. mycelia's account can manage pods in spores and little else. |
| NetworkPolicy | A firewall rule between pods. Spore pods can reach the internet, but not galaxy's internal services. |
| CrashLoopBackOff | The status of a container that keeps dying. Kubernetes waits longer before each retry, up to 5 minutes. |