Skip to content
Spinbox is experimental: APIs change and data may be lost. Status lists what is built and what proves it.

Runners and hosts

View Markdown

A runner is a machine that runs workspaces. In the dashboard and the API a runner is also called a host. On AWS, runners are an autoscaling group of large EC2 machines with local NVMe disks and nested virtualization, spot by default.

  • Keeps a connection open to the control plane, and runs what the control plane says should run on it.
  • Starts each workspace as a QEMU/KVM virtual machine, confined to its own user, cgroup and empty network namespace.
  • Keeps each workspace’s disk on its local NVMe, and uploads its sealed layers to the bucket.
  • Enforces each workspace’s network policies, in nftables, outside the machine.
  • Opens sessions to the relay when someone connects to a workspace.

A runner opens every connection it has. It has no inbound rule at all.

A runner in the installation’s autoscaling group joins by who it is. It signs an AWS GetCallerIdentity request with its instance role, bound to this installation and to its own id, and sends it to the control plane. The control plane passes it to AWS STS and checks that the answer names the runners’ role. There is no token to create, store or rotate.

A machine outside the group can join with a one-time registration token from Admin → Runners.

A runner must run the control plane’s release. One of any other, older or newer, is refused when it registers, and the refusal says which release to run.

The control plane places a workspace on an active runner with room for its disk and memory. It prefers the runner holding fewer of the same team’s workspaces, so one team does not pile onto one machine. A runner whose local disk is more than 70% full stops taking new workspaces until it is back under 65%.

A preset may name a host class, a kind of runner, and its workspaces are placed only on runners of that class. An installation the module’s current release makes has one class, default, and every runner is of it. A spot class can be set to fall back: when AWS refuses spot capacity for its group, the control plane runs the group on demand for 30 minutes, then tries spot again.

On AWS the control plane sizes the runners’ group itself. It knows what nothing else does: that a workspace is waiting, and when the last one stopped.

  • With no runner up, the first workspace to start waits in Waiting for a host while the control plane asks for one. It is placed on the runner as soon as it registers.
  • If a workspace has waited ten minutes, and no runner was added in that time, the control plane adds one.
  • When nothing has run for runner_idle_minutes, it scales the group back to none. During the installation’s quiet_hours, it does so as soon as nothing is running.
  • It never asks for more than max_runners.

While workspaces run and none waits, a runner that holds nothing only it has, for as long as runner_idle_minutes, is emptied and given back to the group, one at a time.

A runner that is going away empties itself first. It suspends each of its workspaces, with memory, to the bucket, and each one resumes on another runner from where it was. This happens when:

  • AWS reclaims a spot runner (about two minutes’ notice),
  • the autoscaling group removes a runner (the group holds it for drain_seconds),
  • a release is rolled out and runners are replaced.

A runner restarting its own process is not a drain: the virtual machines keep running, and the new process adopts them.

ActionThe runnerIts workspacesUndo
DrainStays connected, takes no new workspaceKeep running; with --suspend, are moved offActivate
ActivateTakes workspaces again——
RevokeIts credentials are deleted and it is disconnected for goodRefused while it holds any, unless you choose to leave themNone: replace the machine

See Operating hosts.

A runner can vanish without draining: the machine loses power, or its network. The control plane does not guess. Its workspaces stay where they are, and each one has lost only what the runner had not yet uploaded.

  • The owner moves one with spin workspace recover <workspace>, which says how much would be lost and waits for --accept-data-loss.
  • An administrator can let the control plane move workspaces on its own, after a runner has been silent for rehome_after_seconds, when no more than rehome_max_unpublished_bytes would be lost: the workspace is moved, and its owner is told when it waits for them instead. This is off by default.

The owner is told by email: when a workspace is held after its host was lost, when a stop could not be taken, when it was stopped for not responding, and when it was moved off a lost host. Each event can be switched off.

Installations run Spinbox v20261027.01; the reference is generated from spin v20261114.01 (51f7e0dea). This page as Markdown, for a model or a terminal.