Skip to content
Spinbox is experimental: APIs change and data may be lost. Status lists what is built and what proves it.

Troubleshooting

View Markdown

Each machine’s group says InService when the machine is serving. What its boot said is kept in CloudWatch, even for machines that are gone:

Terminal window
tofu output boot_log # by role, the command that tails its machines' boot logs
aws autoscaling describe-scaling-activities --auto-scaling-group-name <group> --max-items 3

A machine refuses to start on a document that names another release than its image’s. That means the image and spin_version disagree: the images the module names are those of spin_version unless image_ids names others.

No machine runs SSH. The control plane’s and the proxy’s machines take a shell with Session Manager, on the one in service. A runner takes no Session Manager shell: its image carries no agent and its role grants none, so its serial console, its boot log and its metrics say what it does, and one that fails is replaced. When a runner has to be looked at from inside, an administrator opens a recorded root shell on it through the control plane.

Terminal window
aws autoscaling describe-auto-scaling-groups \
--query "AutoScalingGroups[].[AutoScalingGroupName,join(',',Instances[?LifecycleState=='InService'].InstanceId)]" \
--output text
aws ssm start-session --target <instance-id>

On a runner, when its metrics, logs and console have not answered:

Terminal window
spin admin runner list
spin admin shell open <runner-id> --reason "<why>" --incident

--incident opens the shell on the runner as it is, its workspaces still running; without it the runner is emptied first. Either way the session is recorded and the runner is retired once the shell ends, so it costs the runner.

Spin OS has the tools to look with — tcpdump, strace, ss, nft, conntrack, dig, bpftool, htop, iostat — because nothing can be installed on a machine.

  • dig +short app.<domain> should answer with the proxy’s address. If not, the domain’s nameservers are not the zone’s yet.
  • The proxy gets its certificate a few minutes after DNS resolves. Port 80 must be reachable for Let’s Encrypt.
  • If you set proxy_allowed_cidrs, you must be on one of those networks.

A workspace waits in Waiting for a host while no runner has room for it, and says why.

  • No runner is up. The control plane has asked for one. The first runner takes a few minutes to boot and register.
  • The runner cannot start. The account’s spot vCPU quota is too low, or spot has no capacity for the types named. describe-scaling-activities on <name>-runners says which. See Before you start.
  • Every runner takes no new workspace. A runner that was drained stays drained until it is activated:
Terminal window
spin admin runner list
spin admin runner activate <runner-id>
  • max_runners is reached, and the runners are full.
Terminal window
spin workspace status <name> # which party disagrees: control plane, supervisor, host, volume
spin workspace console <name> # what the machine printed while it booted
spin workspace history <name> # every change of state, with the reason

A workspace that failed three times is FAILED until it is started again. The state names the part that failed.

An administrator can follow one start from the request to the guest in Admin → Traces, searching for span_name:workspace.start.

Terminal window
spin workspace egress <name>

It lists what the workspace reached, what was refused, and the lookups that failed, with why. A destination refused by policy is fixed with a policy (spin workspace retag), not with the network. The record lives only while the workspace runs: start it and try again before reading it.

Replace its machine: terminate it without lowering the group’s size, and the group starts another, which joins as a new host. Drain it first if it holds workspaces you want kept.

Terminal window
aws autoscaling terminate-instance-in-auto-scaling-group \
--instance-id <i-...> --no-should-decrement-desired-capacity

Installations run Spinbox v20261027.01; the reference is generated from spin v20261114.01 (51f7e0dea). This page as Markdown, for a model or a terminal.