Provisioning machines over the network
Engineering account · Reviewed 2026-11-18
A fleet is only as easy to run as it is to rebuild. If installing an operating system means finding a USB stick, a monitor and a keyboard, every new machine and every broken one waits for someone to be in the room. This page describes how the Corvis fleet installs and rebuilds its computers over the wired network instead, which steps still need a person, and the small homemade KVM that covers most of those.
It is a first-party engineering account, reviewed against the code that implements each step on the review date. The companion guide How the fleet is laid out describes which machine does what. Public references support the general techniques; they are not independent verification of this fleet.
One machine answers every network boot
Every machine is installed over the network. Exactly one machine holds the network install duty and answers network boots, because two boot servers on one network would both answer the same machine. The configuration refuses to build if more or fewer than one machine holds it.
That machine is also the one machine that can never be wiped this way. It serves every install, its own included, and it holds the fleet's only copy of the shared data, so its entry in the roster deliberately leaves out the network adapter's hardware address that arming a reinstall needs.
An install also depends on the fleet's service stack, where the job queue lives, and on the private network's coordination service, which issues the new machine's one-time key. Serving the boot is one machine's job; completing an install needs those to be up too.
The boot, step by step
- The boot service answers without handing out addresses. It runs as a proxy DHCP server, which adds boot instructions to the network's normal address handout instead of replacing it, and serves a signed network bootloader (iPXE) to machines that ask.
- The bootloader asks for this machine's script. The server answers only a machine with a waiting arm, and only once. Any other machine gets nothing, boots its own disk as usual, and is recorded as a sighting.
- A new computer is offered to the owner. A sighting no known machine accounts for becomes a text message to the owner, once per machine. When the owner approves it, the fleet records the approval, writes the new machine's roster entry, and arms it. The one hand step left is to network-boot it once.
- A known machine is armed on request. Arming refuses an unknown machine, one that must never be wiped, and a machine that already has an arm in progress.
- The machine reaches the installer. A machine with no operating system network-boots by itself. A machine running a Linux that boots through GRUB is sent there remotely: a script checks that it is safe, writes the installer into the boot partition, tells GRUB to use it for the next boot only, and reboots. A machine that already runs the fleet's own system uses a different boot loader, so it is reinstalled by a firmware network boot instead.
- The install guards itself. The installer sends its console log over the network to the boot server, so it can be read without a screen. It asks the server for its install program repeatedly and, if none arrives, reboots into the machine's existing system with the disk untouched. The program refuses to run unless the firmware's Secure Boot is in setup mode. It then wipes and partitions the disk, installs the fleet's NixOS configuration, gives the machine its own Secure Boot keys alongside Microsoft's and the fleet's, and makes the new disk the first boot entry.
- First contact. On first boot the machine joins the fleet's private network with its one-time key and checks in with its new address and host key. The fleet gives it its roster address back, copies its new host key to the machines that connect to it, and runs its first check. Nothing is installed afterwards: the configuration already carries the toolchain.
NixOS supports this route as an image its users build and serve from their own PXE or iPXE setup; it does not ship a ready-made network installer.1
What still needs a screen or a hand
The install needs no monitor, but some firmware settings are not managed by code and need a screen and keyboard:
- putting Secure Boot into setup mode, which both the remote start and the install require;
- turning on network boot, or the boot order that reaches it;
- other firmware switches that block a network install on particular models.
A brand-new computer must also be network-booted once by hand. The KVM below handles these where it can see the firmware.
Wake-on-LAN: armed everywhere, sent by nothing
Every machine's wired adapter and firmware are configured to wake on a magic packet, and the configuration refuses to build a machine that names no wake adapter. Nothing in the fleet's code sends that packet, though. A wake packet is a broadcast on the local network that names the adapter's hardware address, so it cannot travel over a private overlay network; something always on and wired to the same local network has to send it.2 Until a sender like that exists, a machine that is fully off is turned on by a hand.
A homemade KVM
For the screen-and-keyboard steps there is one homemade KVM: an Orange Pi Zero 2W whose USB port presents itself to the target computer as a keyboard and mouse, and a USB HDMI capture dongle that shows the target's screen. Because it is real hardware on the cable, it works at a firmware setup screen that no software inside the machine can reach. The target announces its own name to the KVM, and the cables are moved from machine to machine by hand.
A session drives it through six tools: list the KVM, take a screenshot, point and click on the screenshot, move a firmware-style mouse, press key combinations, and type text. Every input is refused unless a screenshot of the same KVM was taken within the last minute, so nothing is typed blind; the only input sent unseen is a lone Shift press that wakes a blank screen. The KVM draws its power from the target's USB port, so it cannot turn a computer on, which is the other reason Wake-on-LAN matters.
PiKVM's guide ranks this kind of USB capture dongle as the fallback to an HDMI-to-CSI bridge, warning among other things that a dongle may not work in the BIOS.3 On this fleet the capture is held streaming from boot and is used at firmware setup screens.
What this account does not establish
This page describes the code and configuration as they were on the review date. It does not show that every machine can be network-booted or woken; some cannot, and which ones is deliberately left out. It is not a timing or reliability measurement. Machine names, addresses, ports and credentials are omitted.
All citations below lead to public primary material. They explain the general mechanisms used here. The fleet-specific behavior is this page's first-party engineering account, not an independently reproduced result.
-
NixOS manual: Booting from the netboot media (PXE), lines 1-19: the audience of an existing PXE or iPXE setup, and the kernel, initrd and iPXE script users build themselves. ↩
-
Tailscale: Wake-on-LAN with Tailscale and UpSnap, sections What Wake-on-LAN is and is not and Option 1: the packet's hardware address and Layer 2 broadcast, Layer 3 overlay, and an always-on sender on the same wired network. ↩
-
PiKVM: DIY V2 quickstart, the video capture options: the recommended HDMI-to-CSI bridge and the HDMI-USB dongle described as possibly not working in the BIOS. ↩
