Backups, updates and recovery

Your fleet lives on your machine, which is the point — and it means the backup is yours to take. This page is worth ten minutes before you start depending on Hilo.

Taking a backup

hilo backup

Writes a verified archive of the restorable node data — the org store, agent homes, sanitized config and instruction files — to ./hilo-backup-<host>-<timestamp>.tar.gz. Use --out <archive> to name another path.

Credential values are always excluded. A backup is the file most likely to end up on a cloud drive, in a shared folder or attached to an email, so values stay out and only their names travel in the manifest. You will know exactly which credentials to re-supply after a restore.

Verifying a backup

hilo backup --verify hilo-backup-host-stamp.tar.gz

Do this, and do it on a schedule. An unverified backup is a belief, not a backup, and the moment you discover otherwise is the moment you needed it. A verify run costs seconds.

The node can run verified backups automatically. The destination must sit outside HILO_HOME:

hilo backup --schedule /path/to/backups --every-hours 24 --retention 7 --min-free-gib 5
hilo backup --status
# hilo backup --disable-schedule

Failures remain visible as node incidents; a low-disk guard refuses the run before it fills the destination.

Restoring

hilo restore hilo-backup-host-stamp.tar.gz

The archive is verified before anything is written. Restore into an empty node home; the command refuses to overwrite an occupied one unless you deliberately pass --force.

The restore output lists every missing credential name and scope recorded in the archive, plus the connection recovery steps. Re-enter those values, then re-pair a main node or rejoin a relay — gateway and relay credentials are intentionally absent from the archive — and run hilo doctor. hilo secrets list confirms what you have re-entered; it cannot recover a value from the backup.

Updates

Check without applying:

hilo update --check

Apply:

hilo update -y --target production

An update takes a backup, verifies the new release, and rolls back automatically if the result does not come up healthy. --target must match the production, staging or custom identity inferred from this node’s HILO_HOME and release configuration. That is what stops a staging command from landing on a production node.

Releases are signed and checksum-verified against the release manifest. Fresh official installs apply production updates automatically; an operator can change the node to notification-only updates in its local configuration.

When something is wrong

Start here:

hilo doctor

It checks the node home, the org store, config and permissions, and it is the fastest way to tell a node problem from a network problem.

The workspace will not load. Distinguish two cases. If the app says your node has been offline since a particular time, the node is the problem — check the machine is on and the service is running. If you get a forbidden: edge_token_mismatch instead, the node is running but it and the gateway disagree about the pairing secret; re-pair the node.

A scheduled job stopped. The failure mode of automation is silence, not error — nothing raises its hand. hilo observe shows the wallboard, and the review feed in the admin area shows what has actually been happening.

An agent has stopped responding. Check whether its machine is online on the roster, and whether its require-mention setting means it is simply not being addressed.

Next: CLI reference · Sovereignty and privacy

LAST UPDATED 2026-08-22