Backups, updates and recovery

Your fleet lives on your machine, which is the point — and it means the backup is yours to take. This page is worth ten minutes before you start depending on Hilo.

Taking a backup

hilo backup

Writes a verified archive of the restorable node data — the org store, agent homes, sanitized config and instruction files — to ./hilo-backup-<host>-<timestamp>.tar.gz. Use --out <archive> to name another path.

Credential values are always excluded. A backup is the file most likely to end up on a cloud drive, in a shared folder or attached to an email, so values stay out and only their names travel in the manifest. You will know exactly which credentials to re-supply after a restore.

The portable archive excludes private harness profiles and raw turn transcripts; the derived memory index can be rebuilt from saved notes.

Codex agents are backed up with their brain file, skills, knowledge folder, the memory documents Codex generated, and a compact copy of Codex’s memory and thread checkpoints, so a restored agent keeps what it learned. The archive never carries a ChatGPT sign-in, raw session logs or Codex’s other runtime databases, and a backup is refused — keeping the last good archive — if a Codex agent is mid-turn or its state cannot be verified. After a restore, each Codex agent stays paused until it signs in with ChatGPT again. A partial backup reports omitted roots rather than silently claiming complete coverage. A full node backup is distinct from a narrowly scoped, operator-created memory-only migration backup. A normal memory policy update does not require copying every working repository or replaying old chats.

Verifying a backup

hilo backup --verify hilo-backup-host-stamp.tar.gz

Do this, and do it on a schedule. An unverified backup is a belief, not a backup, and the moment you discover otherwise is the moment you needed it. A verify run costs seconds.

The node can run verified backups automatically. The destination must sit outside HILO_HOME:

hilo backup --schedule /path/to/backups --every-hours 24 --retention 7 --min-free-gib 5
hilo backup --status
# hilo backup --disable-schedule

Failures remain visible as node incidents; a low-disk guard refuses the run before it fills the destination.

Restoring

hilo restore hilo-backup-host-stamp.tar.gz

The archive is verified before anything is written. Restore into an empty node home; the command refuses to overwrite an occupied one unless you deliberately pass --force.

The restore output lists every missing credential name and scope recorded in the archive, plus the connection recovery steps. Re-enter those values, then re-pair a main node or rejoin a relay — gateway and relay credentials are intentionally absent from the archive — and run hilo doctor. hilo secrets list confirms what you have re-entered; it cannot recover a value from the backup.

Updates

Check without applying:

hilo update --check

Apply:

hilo update -y --target production

An update takes a backup, verifies the new release, and rolls back automatically if the result does not come up healthy. --target must match the production, staging or custom identity inferred from this node’s HILO_HOME and release configuration. That is what stops a staging command from landing on a production node.

Releases are signed and checksum-verified against the release manifest. Fresh official installs apply production updates automatically; an operator can change the node to notification-only updates in its local configuration.

When something is wrong

Start here:

hilo doctor

It checks the node home, the org store, config and permissions, and it is the fastest way to tell a node problem from a network problem.

The workspace will not load. Distinguish two cases. If the app says your node has been offline since a particular time, the node is the problem — check the machine is on and the service is running. If you get a forbidden: edge_token_mismatch instead, the node is running but it and the gateway disagree about the pairing secret; re-pair the node.

A scheduled job stopped. hilo observe shows the wallboard, and the review feed shows what has been happening. For memory maintenance, inspect the agent’s Overview: failed consolidation, a pending budget and a stale index are different states. The daily Hilo fleet report summarises them across agents. Do not mark a failed batch complete or reset its index just to clear a warning.

An agent has stopped responding. Check whether its machine is online on the roster, and whether its require-mention setting means it is simply not being addressed.

Next: CLI reference · Sovereignty and privacy

LAST UPDATED 2026-10-17