Skip to content
hermeticdocs

Day to day: ssh, logs, rerun

hermetic ssh atlas                  # tailscale ssh, ACL-scoped, no keypair involved
hermetic logs atlas [unit] [-f] [--file agent|errors|gateway]   # streamed over the hermeticd RPC, not a log-shipping pipeline
hermetic agent rerun atlas          # resume a failed bootstrap from its first failed stage

--file is how you read Hermes rather than systemd: Hermes writes its own errors to $HERMES_HOME/logs/errors.log and leaves the journal carrying only its startup banner and uvicorn noise, so a failed agent turn is hermetic logs atlas --file errors, not hermetic logs atlas hermes-gateway. Stage and bootstrap failures stay on the journal path below. --file takes one of agent, errors, gateway and is exclusive with both a [unit] argument and --console — the CLI refuses the combination rather than silently preferring one.

There is no automatic converge. A config change made with agent set (secrets mode, provider, size intent, Hermes settings) is re-rendered and re-uploaded immediately, but nothing on a schedule pushes it to the box. It takes effect when you ask for it — hermetic plan rollout [--agent <name>...] writes the plan and hermetic apply converges each box in place, one at a time, waiting for it to report the new config — or on the agent’s next rerun or recreate. agent rerun is for a stuck bootstrap only — the agent must be in error; it writes a command on the row that hermeticd picks up on its own poll and resumes from the failed stage. rerun answers 409 CONFLICT instead of queuing if a previous rerun command on that row is still unacknowledged, or 409 LOCKED if another operator currently holds the agent’s lock — retry once whichever is holding it clears.

Bootstrap failed on a stage

Section titled “Bootstrap failed on a stage”

An agent stuck in error failed somewhere in bootstrap — a stage script, or a step before any stage even ran (fetching the fleet manifest, downloading or verifying a stage). Both land in error the same way, with the same bootstrap.stages/message shape. To fix it:

  1. hermetic agent get atlas (or the dashboard drawer) and read bootstrap.stages — the failed stage’s exit_code and message name what went wrong.
  2. Read the full stage log on the box: hermetic logs atlas streams /var/log/hermetic/stages/<id>.log for the failing stage. A stage that already succeeded has a completion marker at /var/lib/hermeticd/stages/<id>.ok holding its sha256 — rerun skips anything whose marker still matches, so only the failed stage and anything after it actually re-runs. The fleet manifest itself is cached on the box at /var/lib/hermeticd/fleet.json, if you need to confirm which release it last saw.
  3. Fix the outside cause (a bad SSM value, a full disk, a transient apt mirror — whatever the message points at).
  4. hermetic agent rerun atlas. Watch bootstrap.stages in agent get/the dashboard: hermeticd resumes at the failed stage with a bumped attempt count and re-runs every stage after it. An agent that sits in error for 24 hours without a successful rerun has its runner exit on its own; systemd restarts hermeticd-bootstrap.service and the same resume logic picks up where it left off.