Docs / Server / Persistence and durability

Persistence and durabilityalpha

What survives a restart, a crash, and a kill -9?

Durability in moxzid is a snapshot plus a write-ahead log, per actor, under the directory you pass to -s. The constraint that comes first: without -s the server is entirely in-memory and everything dies with the process. That default is deliberate and explicit either way — a scratch run wants it, and anything real must say otherwise.

What is on disk#

state/
  principal.seed                       32 random bytes: this host's actor namespace
  <hex-actor-id>/
    snapshot.bin                       heap + stable memory + cycle balance, at a seq
    wal.log                            messages accepted since that snapshot
    outcalls.log                       HTTPS outcall responses, keyed by message seq
    history.jsonl                      the recent-message ring, for post-mortems
    module.wasm  actor.json            for actors created at runtime, not by the manifest

The seed is created once and kept, which is why an actor's derived address is the same tomorrow. Two servers must never share a state directory.

The ordering that makes it work#

A message is written to the WAL and flushed and fsynced before it executes, never after. Logging afterwards would look equivalent and is not: a crash between executing and logging silently loses a message the caller was told had been accepted, which is the one failure a durable system may not have. The price is that a message which trapped gets replayed and traps again — which is correct, because the trap is deterministic and replay reproduces exactly the state the crash interrupted.

Replay must be exact, so the non-deterministic inputs are recorded rather than re-sampled:

Input	How replay reproduces it
`ic0.time`	the timestamp is written into the WAL entry and restored before the message re-runs
HTTPS outcall responses	recorded in `outcalls.log` against the message's sequence number and replayed from there
`raw_rand`	derived from a counter that is part of the actor's persisted record, not drawn fresh

Snapshots are written to snapshot.tmp, fsynced, and renamed into place, so a crash mid-write leaves the previous snapshot intact; only then is the WAL truncated. A crash between the rename and the truncation costs a replay, never a loss.

The three failure modes#

Mode	What moxzid does	What comes back
Clean shutdown (`SIGTERM` / `SIGINT`)	finishes the message in flight, starts nothing new, checkpoints every actor, exits `0`	the snapshot. `scripts/moxzid_shutdown.sh` asserts the restart replays nothing
Crash while a message is running	nothing — the process is gone	the snapshot, then every WAL entry after it, including the message that was mid-flight
`kill -9` or power loss	nothing, by definition	the same. `scripts/moxzid_crash.sh` proves it twice: once by snapshot, and once with `--snapshot-below 0` so recovery must replay the log

The two recovery paths are different code, and only one of them runs by default, which is why the gate exercises both. --snapshot-below <BYTES> (default 50 MB) is the switch: actors at or below that heap size are snapshotted after every message, so their log is usually empty and recovery is a file read; larger actors ride the WAL. Setting it to 0 snapshots never and is how the replay path gets tested rather than assumed.

fsync is on. Without it "durable" is a wish rather than a property — a page-cache write survives the process dying and not the machine.

What does not come back#

Lost	Why
A message accepted by the socket but not yet started	durability begins at the WAL append, which happens when the actor thread picks the job up. A queued message dies with the process and its client sees a dropped connection
A half-written WAL record	records are length-prefixed, so a torn tail is detected. Replay stops at the first record it cannot read whole. That message was never acknowledged to anyone, so dropping it is correct
The last line of `history.jsonl` after a power cut	history is a deliberately weaker diagnostic tier: no fsync, plain greppable JSONL. Losing a line costs a post-mortem aid, not state
Query results	a query is never journalled, cannot write, and is rolled back whatever happens
An actor whose stored module no longer loads	reported at startup as `LOST <id> <why>` and skipped. It is named rather than swallowed — a server that starts having silently lost a canister is worse than one that says so

An open question, not a claim: the armed global-timer deadline is host-side state and is not part of a snapshot, which carries memory, stable memory and the cycle balance only. No gate here covers whether a timer armed before a restart still fires afterwards. If it matters to your actor, re-arm from its initialisation path rather than assuming.

Actors created at runtime#

An actor installed over POST /install/<name>, or created by another actor, is durable from the moment it exists: its module and record are written beside the WAL, and the store that makes its messages journalled at all is opened then rather than at the next restart. On recovery these are adopted first, before the manifest's own actors are replayed, and re-installed with their original init argument — a log replayed against a default-initialised heap would rebuild a different actor without saying so.

Inter-actor calls need care that a single-actor design does not. A message that touched other actors snapshots them too, because their commits are real even when the ingress itself failed, and their own stores hold no journal of inter-actor deliveries — a snapshot is the only durable record their state has.

An upgrade forces a checkpoint. This is correctness, not housekeeping: the log holds messages that ran against the old code, so replaying them after a crash would run them against the new code and could land the actor somewhere it never was.

Recovery, and the alpha caveat#

Recovery runs on one thread, one actor at a time, and completes before the server reports ready. It is single-flight: there is no parallel replay, no incremental serve-while-you- recover, and no way to prioritise one actor. Everything that is not a health probe answers 503 starting: <note> until it finishes. That is why the listener binds before recovery — a long replay stays visible instead of looking like a hang — and why clients must wait on GET /health/ready rather than /health.

Practically: keep --snapshot-below above your actors' heap sizes if startup time matters, and stop with SIGTERM so the restart reads a snapshot instead of replaying a journal.

Backup and restore#

kill -TERM $MOXZID_PID && wait        # checkpoints every actor, exits 0
tar -czf backup.tar.gz -C /var/lib moxzi
# ... disaster ...
tar -xzf backup.tar.gz -C /var/lib
systemctl start moxzid                # state intact, still serving

scripts/moxzid_backup.sh executes exactly these steps, including wiping the directory in between. Copying the state directory while the server is running is not a supported backup: stop first, or accept that you may catch a torn write.

Next#

Operating moxzid — shutdown signals, service units, and metrics including moxzid_state_bytes.
Running a server — -s, --snapshot-below, and the readiness wait.
The HTTP API — /health/ready, /history/<actor>, and the install/upgrade verbs.