Resumabilityalpha

Why is every stage checkpointed, and why is on-chain compile time message count rather than instructions?

Resumability means every stage of the compile can stop mid-way, store what it has on the job record, return, and be resumed by the next message with identical results. It is not an optimization; without it the on-chain compiler cannot exist at all.

The constraint that forces it: the Internet Computer kills any update message that exceeds 40 billion instructions — Canister exceeded the limit of 40000000000 instructions for single message execution, error IC0522 — and a compile is far larger than that. A 7.6 MB compile is roughly 451 billion reported instructions. It has to be at least a dozen messages by arithmetic, and in practice thousands.

Checkpointing between stages is not enough#

The obvious design gives each pipeline stage its own message. That fails for two separate reasons, both measured:

• A single stage exceeds the cap. One method in the compiler's own source compiles to over 100 billion instructions inside one synchronous codegen build. No boundary between stages helps.
• The garbage collector only runs at message boundaries. Each IR pass rebuilds the whole IR, so the previous IR becomes garbage the moment the pass starts. Committed wasm memory follows the heap peak and never shrinks. Letting seven passes share one message drove the heap to 3.79 GB against a ~929 MB live set — and the end-of-message collection over that heap is what blew the cap when the pass returned.

So checkpointing happens at three depths.

The third is the interesting one. Codegen's expression descent cannot yield — colouring it async* overflows the engine's wasm stack on the compiler's giant functions — so instead compileExpH increments a counter on entry, reads the real performance counter every 64 entries, and latches fuelExhausted without aborting. The build completes (the bytes are correct; the message is merely over budget). At the next async* seam, buildWithFuel takes an O(1) environment snapshot, and if fuel was exhausted it restores the environment, replays an undo journal LIFO, crosses a real message boundary, and re-runs the build deterministically — identical reservations, identical pool order, identical bytes.

That determinism was proven rather than argued: a forced-rollback mode made every wrapped build snapshot, roll back, replay and re-run — 78 and 80 retries on two probe programs, and both came out byte-identical to the reference. The forcing was then removed, and setFuelForceRollback(b) retained as an audit hook. A build that still exceeds budget after a fresh-budget replay is accepted and logged as FUEL-OVERRUN <name>, so any such build names itself instead of hiding.

The governors#

These are the knobs that decide where a message ends. Every one of them is a measured number, and every one of them was set against a specific failure.

Two of these have counter-intuitive shapes worth stating. Tightening the heap valve does not monotonically help: at 384 MiB the peak was 2,496 MB, at 128 MiB it was 2,304 MB, at 64 MiB it was 2,773 MB — because every boundary retains continuation state, so past the optimum more collection costs more memory than it frees. And onchainStepUnitBudget saturates: 8 units gives 2,817 steps, 24 gives 1,868, 64 gives 1,877. Past 24 the heap-growth valve binds instead.

Why message count is the clock#

On-chain wall-clock time is rounds × block time, and rounds are set by our own message count, not by instructions. The other candidate — the 75 MiB-per-round heap-delta rate limit — sets only a floor, and that floor is far below where a real compile sits.

For evm_engine (671 files):

Three changes got that, none of them about the IC's limits: the codegen "baseline beat" ladder was triggering on an absolute heap size that sat below the live set, so 3,658 of ~7,000 messages did nothing at all; the conclude driver returned to the dispatcher after every sub-step, each using ~0.14 % of the message cap, and now batches; and that batching was only safe after stripping the async* colour from codegen, because await* in a loop nests a CPS continuation per iteration and overflows the on-chain stack while passing every off-chain gate.

The planning rule: compile time ≈ max(dirty floor, message count) × block time, and message count is yours to control — bounded by the fact that the collector only runs between messages, so messages cannot grow so fat that garbage threatens the 6 GiB ceiling.

What you cannot measure this way#

instructionsUsed is not a price. performance_counter reports only what executes inside the canister's own wasm; the IC also bills GC, memory operations and scheduling it cannot see. Measured against real spend it under-reports by 2.3×–3.4×, and the ratio is not constant — heavier phases bill more per reported instruction. Storage and per-message overhead were both ruled out arithmetically as explanations.

Two further traps: performance_counter(1) resets on every inner await async {}, so a budget measured from a function's entry silently fails to accumulate across any unit that yields internally; and the off-chain performance counter is a stub that always grows, so an off-chain probe about instruction budgets returns a confident, meaningless answer. Numbers that matter here were measured on a replica.

Things you cannot do#

Next#

• The compile pipeline — the stages these budgets are cutting up.
• Self-hosting and fixed points — the workload that set every number on this page.
• Byte-identical output — the gate that proves moving a message boundary moved nothing else.