F FourIA staging 2026.10.29 GitHub ↗

Graceful shutdown and emergency checkpoints

On this page

Date: 2026-11-03 Status: Partially implemented (C1, C2, C3-policy) Incident: tenant-tarjetero-dev, 2026-11-03 12:29 → 13:12 UTC (≈43 min outage, ≈7 h 45 m of work lost)


1. What happened (evidence)

Time (UTC)FactSource
05:26Last successful archive 33c023e2 (450 MB)R2 stable-backup.json
12:29:40Last healthy gateway /health (16 ms)Workers Observability
12:33:43listProcesses returned 65 processes (1 running) — the only live process is a start-openclaw.sh wrapper; no gatewayWorker log
12:34:35Every browser WS closed code=4000 inbound silence probe timeoutWorker log
~12:39 → 12:42:32sandbox.restoreBackup timed out at RESTORE_BACKUP_TIMEOUT_MS (180 s) → restore-target.json outcome: unknownR2 + persistence.phase
12:45 → 13:11Every boot refused with RestoreInFlightError (restore-outcome-unknown); boot job fails; loading page loopsWorker logs
13:03Operator containers ssh fails inside the Sandbox DO: ContainerControlConnection.fetchUpgradeAttempt → AbortErrorSandbox DO
13:11:19Rollout to standard-3 (2 vCPU / 8 GiB / 16 GB) replaces the instanceContainers API
13:14:43New instance answers sandbox.exec in 13 ms (was 30 s timeouts)Container dataset

Four independent defects combined:

  1. No memory signal. attachGatewayResources only samples inside /api/health*, and nothing polls health while the gateway is healthy (the Control UI talks to the gateway directly). gateway-resources Analytics Engine rows stopped at 2026-11-02 19:20 — the watchdog was blind for ~17 h, including the whole incident.
  2. No checkpoint on the way down. A 0.5 vCPU / 4 GiB instance running two heavy agent sessions (xlsx parsing through an MCP, background exec jobs) exhausted the VM; there was no snapshot between 05:26 and the kill.
  3. “Unknown” read as “empty”. containerHasState() swallows every probe error and returns false, so an unreachable container is indistinguishable from a fresh one and the recovery path chose a destructive restore on a starved VM. The restore then timed out, leaving an unknown latch that blocked every later boot.
  4. The scheduled snapshot could not run. The 12:35 backup job was interrupted at its gate step, and afterwards restoreNotPending() refused every snapshot because restore-needed was set — so the recovery path could not save the state it was about to destroy.

2. Data-loss budget (the number the policy must satisfy)

Today: 7 h 45 m lost, discovered only because the instance had to be replaced.

EnvironmentMax acceptable lossMechanism
dev15 mininterval archive + emergency checkpoint
staging15 minsame
production5 minsame, plus Tier 1 container snapshots once the app moves to durable_object scheduling (persistent-workspace-phase1.md)

Everything below is derived from that budget, not from a fixed cron.

3. Changes

C1 — containerHasState() must distinguish absent from unknown (highest priority)

apps/fouria/src/persistence.ts

export type StatePresence = 'present' | 'absent' | 'unknown';
async function probeContainerState(sandbox: Sandbox): Promise<StatePresence>;
  • present — any probe path reports exists: true.
  • absent — every path returned a definite { exists: false } (no throw).
  • unknown — any probe threw (control channel down, timeout, OperationInterruptedError).

restoreIfNeeded restores only on absent (or an explicit restore-needed signal). On unknown it throws a retryable ContainerStateUnknownError, which the durable boot job retries with backoff instead of running a destructive restore. Do not write restore-needed on the unknown path.

This single change would have prevented the entire 12:39 → 13:12 loop.

C2 — Sample memory on the scheduler, not on health polls

  • Move the sampleGatewayResources() + maybeRestartGatewayForMemory() call into the ActivityScheduler alarm (apps/fouria/src/activity/scheduler.ts), one sample per GATEWAY_RESOURCE_SAMPLE_INTERVAL_MS (60 s), independent of request traffic.
  • Keep the health-path sample as a fast path (it already throttles), but pass executionCtx from healthWsHandler so the detached watchdog promise is not cancelled when the frame is sent (routes/public.ts:719 calls attachGatewayResources(snapshot, sandbox, c.env) with no context).
  • Record rssBytes/threads/containerLimitBytes on every sample, including when no gateway is running (resources: null + a state label), so a gap in the series is visibly “not sampling” rather than “healthy”.

C3 — Two-stage memory policy: checkpoint at 60 %, replace at 75 %

New config in apps/fouria/src/config.ts:

KeyValueMeaning
GATEWAY_MEMORY_CHECKPOINT_RATIO0.60≥ this fraction of the cgroup limit → emergency checkpoint
GATEWAY_RSS_RESTART_RATIO0.75 (unchanged)≥ this → replace the gateway
EMERGENCY_SNAPSHOT_TIMEOUT_MS180_000bound for the checkpoint

Order is mandatory, and it is what makes the shutdown graceful:

  1. yellow (60 %): enqueue a durable snapshot job with trigger: 'memory-pressure'; do not restart. Cooldown EMERGENCY_SNAPSHOT_COOLDOWN_MS (10 min) so a tenant hovering at 60 % does not snapshot continuously.
  2. red (75 %): if the newest archive is older than the loss budget, run the checkpoint first (bounded by EMERGENCY_SNAPSHOT_TIMEOUT_MS), then killGateway + ensureGateway({ skipRestore: true }) (today’s hardRestartGateway). A failed or timed-out checkpoint must not block the restart — it is recorded as persistence-snapshot { outcome: 'failed' } and the restart proceeds (the alternative is losing everything).

State-machine function stays pure and unit-testable:

export function decideMemoryAction(
  sample,
  { limitBytes, newestArchiveAt, now },
): 'none' | 'checkpoint' | 'checkpoint-and-restart' | 'restart';

C4 — The emergency checkpoint must not be gated by the restore latch

scheduleBackupIfNeeded starts with restoreNotPending(), which refused today’s snapshot. Add an explicit intent rather than weakening the gate:

scheduleBackupIfNeeded(sandbox, bucket, env, ctx, {
  trigger: 'memory-pressure',
  ignoreRestoreLatch: true, // only when probeContainerState() === 'present'
});

Guarded by probeContainerState() === 'present', so a genuinely restoring tenant is never snapshotted half-applied. Also bypasses the interval floor (BACKUP_INTERVAL_MINUTES) — an emergency save is never “too soon”.

C5 — Checkpoint before any rollout or instance-type change

The standard-3 rollout today had no checkpoint: the park path (checkpoint-park / resume) exists but only Lerma’s @upgrade_pipeline uses it.

  • Lerma.Cloudflare.Containers.change_instance_type/2 must call POST /control { X-FourIA-Control: checkpoint-park } first, exactly like the upgrade pipeline (fail-closed when the checkpoint is unavailable, exempt a sleeping instance), and resume after the rollout.
  • The same applies to POST /accounts/{id}/containers/applications/{app}/rollouts issued by the rescue tool (apps/fouria/src/diagnostics.ts): refuse to roll out an instance whose newest archive is older than the loss budget unless {"force": true}.

C6 — A crash artifact, because a VM kill leaves no trace

  • start-openclaw.sh already mirrors the gateway stdout into /tmp/gateway-stdout.log. On abnormal exit (non-zero, signal, or missing after a control-plane probe) write /tmp/gateway-crash.json ({ lastSeenAt, pid, rssBytes, threads, exitCode, signal, tail: last 200 lines }) and upload it to crashes/<ts>.json in BACKUP_BUCKET on the next boot.
  • Emit gateway.process.gone { lastSeenAt, lastRssBytes, exitCode?, signal? } so the absence of a process is an event, not an inference. Today the SDK produced zero exit records for the dead gateway.

C7 — Stop the log/privacy leak while we are here

[WS] Container -> Client (raw) logs every frame with full model thinking, tool arguments and tool results (≈48k events per 6 min during one agent session), and Workers Logs persist them for 7 days — contradicting public/articles/PRIVACY.md. Gate behind DEBUG_WS_FRAMES and log frame type/method/seq only.

4. Order of work

#ChangeRiskValue
1C1 absent/unknown probelowremoves the destructive-restore loop
2C2 scheduler samplinglowrestores visibility + watchdog coverage
3C3 + C4 emergency checkpointmediumbounds data loss to the budget
4C6 crash artifactlowmakes the next incident diagnosable
5C5 park before rollout (Lerma)mediumprotects manual resizes
6C7 frame-log gatinglowprivacy + log volume

5. Tests and docs required by the repo covenants

  • src/persistence.test.ts — probe present / absent / unknown; unknown defers instead of restoring; no restore-needed on unknown.
  • src/gateway/memory.test.ts — decideMemoryAction table (none / checkpoint / restart / cooldown / not-dev / instance-too-small).
  • src/activity/scheduler.test.ts — the alarm samples and enqueues the checkpoint; a failed checkpoint does not block the restart.
  • src/routes/public.test.ts — WS path passes executionCtx.
  • src/diagnostics.test.ts — rollout refused when the newest archive is stale.
  • Corpus: test/business-rules/operational-modes.txt, cost-tracking.txt (persistence-snapshot { trigger: 'memory-pressure' }).
  • Docs: apps/fouria/docs/WORKER-DEVELOPMENT.md, docs/plans/sandbox-memory-usage.md, docs/ENV-VARIABLES.md + .dev.vars.example for the new keys.
  • Changeset: bun run changeset --patch fouria.

6. Out of scope

  • Tier 1 container snapshots stay gated on the Phase 1 measurement (persistent-workspace-phase1.md); they need the tenant app on scheduling_policy: durable_object, which is a separate migration.
  • Restoring the work lost on 2026-11-03: not recoverable — 33c023e2 (05:26) is the baseline. The fixes above exist so the next occurrence costs minutes, not hours.

7. Implementation status

Landed in this change:

  • C1 — probeContainerState() returns present | absent | unknown (apps/fouria/src/persistence.ts). restoreIfNeeded restores only on absent, and throws the retryable ContainerStateUnknownError on unknown, without writing the restore-needed latch. A runtime without an exists() probe keeps the historical “assume empty” reading so it can still restore.
  • C2 — checkGatewayMemoryHealth() is the single sampler for both the request path and the durable ActivityScheduler alarm (runMemoryWatch()), so a tenant is observed even when no browser polls /api/health. Every fresh sample now writes a gateway-memory-sample row carrying its state (running / absent) and the container limit, so a gap means “nobody sampled” instead of “nothing happened”. The health WebSocket handler now passes its executionCtx, which previously let the detached watchdog promise be cancelled with the frame.
  • C3 (policy only) — decideMemoryAction() implements the two-stage ladder (60 % checkpoint / 75 % restart, checkpoint-and-restart when the newest archive is older than GATEWAY_MEMORY_LOSS_BUDGET_MS), and the alarm enqueues the checkpoint as a trigger: 'memory-pressure' snapshot job before the restart. maybeRestartGatewayForMemory delegates to the same policy so the thresholds cannot drift; the request path still restarts only.

Still open: C4 (bypassing the restore latch for a proven-present container), C5 (park before a manual rollout from Lerma), C6 (crash artifact + gateway.process.gone) and C7 (frame-log gating).