# Graceful shutdown and emergency checkpoints

**Date:** 2026-11-03
**Status:** Partially implemented (C1, C2, C3-policy)
**Incident:** `tenant-tarjetero-dev`, 2026-11-03 12:29 → 13:12 UTC (≈43 min outage, ≈7 h 45 m of work lost)

---

## 1. What happened (evidence)

| Time (UTC)        | Fact                                                                                                                   | Source                   |
| ----------------- | ---------------------------------------------------------------------------------------------------------------------- | ------------------------ |
| 05:26             | Last successful archive `33c023e2` (450 MB)                                                                            | R2 `stable-backup.json`  |
| 12:29:40          | Last healthy gateway `/health` (16 ms)                                                                                 | Workers Observability    |
| 12:33:43          | `listProcesses returned 65 processes (1 running)` — the only live process is a `start-openclaw.sh` wrapper; no gateway | Worker log               |
| 12:34:35          | Every browser WS closed `code=4000 inbound silence probe timeout`                                                      | Worker log               |
| ~12:39 → 12:42:32 | `sandbox.restoreBackup` timed out at `RESTORE_BACKUP_TIMEOUT_MS` (180 s) → `restore-target.json` `outcome: unknown`    | R2 + `persistence.phase` |
| 12:45 → 13:11     | Every boot refused with `RestoreInFlightError (restore-outcome-unknown)`; boot job fails; loading page loops           | Worker logs              |
| 13:03             | Operator `containers ssh` fails inside the Sandbox DO: `ContainerControlConnection.fetchUpgradeAttempt` → `AbortError` | Sandbox DO               |
| 13:11:19          | Rollout to `standard-3` (2 vCPU / 8 GiB / 16 GB) replaces the instance                                                 | Containers API           |
| 13:14:43          | New instance answers `sandbox.exec` in 13 ms (was 30 s timeouts)                                                       | Container dataset        |

Four independent defects combined:

1. **No memory signal.** `attachGatewayResources` only samples inside `/api/health*`,
   and nothing polls health while the gateway is healthy (the Control UI talks to
   the gateway directly). `gateway-resources` Analytics Engine rows stopped at
   2026-11-02 19:20 — the watchdog was blind for ~17 h, including the whole
   incident.
2. **No checkpoint on the way down.** A 0.5 vCPU / 4 GiB instance running two heavy
   agent sessions (xlsx parsing through an MCP, background `exec` jobs) exhausted the
   VM; there was no snapshot between 05:26 and the kill.
3. **"Unknown" read as "empty".** `containerHasState()` swallows every probe error
   and returns `false`, so an unreachable container is indistinguishable from a fresh
   one and the recovery path chose a **destructive restore** on a starved VM. The
   restore then timed out, leaving an `unknown` latch that blocked every later boot.
4. **The scheduled snapshot could not run.** The 12:35 `backup` job was interrupted at
   its `gate` step, and afterwards `restoreNotPending()` refused every snapshot because
   `restore-needed` was set — so the recovery path could not save the state it was
   about to destroy.

## 2. Data-loss budget (the number the policy must satisfy)

Today: **7 h 45 m lost**, discovered only because the instance had to be replaced.

| Environment | Max acceptable loss | Mechanism                                                                                                                  |
| ----------- | ------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| dev         | 15 min              | interval archive + emergency checkpoint                                                                                    |
| staging     | 15 min              | same                                                                                                                       |
| production  | 5 min               | same, plus Tier 1 container snapshots once the app moves to `durable_object` scheduling (`persistent-workspace-phase1.md`) |

Everything below is derived from that budget, not from a fixed cron.

## 3. Changes

### C1 — `containerHasState()` must distinguish _absent_ from _unknown_ (highest priority)

`apps/fouria/src/persistence.ts`

```ts
export type StatePresence = 'present' | 'absent' | 'unknown';
async function probeContainerState(sandbox: Sandbox): Promise<StatePresence>;
```

- `present` — any probe path reports `exists: true`.
- `absent` — **every** path returned a definite `{ exists: false }` (no throw).
- `unknown` — any probe threw (control channel down, timeout, `OperationInterruptedError`).

`restoreIfNeeded` restores only on `absent` (or an explicit `restore-needed`
signal). On `unknown` it throws a retryable `ContainerStateUnknownError`, which the
durable boot job retries with backoff instead of running a destructive restore.
Do **not** write `restore-needed` on the `unknown` path.

This single change would have prevented the entire 12:39 → 13:12 loop.

### C2 — Sample memory on the scheduler, not on health polls

- Move the `sampleGatewayResources()` + `maybeRestartGatewayForMemory()` call into the
  `ActivityScheduler` alarm (`apps/fouria/src/activity/scheduler.ts`), one sample per
  `GATEWAY_RESOURCE_SAMPLE_INTERVAL_MS` (60 s), independent of request traffic.
- Keep the health-path sample as a fast path (it already throttles), but pass
  `executionCtx` from `healthWsHandler` so the detached watchdog promise is not
  cancelled when the frame is sent (`routes/public.ts:719` calls
  `attachGatewayResources(snapshot, sandbox, c.env)` with no context).
- Record `rssBytes`/`threads`/`containerLimitBytes` on every sample, including when
  no gateway is running (`resources: null` + a `state` label), so a gap in the series
  is visibly "not sampling" rather than "healthy".

### C3 — Two-stage memory policy: checkpoint at 60 %, replace at 75 %

New config in `apps/fouria/src/config.ts`:

| Key                               | Value              | Meaning                                                    |
| --------------------------------- | ------------------ | ---------------------------------------------------------- |
| `GATEWAY_MEMORY_CHECKPOINT_RATIO` | `0.60`             | ≥ this fraction of the cgroup limit → emergency checkpoint |
| `GATEWAY_RSS_RESTART_RATIO`       | `0.75` (unchanged) | ≥ this → replace the gateway                               |
| `EMERGENCY_SNAPSHOT_TIMEOUT_MS`   | `180_000`          | bound for the checkpoint                                   |

Order is mandatory, and it is what makes the shutdown _graceful_:

1. `yellow` (60 %): enqueue a durable `snapshot` job with
   `trigger: 'memory-pressure'`; do not restart. Cooldown
   `EMERGENCY_SNAPSHOT_COOLDOWN_MS` (10 min) so a tenant hovering at 60 % does not
   snapshot continuously.
2. `red` (75 %): if the newest archive is older than the loss budget, run the
   checkpoint **first** (bounded by `EMERGENCY_SNAPSHOT_TIMEOUT_MS`), then
   `killGateway` + `ensureGateway({ skipRestore: true })` (today's
   `hardRestartGateway`). A failed or timed-out checkpoint must **not** block the
   restart — it is recorded as `persistence-snapshot { outcome: 'failed' }` and the
   restart proceeds (the alternative is losing everything).

State-machine function stays pure and unit-testable:

```ts
export function decideMemoryAction(
  sample,
  { limitBytes, newestArchiveAt, now },
): 'none' | 'checkpoint' | 'checkpoint-and-restart' | 'restart';
```

### C4 — The emergency checkpoint must not be gated by the restore latch

`scheduleBackupIfNeeded` starts with `restoreNotPending()`, which refused today's
snapshot. Add an explicit intent rather than weakening the gate:

```ts
scheduleBackupIfNeeded(sandbox, bucket, env, ctx, {
  trigger: 'memory-pressure',
  ignoreRestoreLatch: true, // only when probeContainerState() === 'present'
});
```

Guarded by `probeContainerState() === 'present'`, so a genuinely restoring tenant is
never snapshotted half-applied. Also bypasses the interval floor
(`BACKUP_INTERVAL_MINUTES`) — an emergency save is never "too soon".

### C5 — Checkpoint before any rollout or instance-type change

The `standard-3` rollout today had no checkpoint: the park path
(`checkpoint-park` / `resume`) exists but only Lerma's `@upgrade_pipeline` uses it.

- `Lerma.Cloudflare.Containers.change_instance_type/2` must call
  `POST /control { X-FourIA-Control: checkpoint-park }` first, exactly like the
  upgrade pipeline (fail-closed when the checkpoint is unavailable, exempt a sleeping
  instance), and `resume` after the rollout.
- The same applies to `POST /accounts/{id}/containers/applications/{app}/rollouts`
  issued by the rescue tool (`apps/fouria/src/diagnostics.ts`): refuse to roll out an
  instance whose newest archive is older than the loss budget unless
  `{"force": true}`.

### C6 — A crash artifact, because a VM kill leaves no trace

- `start-openclaw.sh` already mirrors the gateway stdout into
  `/tmp/gateway-stdout.log`. On abnormal exit (non-zero, signal, or missing after a
  control-plane probe) write `/tmp/gateway-crash.json`
  (`{ lastSeenAt, pid, rssBytes, threads, exitCode, signal, tail: last 200 lines }`)
  and upload it to `crashes/<ts>.json` in `BACKUP_BUCKET` on the next boot.
- Emit `gateway.process.gone { lastSeenAt, lastRssBytes, exitCode?, signal? }` so the
  absence of a process is an event, not an inference. Today the SDK produced **zero**
  exit records for the dead gateway.

### C7 — Stop the log/privacy leak while we are here

`[WS] Container -> Client (raw)` logs every frame with full model thinking, tool
arguments and tool results (≈48k events per 6 min during one agent session), and
Workers Logs persist them for 7 days — contradicting `public/articles/PRIVACY.md`.
Gate behind `DEBUG_WS_FRAMES` and log frame `type`/`method`/`seq` only.

## 4. Order of work

| #   | Change                         | Risk   | Value                                   |
| --- | ------------------------------ | ------ | --------------------------------------- |
| 1   | C1 absent/unknown probe        | low    | removes the destructive-restore loop    |
| 2   | C2 scheduler sampling          | low    | restores visibility + watchdog coverage |
| 3   | C3 + C4 emergency checkpoint   | medium | bounds data loss to the budget          |
| 4   | C6 crash artifact              | low    | makes the next incident diagnosable     |
| 5   | C5 park before rollout (Lerma) | medium | protects manual resizes                 |
| 6   | C7 frame-log gating            | low    | privacy + log volume                    |

## 5. Tests and docs required by the repo covenants

- `src/persistence.test.ts` — probe `present` / `absent` / `unknown`; `unknown` defers
  instead of restoring; no `restore-needed` on `unknown`.
- `src/gateway/memory.test.ts` — `decideMemoryAction` table (none / checkpoint /
  restart / cooldown / not-dev / instance-too-small).
- `src/activity/scheduler.test.ts` — the alarm samples and enqueues the checkpoint;
  a failed checkpoint does not block the restart.
- `src/routes/public.test.ts` — WS path passes `executionCtx`.
- `src/diagnostics.test.ts` — rollout refused when the newest archive is stale.
- Corpus: `test/business-rules/operational-modes.txt`, `cost-tracking.txt`
  (`persistence-snapshot { trigger: 'memory-pressure' }`).
- Docs: `apps/fouria/docs/WORKER-DEVELOPMENT.md`, `docs/plans/sandbox-memory-usage.md`,
  `docs/ENV-VARIABLES.md` + `.dev.vars.example` for the new keys.
- Changeset: `bun run changeset --patch fouria`.

## 6. Out of scope

- Tier 1 container snapshots stay gated on the Phase 1 measurement
  (`persistent-workspace-phase1.md`); they need the tenant app on
  `scheduling_policy: durable_object`, which is a separate migration.
- Restoring the work lost on 2026-11-03: not recoverable — `33c023e2` (05:26) is the
  baseline. The fixes above exist so the next occurrence costs minutes, not hours.

## 7. Implementation status

Landed in this change:

- **C1** — `probeContainerState()` returns `present | absent | unknown`
  (`apps/fouria/src/persistence.ts`). `restoreIfNeeded` restores only on
  `absent`, and throws the retryable `ContainerStateUnknownError` on `unknown`,
  without writing the `restore-needed` latch. A runtime without an `exists()`
  probe keeps the historical "assume empty" reading so it can still restore.
- **C2** — `checkGatewayMemoryHealth()` is the single sampler for both the
  request path and the durable `ActivityScheduler` alarm
  (`runMemoryWatch()`), so a tenant is observed even when no browser polls
  `/api/health`. Every fresh sample now writes a `gateway-memory-sample` row
  carrying its `state` (`running` / `absent`) and the container limit, so a gap
  means "nobody sampled" instead of "nothing happened". The health WebSocket
  handler now passes its `executionCtx`, which previously let the detached
  watchdog promise be cancelled with the frame.
- **C3 (policy only)** — `decideMemoryAction()` implements the two-stage ladder
  (60 % checkpoint / 75 % restart, `checkpoint-and-restart` when the newest
  archive is older than `GATEWAY_MEMORY_LOSS_BUDGET_MS`), and the alarm enqueues
  the checkpoint as a `trigger: 'memory-pressure'` snapshot job before the
  restart. `maybeRestartGatewayForMemory` delegates to the same policy so the
  thresholds cannot drift; the request path still restarts only.

Still open: **C4** (bypassing the restore latch for a proven-present container),
**C5** (park before a manual rollout from Lerma), **C6** (crash artifact +
`gateway.process.gone`) and **C7** (frame-log gating).
