# Durable-agents rules audit — FourIA tenant runtime (second pass)

**Status:** P0, F3 and F5 shipped; the Next slice (**F7, F8, F9**) shipped
2026-10-24. Every finding is closed except the two deliberate deferrals: the
`BackupWatch` fold into `ActivityScheduler` (R4) and the fleet aggregate
(R14 / companion A5). Each finding carries its own status line.
**Date:** 2026-10-22
**Scope:** `apps/fouria` (tenant User Worker + its Durable Objects, container lifecycle,
persistence) and `workers/fouria-dispatch` (fleet cron, routing).
**Rubric:** [`skills/cloudflare-durable-agents/SKILL.md`](https://github.com/0xCAB0/fouria/blob/develop/skills/cloudflare-durable-agents/SKILL.md)
— R1–R15, the §5 failure matrix, and the §6 checklist.
**Companion:** [`docs/plans/durable-jobs-availability-audit.md`](/staging/llms/platform/reference/plans/durable-jobs-availability-audit/index.md)
(same date) already catalogues the job-lifecycle, availability and scale findings as
J1–J5 / A1–A5 / S1–S4 with a P0–P2 roadmap. This is a deliberately different pass:
it scores the implementation **against the rules**, re-derives the same defects
where the rule text sharpens the failure mode, and adds what the first pass missed
(catalog I/O on read paths, the unlocked catalog handle, timer-only lease renewal,
watch cadence, per-event status writes, dead deploy surface).

Severity scale used below: **High** = silent correctness loss or fleet-wide outage
vector; **Medium** = avoidable cost, wasted work, or a latent correctness bug;
**Low** = hygiene with a clear cheap fix.

---

## 1. Scorecard

| Rule                                            | Verdict     | Basis                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| ----------------------------------------------- | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **R1** One object per unit of coordination      | **Pass**    | Per-tenant Worker scripts with tenant-named objects (`getActivityScheduler`, `getSandbox(env, 'openclaw')`). No platform-wide singleton. `BackupWatch` and `TelegramIngress` add objects per tenant, never shared ones.                                                                                                                                                                                                                                                                                   |
| **R2** SQLite is the truth                      | **Pass**    | Job records are persisted before the caller is acknowledged and executed by the alarm (`activity/scheduler.ts:178-182`). State is read from storage on every alarm, not from instance fields.                                                                                                                                                                                                                                                                                                             |
| **R3** Never do the work in the request         | **Pass**    | Default admin snapshots enqueue durable jobs; `?inline=1` is the explicit synchronous escape hatch and no-ACTIVITY environments retain an inline fallback (**F5 shipped**). `getBackupHistory` is pure; reconciliation is explicit/durable (**F3 shipped**).                                                                                                                                                                                                                                              |
| **R4** One alarm per object → alarm queue       | **Partial** | `scheduleCombinedAlarm` arms the minimum due across boot/backup/maintenance/activity and clears on `null` (`activity/scheduler.ts:834-846`, `activity/alarm.ts:113-133`) — textbook. The fleet sweep is now paginated and queue-backed (**F1/F2 shipped**), and `BackupWatch` no longer execs into the container while a snapshot owns the slot or the floor has not elapsed (**F7 shipped**). Remaining: `BackupWatch` is still a second per-tenant alarm; the fold into `ActivityScheduler` stays Next. |
| **R5** Alarms are at-least-once                 | **Pass**    | Job steps retry with a cursor; lock release is an etag-CAS tombstone; busy slots are rejected explicitly. The catalog handle is an etag CAS (**F4**), and snapshot leases renew at phase boundaries plus a serialized heartbeat (**F5**).                                                                                                                                                                                                                                                                 |
| **R6** Stream, never buffer                     | **Pass**    | Restore/encrypt/decrypt are streamed through `FixedLengthStream` (memory-bounded), and R2 enumeration now follows `truncated`/`cursor` through `listAllObjects` (**F6 shipped**).                                                                                                                                                                                                                                                                                                                         |
| **R7** `blockConcurrencyWhile` constructor-only | **Pass**    | Only used by the unreleased `SandboxV1` container lifecycle (`sandbox-v1.ts:213`); no hot-path use.                                                                                                                                                                                                                                                                                                                                                                                                       |
| **R8** Storage awaits interleave                | **Pass**    | `mutatePersistenceStore` re-reads the etag and re-applies the mutation on conflict, so `backup-handle.json` can no longer be lost-updated across awaits (**F4 shipped**).                                                                                                                                                                                                                                                                                                                                 |
| **R9** Assume death; add a watchdog             | **Pass**    | Real watchdogs exist (stuck start, boot-job lease, per-step deadline), and the dispatch sweep now paginates, queues and jitters deliveries instead of running one unbounded pass (**F1/F2 shipped**).                                                                                                                                                                                                                                                                                                     |
| **R10** Budget the turn                         | **Pass**    | Exactly one step per alarm with `STEP_LEASE_MS` / `STEP_DEADLINE_MS` / `STEP_CONTINUE_DELAY_MS` (`activity/job-steps.ts:29-52`); cadence floor is `2 ×` the last snapshot duration.                                                                                                                                                                                                                                                                                                                       |
| **R11** Hibernate WebSockets                    | **N/A**     | Sockets are Worker↔container proxies, not DO-hosted. The polling-loop cost is real but already owned by A5 / §S3 in the companion doc.                                                                                                                                                                                                                                                                                                                                                                    |
| **R12** Model failover                          | **N/A**     | OpenClaw owns provider routing; this layer never calls a model.                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| **R13** Sub-agents talk in natural language     | **N/A**     | No sub-agent runtime in the Worker.                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| **R14** Observability                           | **Partial** | Structured events are strong, but there is no fleet-level aggregate (A5). Telegram ingress now flushes its status in one coalesced write per burst instead of rewriting it per webhook (**F8 shipped**).                                                                                                                                                                                                                                                                                                  |
| **R15** Tenant boundaries are explicit          | **Pass**    | `sandbox-tenant.ts` resolves the tenant from the per-tenant env instead of `ctx.containerId`, so the R2 bridges cannot cross tenants. The unused `SandboxV1` deploy surface is removed (**F9 shipped**), leaving the class + `v7` migration for the cutover.                                                                                                                                                                                                                                              |

Ten of fifteen rules now pass (R1, R2, R3, R5, R6, R7, R8, R9, R10, R15), two remain
partial (R4, R14) and three are not applicable (R11–R13). **No rule is a
blanket failure** — the P0 slice closed the three rule gaps (R6, R8, R9), and the
next slice (F7/F8/F9) removed the remaining wasted work behind the two partials:
`BackupWatch` no longer probes a container it cannot snapshot, and ingress status
writes are coalesced. R4 stays partial only because the watch is still a second
alarm (the fold is Next); R14 only because there is no fleet aggregate (A5).

---

## 2. Findings

### F1 — Fleet fan-out: silently truncated at 1,000 tenants, then unbounded

**Rules:** R9 (watchdog / at scale), R3 (work outside the request), R1 (one object per unit).
**Severity:** High.
**Status:** ✅ Shipped — paginated sweep + `CONTROL_QUEUE` fan-out (see below).
**Existing ID:** §S2 / Finding S1 / **P0 #2** in the companion audit — same defect. This
entry adds the rule basis and the concrete queue shape; do not file it twice.

**Evidence** — `workers/fouria-dispatch/src/index.ts:869-903`:

- `:870` — `env.TENANT_INDEX.list({ prefix: 'slug:' })` with **no `cursor` loop and no
  `list_complete` check**. KV returns at most 1,000 keys per page.
- `:871-877` — `Promise.all` over one KV `get` per key, unbounded.
- `:885-888` — `for (… ) ctx.waitUntil(forwardTenantControl(…, 'backup'))`, i.e. one
  concurrent fetch per tenant, also unbounded.
- Production fires this every 15 minutes (`workers/fouria-dispatch/wrangler.jsonc:34`);
  staging hourly (`:73`).

**Impact.** Past 1,000 tenants the remainder never receive a scheduled backup and
nothing errors — a silent correctness loss that only surfaces at restore time.
Before that, the invocation runs ~2 metered operations per tenant (one KV `get`,
one dispatch `fetch`), so the 1,000-per-invocation subrequest budget (and the
separate KV-operations budget, also 1,000) is exhausted around ~500 tenants. The
cron's `waitUntil` fan-out then has to settle 1,000 concurrent fetches inside the
cron wall. This is the same cancellation class that killed request-scoped restores.

**Fix.** Paginate (`while (!listed.list_complete)`), then enqueue one message per
tenant onto the existing `TASKS` queue (`apps/fouria/wrangler.jsonc:45-47`, consumer
at `apps/fouria/src/index.ts:1592`) and let the queue consumer call
`forwardTenantControl`. Give the consumer `max_concurrency` and a bounded
`max_batch_size`; deliveries are already idempotent because the tenant enqueues a
durable job and answers `202`. **Effort:** M.

**Shipped.** `collectTenantSlugs` (`workers/fouria-dispatch/src/index.ts`) follows
`list_complete` / `cursor` and dedupes keys, and the sweep enqueues through its own
`CONTROL_QUEUE` producer (100 messages per `sendBatch`) rather than the tenant
`TASKS` queue, which would have coupled fleet control work to tenant automation
traffic. The consumer re-resolves `slug:<slug>` at delivery time, skips
missing/frozen/nameless tenants, and `ack`s `delivered`/`skipped` while `retry()`ing
`failed`. When the binding is absent (local/older environments), a bounded
concurrency-8 direct path preserves the old behavior. Queue definitions live per
environment in `wrangler.jsonc`; `deploy-dispatch.yml` creates them idempotently.

---

### F2 — Every tenant's scheduled work starts on the same wall-clock minute

**Rules:** R9 (fleet blast radius), R10 (budget the turn).
**Severity:** High (blast radius), cheap to fix.
**Status:** ✅ Shipped — deterministic per-slug `delaySeconds` on the queue message.
**Existing ID:** new; adjacent to P1 #9 (cadence cannot be met on large tenants).

**Evidence.**

- The dispatch cron is a fixed `*/15 * * * *` (`workers/fouria-dispatch/wrangler.jsonc:34`).
  Each delivery calls `/control` → `enqueueBackupJob` → `scheduleCombinedAlarm`
  (`apps/fouria/src/index.ts:578-586`, `activity/scheduler.ts:178-182`), which arms the
  alarm **immediately**.
- The interval gate is 5 minutes by default (`config.ts:252`) and otherwise
  `2 ×` the last snapshot duration, so a 15-minute tick clears it for most tenants on
  most ticks: each size class of tenant snapshots in lockstep.
- A snapshot is a multi-minute container stage plus `createBackup` (~260 s observed)
  plus encryption.
- Second fixed cadence: every tenant Worker runs `*/2 * * * *` (`apps/fouria/wrangler.jsonc:73`),
  i.e. 720 invocations per tenant per day, for the billable drain.

**Impact.** Two distinct problems. (1) **Synchronized load**: at minute 0/15/30/45
every tenant's container stage, R2 upload and encrypt start together — a self-inflicted
thundering herd whose peak is proportional to fleet size. (2) **Shared blast radius**:
a bad snapshot step or a Cloudflare incident in that window hits every tenant at once
instead of degrading gradually. Neither is visible in per-tenant metrics.

**Fix.** Derive a stable per-tenant offset from the slug (e.g. `fnv1a(slug) % interval`)
and either (a) have the queue consumer delay each message by that offset before
`forwardTenantControl`, or (b) compute the next due time from
`controller.scheduledTime + offset` and let the tenant's interval gate absorb it.
Prefer (a): the queue owns the delay, the tenant stays stateless. **Effort:** S.

**Shipped.** `fanoutJitterSeconds` derives a stable FNV-1a offset from the slug into
a 300 s window (overridable with `CONTROL_FANOUT_JITTER_SECONDS`, `0` disables,
capped at 86400) and sets it as the message's `delaySeconds`. The offset is
deterministic, so a redelivery keeps the same schedule, and it only applies on the
queue path — the direct fallback delivers immediately. Over a 15-minute tick the
fleet now spreads across five minutes instead of one.

The second fixed cadence noted above (`*/2 * * * *` per tenant Worker for the
billable drain) is a separate trigger and is out of this slice.

---

### F3 — A dashboard `GET` reconciles, prunes and writes the catalog

**Rules:** R3 (never do the work in the request), R5 (idempotency / side effects),
R14 (reads should be cheap and observable).
**Severity:** Medium (read amplification + a GET that deletes objects).
**Status:** ✅ Shipped — pure read + explicit `reconcileBackupCatalog` (see below).
**Existing ID:** new; §S3 covers health/WS polling, not the backups read path.

**Evidence.**

- `getBackupHistory` calls `reconcileOrphanedBackups` on every invocation
  (`persistence.ts:3121`). The same reconcile also runs before the interval gate in
  `scheduleBackupIfNeeded` (`:3567`) and in `restoreIfNeeded` (`:1393`).
- `reconcileOrphanedBackups` (`persistence.ts:956`) does `bucket.list({ limit: 1000 })`
  (`:970`), reads `meta.json` per candidate archive, **deletes over-cap archives on
  read** (`:1056-1059` → `cleanupOldBackups`), and may write the handle back
  (`:1072`).
- The prune fan-out is unbounded: `Promise.all(toRemove.map((entry) =>
  deleteBackupObjects(bucket, entry.id)))` (`persistence.ts:1988`), where each delete
  is a sequence of up to four R2 deletes (`:1952-1965`).
- `GET /api/admin/storage/backups` is that read (`routes/api/storage.ts:203-205`).
  `BackupsPage` calls it on load (`client/pages/BackupsPage.tsx:66`), and the history
  panel then calls it every 5 s for up to 5 minutes while an async snapshot/restore is
  pending (`client/components/BackupHistoryPanel.tsx:15-17`, `:127-139`) — ~60 full
  reconciles per operation.

**Impact.** A read endpoint performs retention deletes and writes, so an admin
opening a page can change storage. Cost: one R2 list plus N meta reads per poll, per
open tab, with no caching. Correctness: two concurrent reads can both prune and both
write the handle (see **F4**). Observability: a failed delete surfaces as a failed
`GET`, which reads like a dashboard bug rather than a storage fault.

**Fix.** Split the read from the repair: `getBackupHistory` returns the persisted
catalog (`getPersistenceStore`) and never mutates. Move reconcile + prune into the
existing durable maintenance/backup step (it already runs on a lease and already has a
cursor), or into an explicit `POST /api/admin/storage/reconcile` used by the operator
and by the scheduled job. Cache the panel response for the poll window instead of
re-listing on every tick. **Effort:** M.

**Shipped.** `getBackupHistory` is now a **pure read** — `getPersistenceStore` +
`getStableBackup` + `isRecoveryHeld` in parallel, no list, write or delete. The write
path is explicit: `reconcileCatalogState` (adoption + retention prune inside the
etag-CAS mutation, reporting the ids it changed) → `repairBackupCatalog` (sweep +
in-place encryption) → the refreshed history, exposed as `reconcileBackupCatalog`.
The durable `repair` step calls it **before** the `gate` step of every scheduled
snapshot, so the catalog still self-heals after a killed invocation (issue #344);
`scheduleBackupIfNeeded` keeps its inline reconcile and restore keeps its own.
Operators get `POST /api/admin/storage/reconcile` (ADMIN/ROOT) as the manual lever,
and the `debug/reprovision` guard reconciles before deciding there is nothing
restorable, so a crashed upload cannot be mistaken for an unrecoverable tenant. The
history panel now polls the durable job (`getBackupJobStatus`) while a snapshot is in
flight instead of re-listing the catalog every 5 s, falling back to the history diff
only when the server returns no job id. R3 stays **Partial** only because of the
synchronous admin snapshot (F5).

---

### F4 — `backup-handle.json` is written without compare-and-set

**Rules:** R8 (interleaving across awaits), R5 (idempotent alarms), R2 (storage is truth).
**Severity:** Medium (a committed archive can silently disappear from the catalog).
**Status:** ✅ Shipped — `mutatePersistenceStore` etag CAS.
**Existing ID:** new.

**Evidence.**

- `savePersistenceStore` is an unconditional `bucket.put(HANDLE_KEY, …)`
  (`persistence.ts:611-613`). There is no `onlyIf`, no version, no retry.
- It is called from a read path (`:1072`) and from every write path: restore
  signal/latch (`:1183`, `:1546`), snapshot commit (`:2792`, `:2878`, `:2917`), and the
  scheduler timestamp (`:3652`).
- The lock path already does this correctly, with etag CAS and a tombstone
  (`persistence.ts:3249`, `:3252`, `:3268`, `:3302`).

**Impact.** Two writers that both read the handle and then put it back race: last
writer wins. The realistic sequence is a polled `GET` (reconcile) overlapping a
snapshot finalize — the finalize adds `backups/<id>` and the poll writes back the
pre-snapshot store, so a successfully uploaded archive is no longer in the catalog
(the archive objects remain, so **F3's** reconcile can adopt it later, but
`stable/config` and the retention decision were made against a stale list). During a
restore the same race can drop a `restore-config-pending` marker.

**Fix.** `put(..., { onlyIf: { etagMatches: read.etag } })` with a small read-retry
loop, mirroring `releaseLock` (`persistence.ts:3302`); expose `savePersistenceStore`
as `mutatePersistenceStore(fn)` so callers cannot forget. **Effort:** S.

**Shipped.** `mutatePersistenceStore` re-reads the handle and re-applies the caller's
mutation against the fresh catalog on conflict (5 attempts, then
`PersistenceStoreConflictError`) — not a blind merge, because prune/removal
mutations must legitimately shrink the catalog. Writes use `etagMatches` when an
etag exists and `etagDoesNotMatch: '*'` for a create-only handle; a mutation may
return `null` to skip the write. Reconcile, snapshot finalization, the retention
prune and the schedule-timestamp commit now all run inside that CAS. The restore
markers were left on their existing paths in this slice.

---

### F5 — The restore/backup lease outlives its renewal mechanism

**Rules:** R5 (at-least-once, but only one owner), R7 (no long blocking work on the hot
path), R10 (budget).
**Severity:** Medium.
**Status:** ✅ Shipped — 12-minute lease, serialized phase/heartbeat renewals,
structured renewal events, and durable-by-default admin snapshots.
**Existing ID:** adjacent to J4 (lease/orphan handling) but distinct.

**Evidence.**

- The former 60 s lease and timer-only heartbeat could expire during long archive
  operations; the lease now derives its 12-minute TTL from `STEP_DEADLINE_MS` and
  renews at phase boundaries plus a serialized 3-minute heartbeat.
- The operation it protects is much longer: `sandbox.createBackup()` was observed at
  ~260 s (`persistence.ts:2619` records its duration), and encryption/decryption runs
  after it while the lease is still held.
- The repo has already hit CPU-bound streaming: the restore decrypt path was
  rewritten after `Worker exceeded CPU time limit` (see the `ByteQueue` note in
  `AGENTS.md`).
- Admin snapshot requests enqueue a durable job by default when `ACTIVITY` exists;
  `?inline=1` is the explicit synchronous path, while `?async=1` preserves its
  accepted-then-background behavior. Without `ACTIVITY`, default requests run inline.

**Impact.** A JS heartbeat can still be delayed by synchronous or CPU-saturated work;
explicit renewals protect the slow boundaries and the longer TTL bounds ordinary
delays. Renewal failures/late renewals are now observable. The longer TTL means a
dead holder can block takeover for up to 12 minutes (and the existing bounded waiter
can time out first); this is an intentional trade-off against false expiry while a
durable step is legitimately active.

**Fix.** Renewals serialize on a promise chain and finish before release tombstoning;
`renew` is best-effort and never aborts the operation. Phase renewals cover legacy
stage, createBackup, encryption, config capture, restore decryption and archive apply.
Events: `persistence.lock.renewed`, `persistence.lock.renewal_late` (at half-TTL),
and `persistence.lock.renewal_failed`. Durable-by-default snapshots return `202`
with the job id and `409` on an occupied job slot. **Effort:** S.

---

### F6 — R2 enumeration is single-page everywhere

**Rules:** R6 (stream/bound), R9 (silent truncation).
**Severity:** Medium (latent: today's tenants are far below the caps).
**Status:** ✅ Shipped — `listAllObjects` follows `truncated`/`cursor`.
**Existing ID:** new (the companion's S1 is KV, not R2).

**Evidence.**

- `persistence.ts:970` and `:1093` — `bucket.list({ prefix: BACKUPS_PREFIX, limit: 1000 })`.
  At ~4 objects per archive (`data.sqsh`, `data.sqsh.enc`, `meta.json`, config), that
  is ~250 archives; retention is 3 by default, so this only bites after years of
  orphans or a mis-set `maxVersions`.
- `persistence.ts:2162` and `:3132` — `bucket.list({ prefix: CONFIG_PREFIX, limit: 500 })`
  while config retention is **30 days** and a config version is written on every
  snapshot — a busy tenant crosses 500 versions in ~0 days only if snapshots are very
  frequent, but the cap is silently applied after 500 entries and the UI stops showing
  older versions with no indication that the list was truncated.

**Impact.** Same class as **F1**, one layer down: truncation is silent, so "list" looks
authoritative when it is not. The config-version list is user-visible, which makes the
truncation a wrong answer rather than a slow one.

**Fix.** One `listAll(bucket, prefix)` helper that follows `truncated`/`cursor`
(`R2ListOptions.cursor`), used by all four call sites; return the count so the UI can
say "showing the newest N" if a cap is still intentional. **Effort:** S.

**Shipped.** `listAllObjects(bucket, prefix)` follows `truncated`/`cursor` with a
forward-progress guard (a truncated page without a new cursor logs and stops rather
than looping). It backs the backup reconcile sweep, the orphan archive sweep,
`cleanupOldConfigVersions` and `listConfigVersions`. Default page sizes are
unchanged; only the silent truncation is gone.

---

### F7 — `BackupWatch` adds a second per-tenant alarm that execs into the container

**Rules:** R4 (one alarm per object → queue), R1 (coordination atom), R10 (budget).
**Severity:** Medium (latent: gated off by default, but it is the intended default).
**Status:** ✅ Shipped — the tick is gated on the scheduler's job record (the fold
into `ActivityScheduler` remains, as planned, until the flag graduates).
**Existing ID:** new.

**Evidence.**

- `BackupWatch` owns its own alarm (`backup-watch/durable-object.ts:31-59`), separate
  from `ActivityScheduler` — deliberate for flag rollback, but it means two alarms per
  tenant.
- Every tick calls `sandbox.isRuntimeActive()` (`:41`) **and** `scanTenantChanges`
  (`:42`), which is a container `exec` running `find` over the state roots
  (`change-detection.ts:102-107`, `buildChangeScanScript` at `:76`).
- Default cadence is **2 minutes** (`config.ts:236`), and the tick re-arms itself at
  `now + checkMs` (`watch.ts:127`, `:156`).
- When a snapshot is already active, the tick's only outcome is `BACKUP_JOB_BUSY`
  (`durable-object.ts:47-52`), then it re-arms.

**Impact.** While enabled, that is a container RPC pair every 2 minutes per running
tenant (720/day), i.e. 1,440 control-channel operations per tenant per day purely to
decide "no change" — the exact churn the repo fights elsewhere (`Establishing RPC
connection`, process-count noise). It also keeps a third DO warm per tenant, and a
`busy` tick is pure waste.

**Fix.** Before probing, read the `ActivityScheduler`'s interval/next-due state (a
cheap DO read, no container RPC) and skip the tick when a snapshot is already
scheduled or the floor has not elapsed. Fold change detection into
`ActivityScheduler` when the flag graduates, so there is one alarm per tenant again.
**Effort:** S (gate), M (fold).

**Design (2026-10-24 slice).** Gate the tick, fold later.

- `ActivityScheduler.getSnapshotGate()` returns `{ active, lastCommittedSnapshotAt }`
  from **one** `backup-jobs` storage read (`getSnapshotJobGate`): `active` is the
  queued/running/retrying job, `lastCommittedSnapshotAt` is the newest
  `succeeded` `snapshot` job's `updatedAt`. No container RPC and no R2 read.
- `runBackupWatchTick` consults the gate **before** `isRunning()`/`scan()`:
  an active job re-arms at the next check; a committed snapshot newer than
  `floorMs` does the same. The floor is measured from
  `max(watch.lastSnapshotAt, gate.lastCommittedSnapshotAt)` so snapshots taken by
  the dispatch cron or an operator also suppress a duplicate scan.
- A gate read failure is **fail-open** (logged as `gate`, then the normal
  probe/scan runs): an unavailable scheduler must not blind the watch.
- The post-scan floor check is removed, because the pre-gate already enforces it.
  Behaviour change to note: a changed tenant inside the floor is no longer
  scanned, so `dirtySince` starts at the first scan after the floor instead of
  at the first in-floor tick. `maxDeferMs` still bounds starvation.
- The separate DO and alarm stay (fold is deferred until the flag graduates), so
  R4 remains **Partial** — but a tick that does no container work is now free.

**Shipped.** `getSnapshotJobGate` (`activity/backup-jobs.ts`) answers both
questions from one `backup-jobs` read, `ActivityScheduler.getSnapshotGate()` and
`ActivitySchedulerStub.getSnapshotGate()` expose it, and `runBackupWatchTick`
consults it before `isRunning()`/`scan()`. Tests:
`src/backup-watch/watch.test.ts` (active slot, self/other-trigger floor,
floor elapsed, fail-open gate), `src/backup-watch/durable-object.test.ts` (the
gate is read before any container RPC; a cron-committed snapshot suppresses the
scan), `src/activity/backup-jobs.test.ts` / `scheduler.test.ts` (gate shape:
active flag, newest succeeded snapshot, restore/failed jobs excluded, storage-only
path). Corpus: `test/business-rules/operational-modes.txt`.

---

### F8 — Telegram ingress rewrites its whole status object on every webhook

**Rules:** R14 (per-thread detail, aggregate summaries), R2 (incremental persistence).
**Severity:** Low–Medium.
**Status:** ✅ Shipped — one coalesced flush per burst, alarm-backed.
**Existing ID:** new (the companion does not cover ingress write cost).

**Evidence.**

- `recordIngress` (`telegram-ingress/route.ts:48-60`) is called on the forwarded path
  (`:157`) and the queued path (`:197`), and is dispatched with `waitUntil`.
- It calls `recordEvent` on the DO (`TelegramIngress.ts:34-35`), which
  `recordIngressEvent` implements as a read-modify-write of the full bounded status
  object (`telegram-ingress/status.ts`).

**Impact.** One DO storage write per inbound Telegram update, serialized on a single
object. For a busy tenant this is both a write amplification (the ring of 20 plus
counters are rewritten for each event) and an avoidable DO wake. `waitUntil` also
means a webhook response can complete while the write is still queued.

**Fix.** Coalesce: keep counters in memory for the tick and flush at most once every
N seconds (or on the last event of a burst), or write one Analytics Engine datapoint
per event and keep only the small "last N" ring in the DO. **Effort:** S.

**Design (2026-10-24 slice).** Coalesce in the DO, flush on an alarm.

- `src/telegram-ingress/coalesce.ts` holds the policy as a pure, injected module
  (storage, clock, timer) so it is unit-testable; `TelegramIngress` is the thin
  adapter. `status.ts` gains the pure `addIngressEvent` / `mergeIngressStatus`
  helpers it composes, and `recordIngressEvent` keeps using them.
- `recordEvent` accumulates into an in-memory pending batch and arms **one**
  alarm for `TELEGRAM_INGRESS_STATUS_FLUSH_MS` (1 s) after the first event. A
  burst therefore costs one read-modify-write instead of one per update.
  `TELEGRAM_INGRESS_STATUS_MAX_PENDING_EVENTS` (25) bounds the batch: a burst
  larger than the batch flushes immediately, so memory and staleness are bounded.
- `getStatus` returns the persisted status **merged with the pending batch**, so
  the dashboard and its WebSocket see an event as soon as it is recorded (no
  read-your-writes regression).
- `alarm()` flushes; a failed write restores the batch and rethrows so the alarm
  retry is the durability mechanism. Events recorded while a flush is in flight
  are chained behind it, never dropped.
- Accepted trade-off (documented): an eviction inside the ≤1 s window loses those
  events. These are UI counters/ring, not billing records; the alternative is the
  per-event write this finding is about, and the window is bounded and small.

**Shipped.** `telegram-ingress/coalesce.ts` (`createIngressCoalescer`, injected
storage/clock/alarm timer), pure batch helpers in `telegram-ingress/status.ts`
(`emptyIngressPending` / `addIngressEvent` / `mergeIngressPending` /
`mergeIngressStatus`, with `recordIngressEvent` rebuilt on them), the
`TelegramIngress.alarm()` flush, and the two `config.ts` constants. Tests:
`src/telegram-ingress/coalesce.test.ts` (one write per burst, single arm, merged
reads, bounded ring/cap, failed-write restore, in-flight flush, empty flush),
`src/telegram-ingress/status.test.ts` (pure batch merge), and
`src/telegram-ingress/TelegramIngress.test.ts` (DO seam: burst writes once, alarm
flushes, rate limiter untouched). Corpus:
`test/business-rules/operational-modes.txt`.

---

### F9 — `SandboxV1` is deployed but unreachable

**Rules:** R15 (explicit boundaries), R9 (operational surface).
**Severity:** Medium (deploy cost + drift risk), not a correctness bug.
**Status:** ✅ Shipped — deploy surface removed; class, image and migration kept.
**Existing ID:** new.

**Evidence.**

- `wrangler.jsonc` declares a second container app `fouria-sandbox-v1` with
  `scheduling_policy: "durable_object"` and its own image (`:156-168`), a DO binding
  (`:176-178`), and the same in staging (`:337`) and production (`:486`).
- The class exists and is exported, but `resolveSandboxRuntime()` reports `v1` as
  `pending` and no route selects it (`sandbox-runtime.ts`), as documented in the
  module header (`sandbox-v1.ts:20-31`).
- A `Dockerfile.v1` is maintained alongside the real Dockerfile.

**Impact.** Every deploy builds/pushes an extra container image and registers an
extra container application that serves no traffic. It is a silent drift surface: a
breaking change in `Dockerfile.v1` or `sandbox-v1.ts` fails a deploy for a component
nobody uses, and the migration ledger (`v7`) permanently records a class that is not
reachable.

**Fix.** Either (a) drop the container app + binding from all three environments and
keep the class + migration until the SDK 1.0 cutover actually lands, or (b) finish the
cutover behind `SANDBOX_RUNTIME=v1` and delete the legacy path. Do not ship a third
state. **Effort:** S.

**Design (2026-10-24 slice).** Option (a), deploy surface only.

- Remove the three `containers[]` entries (`fouria-sandbox-v1`,
  `fouria-sandbox-v1-staging`, and the production one) and the three `SandboxV1`
  `durable_objects` bindings. Keep `Dockerfile.v1`, the `SandboxV1` class, its
  `index.ts` export and migration `v7`: migrations are append-only, and the class
  plus the F4E1 `DirectoryBackup` surface are the foundation the cutover builds on.
- Verified against the live account before changing anything: **no**
  `*-sandbox-v1` container application exists today, so the removal destroys
  nothing and cannot interrupt a tenant. Tenant containers come from lerma's own
  manifest, not from this file.
- Removing the binding means `SANDBOX_RUNTIME=v1` is no longer instantiable from
  this config; restoring the container entry is an explicit prerequisite of the
  cutover. That is recorded in `docs/plans/sandbox-v1-scaffold.md`.
- The `sandbox-v1-validation` lane is unaffected: it builds and publishes both
  images from its own branch/worktree (`publish-validation.sh` only requires
  `Dockerfile.v1` to exist) and never deploys this Worker from `develop`.

**Shipped.** `wrangler.jsonc` no longer declares any `SandboxV1` container
application or DO binding in the local, staging or production environment; the
migration `v7` entry, the class, `Dockerfile.v1` and the `index.ts` export stay.
`vite.config.ts` lost the E2E `fouria-sandbox-v1` rename branch (there is no
named app to rename), and the generated config now carries exactly one container
app (`fouria-sandbox`) and bindings `Sandbox`/`ACTIVITY`/`TELEGRAM_INGRESS`/
`BACKUP_WATCH`. Docs: `apps/fouria/docs/WORKER-DEVELOPMENT.md`,
`docs/plans/sandbox-v1-scaffold.md` (registration + the cutover prerequisite).

---

### F10 — The skill's repo map is stale

**Rules:** documentation supporting R1/R4/R15 (the map is how the next author finds
the correct object).
**Severity:** Low (but actively misleading).
**Status:** ✅ Shipped — map corrected (see the committed diff).
**Existing ID:** n/a.

**Evidence.** `skills/cloudflare-durable-agents/SKILL.md:345` maps `CustomerRegistry`
DO to `apps/fouria/src/customer-registry.ts`. That file does not exist; migrations
`v2`/`v3` created and then deleted the class (`wrangler.jsonc:104-113`). `BackupWatch`
and `SandboxV1` are absent from the map even though both are deployed.

**Fix.** Correct the table in the same change as this audit (done — see the diff),
and add a one-line rule that the map must be updated when a DO class is added,
renamed, or deleted. **Effort:** trivial.

---

## 3. What already satisfies the rules — do not regress

These are the load-bearing pieces; each maps to a rule and should be treated as a
constraint on any refactor.

- **Durable boot job.** Cold boot (restore → launch) runs on the `ActivityScheduler`
  alarm, not in `waitUntil` — R3/R9. `activity/boot-job.ts`, requested from
  `/api/warmup`, non-passive `/api/health` and `/api/startup/ws`.
- **Combined alarm with a generation guard.** Four job families share one alarm; the
  minimum due wins, and `null` deletes the alarm so an idle tenant is silent — R4.
  `activity/scheduler.ts:834-846`, `activity/alarm.ts:113-133`.
- **Per-step cursor and lease.** One step per invocation, `STEP_LEASE_MS = 6 min`,
  `STEP_DEADLINE_MS = 12 min`, `STEP_CONTINUE_DELAY_MS = 1 s` — R10/R5.
  `activity/job-steps.ts:29-52`.
- **Deferral does not consume an attempt.** A maintenance job blocked by an active
  restore is rescheduled instead of failed — R5. `activity/scheduler.ts`.
- **Stop-time idleness re-check.** `stopIfIdle()` re-reads the idle window and process
  list at stop time; a tenant that became active is `skipped`, not stopped — R9.
  `sandbox-do.ts`.
- **Queue-based Telegram ingress.** Cold updates are enqueued and the container is
  woken by the consumer, not by the webhook — R3/R9.
- **Streamed backup crypto.** `FixedLengthStream` + `ByteQueue` keep restore/encrypt
  memory- and CPU-bounded — R6.
- **Pure catalog reads, explicit catalog writes.** `getBackupHistory` never lists,
  prunes, sweeps or writes; healing lives in `reconcileBackupCatalog` (durable
  `repair` step, `POST /api/admin/storage/reconcile`, restore, reprovision) — R3/R5.
- **Non-waking liveness.** `/healthz` does one `isRuntimeActive()` read and never
  starts a container — R9.
- **Tenant resolution from the Worker env.** `sandbox-tenant.ts` avoids the fixed
  container id, so the R2 bridges cannot list another tenant — R15.
- **The synchronous `checkpoint` control action is intentional.** It must fail closed
  before a rollout, so it is the one accepted exception to R3 — do not "fix" it into a
  queue without a verified-completion barrier.

---

## 4. Order of work

Ties into the companion's roadmap instead of replacing it. "Now" means the next
PR-sized slice.

| When      | Work                                                                 | Finding | Companion ID  | Status                    |
| --------- | -------------------------------------------------------------------- | ------- | ------------- | ------------------------- |
| **Now**   | Paginate KV + queue fan-out with bounded concurrency                 | F1      | P0 #2 / S1    | Shipped                   |
| **Now**   | Per-tenant jitter on the fan-out                                     | F2      | new           | Shipped                   |
| **Now**   | Etag CAS on the catalog handle                                       | F4      | new           | Shipped                   |
| **Now**   | `listAll` for R2 (backups + configs)                                 | F6      | new           | Shipped                   |
| **Now**   | Fix the skill's repo map                                             | F10     | n/a           | Shipped                   |
| **Next**  | Make `GET /storage/backups` read-only; move reconcile/prune to a job | F3      | new           | Shipped                   |
| **Next**  | Renew the lease at phase boundaries; TTL from `STEP_DEADLINE_MS`     | F5      | J4 adjacent   | Shipped                   |
| **Next**  | Gate / fold `BackupWatch` before enabling it by default              | F7      | new           | Shipped (gate; fold Next) |
| **Next**  | Coalesce ingress status writes                                       | F8      | new           | Shipped                   |
| **Next**  | Resolve `SandboxV1`: remove the deploy surface or finish the cutover | F9      | new           | Shipped (option a)        |
| **Later** | Checkpointed steps, DLQ, pushed telemetry, durable billing ledger    | —       | P0 #3, P1, P2 | Later                     |

Rationale for the order: F1 and F2 are the only findings that combine **silent
correctness loss** with **fleet-wide blast radius**; F4/F6 are small diffs that remove
the "last writer wins" and "silent truncation" classes before they are exercised; F3
shipped by splitting the catalog read from an explicit, CAS-guarded reconcile; F5
renews the lease at phase boundaries; the F7/F8/F9 slice removes the remaining
recurring waste (a container probe the watch cannot use, a status write per webhook,
and an image + container app per deploy) without touching a tenant's data path. What
is left is the "Later" row — chiefly the fleet aggregate (A5) behind R14's remaining
partial and the `BackupWatch` fold behind R4's.

---

## 5. Reconciliation with the companion audit

| This doc | Companion ID     | Relationship                                                              |
| -------- | ---------------- | ------------------------------------------------------------------------- |
| F1       | §S2 / S1 / P0 #2 | Same defect; this adds R9 framing + the queue consumer shape.             |
| F2       | —                | New: jitter/blast radius. Adjacent to P1 #9 (cadence), different fix.     |
| F3       | —                | New: the backups **read** path; §S3 is health/startup polling.            |
| F4       | —                | New: catalog handle CAS. J4 covers lease/orphan recovery, not this write. |
| F5       | J4 adjacent      | New mechanism: timer-only renewal vs a 60 s TTL.                          |
| F6       | —                | New: R2 pagination (S1 is KV).                                            |
| F7       | —                | New: `BackupWatch` cadence and container RPCs.                            |
| F8       | —                | New: ingress write amplification; R14/A5 cover fleet aggregates.          |
| F9       | —                | New: unused deploy surface.                                               |
| F10      | —                | New: documentation.                                                       |

Nothing in the companion is contradicted here. Where both touch the same code (F1,
F5) the companion's roadmap entry stays authoritative for scheduling; this doc only
adds the rule citation and the implementation shape.

---

## 6. Verification delivered for the P0 slice

Per the repository covenant, each change needed its corpus, tests and docs in the same
PR. What this slice actually ships:

- **Corpus:** `test/business-rules/operational-modes.txt` (fan-out pagination, queue
  delivery/retry, jitter, catalog CAS) and `tenant-isolation.txt` (the sweep resolves
  each tenant at delivery time and never crosses slugs).
- **Unit tests:** `workers/fouria-dispatch/test/unit/control-queue.test.ts` (KV
  pagination, batching, jitter bounds and determinism, live re-resolution, ack/retry,
  fallback concurrency) and the extended `cron-fanout.test.ts`; in `apps/fouria`,
  `src/persistence.test.ts` covers CAS conflicts/retry exhaustion, mutation-shaped
  prune, create-only handles and paginated R2 listing.
- **Docs:** `workers/fouria-dispatch/README.md` (sweep semantics, queue, jitter and the
  `CONTROL_QUEUE` binding), `apps/fouria/docs/WORKER-DEVELOPMENT.md` (CAS catalog
  writes and `listAllObjects`).
- **Changeset:** both touched packages (`fouria`, `fouria-dispatch`) are CalVer and
  `ignore`d by Changesets, so the PR carries an **empty** changeset whose body records
  the change.

Remaining findings keep the obligations recorded for their own PRs: reconcile ownership
in `apps/fouria/docs/OPERATOR-GUIDE.md`, lease renewal tests under `src/activity/`, and
`docs/TESTING.md` if a new E2E covers the fan-out.

### Verification for the F7/F8/F9 slice

- **Corpus:** `test/business-rules/operational-modes.txt` gains three entries — the
  watch gate is read before any container RPC, the ingress status write is coalesced,
  and `SandboxV1` has no container application or DO binding.
- **Unit tests:** `src/backup-watch/watch.test.ts` + `durable-object.test.ts` +
  `src/activity/backup-jobs.test.ts` / `scheduler.test.ts` (F7);
  `src/telegram-ingress/coalesce.test.ts` + `status.test.ts` +
  `TelegramIngress.test.ts` (F8). F9 is config-only and is verified by the generated
  `dist/fouria/wrangler.json` (one container app, four DO bindings, `v7` retained).
- **Docs:** `apps/fouria/docs/WORKER-DEVELOPMENT.md` (gated watch, coalesced ingress,
  class-only `SandboxV1` registration) and `docs/plans/sandbox-v1-scaffold.md`
  (deploy surface removed + cutover prerequisite).
- **Changeset:** `apps/fouria` is CalVer and `ignore`d, so the PR carries an **empty**
  changeset whose body records the change.

---

## 7. Explicit non-recommendations

- **Do not move the boot back into `waitUntil()`** — that regression is documented and
  was the cause of a truncated restore.
- **Do not add a global scheduler DO** to own per-tenant alarms. It would recreate the
  singleton bottleneck R1 forbids; pagination + a queue is the correct fix for the
  fleet sweep.
- **Do not make every alarm a `Workflow`.** WFP does not support Workflows; the
  existing step cursor already gives resumability without a second execution model.
- **Do not add multi-region container replicas.** The OpenClaw state DB is
  single-writer; availability is an RTO problem (companion §3.2).
- **Do not remove the `checkpoint` control action's synchronous contract.** It is the
  one place where waiting for a verified archive is the point.
