Durable-agents rules audit — FourIA tenant runtime (second pass)
On this page
Status: P0, F3 and F5 shipped — F1–F6 and F10 are implemented; remaining findings
stay Next/Later. Each finding carries its own status line.
Date: 2026-10-22
Scope: apps/fouria (tenant User Worker + its Durable Objects, container lifecycle,
persistence) and workers/fouria-dispatch (fleet cron, routing).
Rubric: skills/cloudflare-durable-agents/SKILL.md
— R1–R15, the §5 failure matrix, and the §6 checklist.
Companion: docs/plans/durable-jobs-availability-audit.md
(same date) already catalogues the job-lifecycle, availability and scale findings as
J1–J5 / A1–A5 / S1–S4 with a P0–P2 roadmap. This is a deliberately different pass:
it scores the implementation against the rules, re-derives the same defects
where the rule text sharpens the failure mode, and adds what the first pass missed
(catalog I/O on read paths, the unlocked catalog handle, timer-only lease renewal,
watch cadence, per-event status writes, dead deploy surface).
Severity scale used below: High = silent correctness loss or fleet-wide outage vector; Medium = avoidable cost, wasted work, or a latent correctness bug; Low = hygiene with a clear cheap fix.
1. Scorecard
| Rule | Verdict | Basis |
|---|---|---|
| R1 One object per unit of coordination | Pass | Per-tenant Worker scripts with tenant-named objects (getActivityScheduler, getSandbox(env, 'openclaw')). No platform-wide singleton. BackupWatch and TelegramIngress add objects per tenant, never shared ones. |
| R2 SQLite is the truth | Pass | Job records are persisted before the caller is acknowledged and executed by the alarm (activity/scheduler.ts:178-182). State is read from storage on every alarm, not from instance fields. |
| R3 Never do the work in the request | Pass | Default admin snapshots enqueue durable jobs; ?inline=1 is the explicit synchronous escape hatch and no-ACTIVITY environments retain an inline fallback (F5 shipped). getBackupHistory is pure; reconciliation is explicit/durable (F3 shipped). |
| R4 One alarm per object → alarm queue | Partial | scheduleCombinedAlarm arms the minimum due across boot/backup/maintenance/activity and clears on null (activity/scheduler.ts:834-846, activity/alarm.ts:113-133) — textbook. The fleet sweep is now paginated and queue-backed (F1/F2 shipped). Remaining: BackupWatch is still a second per-tenant alarm doing container work (F7). |
| R5 Alarms are at-least-once | Pass | Job steps retry with a cursor; lock release is an etag-CAS tombstone; busy slots are rejected explicitly. The catalog handle is an etag CAS (F4), and snapshot leases renew at phase boundaries plus a serialized heartbeat (F5). |
| R6 Stream, never buffer | Pass | Restore/encrypt/decrypt are streamed through FixedLengthStream (memory-bounded), and R2 enumeration now follows truncated/cursor through listAllObjects (F6 shipped). |
R7 blockConcurrencyWhile constructor-only | Pass | Only used by the unreleased SandboxV1 container lifecycle (sandbox-v1.ts:213); no hot-path use. |
| R8 Storage awaits interleave | Pass | mutatePersistenceStore re-reads the etag and re-applies the mutation on conflict, so backup-handle.json can no longer be lost-updated across awaits (F4 shipped). |
| R9 Assume death; add a watchdog | Pass | Real watchdogs exist (stuck start, boot-job lease, per-step deadline), and the dispatch sweep now paginates, queues and jitters deliveries instead of running one unbounded pass (F1/F2 shipped). |
| R10 Budget the turn | Pass | Exactly one step per alarm with STEP_LEASE_MS / STEP_DEADLINE_MS / STEP_CONTINUE_DELAY_MS (activity/job-steps.ts:29-52); cadence floor is 2 × the last snapshot duration. |
| R11 Hibernate WebSockets | N/A | Sockets are Worker↔container proxies, not DO-hosted. The polling-loop cost is real but already owned by A5 / §S3 in the companion doc. |
| R12 Model failover | N/A | OpenClaw owns provider routing; this layer never calls a model. |
| R13 Sub-agents talk in natural language | N/A | No sub-agent runtime in the Worker. |
| R14 Observability | Partial | Structured events are strong, but there is no fleet-level aggregate (A5), and Telegram ingress rewrites a whole status object per webhook (F8). |
| R15 Tenant boundaries are explicit | Pass | sandbox-tenant.ts resolves the tenant from the per-tenant env instead of ctx.containerId, so the R2 bridges cannot cross tenants. The unused SandboxV1 app is a deploy-surface problem, not an isolation hole (F9). |
Ten of fifteen rules now pass (R1, R2, R3, R5, R6, R7, R8, R9, R10, R15), two remain partial (R4, R14) and three are not applicable (R11–R13). No rule is a blanket failure — the P0 slice closed the three rule gaps (R6, R8, R9); the remaining partials are the Next findings F7 and F8.
2. Findings
F1 — Fleet fan-out: silently truncated at 1,000 tenants, then unbounded
Rules: R9 (watchdog / at scale), R3 (work outside the request), R1 (one object per unit).
Severity: High.
Status: ✅ Shipped — paginated sweep + CONTROL_QUEUE fan-out (see below).
Existing ID: §S2 / Finding S1 / P0 #2 in the companion audit — same defect. This
entry adds the rule basis and the concrete queue shape; do not file it twice.
Evidence — workers/fouria-dispatch/src/index.ts:869-903:
:870—env.TENANT_INDEX.list({ prefix: 'slug:' })with nocursorloop and nolist_completecheck. KV returns at most 1,000 keys per page.:871-877—Promise.allover one KVgetper key, unbounded.:885-888—for (… ) ctx.waitUntil(forwardTenantControl(…, 'backup')), i.e. one concurrent fetch per tenant, also unbounded.- Production fires this every 15 minutes (
workers/fouria-dispatch/wrangler.jsonc:34); staging hourly (:73).
Impact. Past 1,000 tenants the remainder never receive a scheduled backup and
nothing errors — a silent correctness loss that only surfaces at restore time.
Before that, the invocation runs ~2 metered operations per tenant (one KV get,
one dispatch fetch), so the 1,000-per-invocation subrequest budget (and the
separate KV-operations budget, also 1,000) is exhausted around ~500 tenants. The
cron’s waitUntil fan-out then has to settle 1,000 concurrent fetches inside the
cron wall. This is the same cancellation class that killed request-scoped restores.
Fix. Paginate (while (!listed.list_complete)), then enqueue one message per
tenant onto the existing TASKS queue (apps/fouria/wrangler.jsonc:45-47, consumer
at apps/fouria/src/index.ts:1592) and let the queue consumer call
forwardTenantControl. Give the consumer max_concurrency and a bounded
max_batch_size; deliveries are already idempotent because the tenant enqueues a
durable job and answers 202. Effort: M.
Shipped. collectTenantSlugs (workers/fouria-dispatch/src/index.ts) follows
list_complete / cursor and dedupes keys, and the sweep enqueues through its own
CONTROL_QUEUE producer (100 messages per sendBatch) rather than the tenant
TASKS queue, which would have coupled fleet control work to tenant automation
traffic. The consumer re-resolves slug:<slug> at delivery time, skips
missing/frozen/nameless tenants, and acks delivered/skipped while retry()ing
failed. When the binding is absent (local/older environments), a bounded
concurrency-8 direct path preserves the old behavior. Queue definitions live per
environment in wrangler.jsonc; deploy-dispatch.yml creates them idempotently.
F2 — Every tenant’s scheduled work starts on the same wall-clock minute
Rules: R9 (fleet blast radius), R10 (budget the turn).
Severity: High (blast radius), cheap to fix.
Status: ✅ Shipped — deterministic per-slug delaySeconds on the queue message.
Existing ID: new; adjacent to P1 #9 (cadence cannot be met on large tenants).
Evidence.
- The dispatch cron is a fixed
*/15 * * * *(workers/fouria-dispatch/wrangler.jsonc:34). Each delivery calls/control→enqueueBackupJob→scheduleCombinedAlarm(apps/fouria/src/index.ts:578-586,activity/scheduler.ts:178-182), which arms the alarm immediately. - The interval gate is 5 minutes by default (
config.ts:252) and otherwise2 ×the last snapshot duration, so a 15-minute tick clears it for most tenants on most ticks: each size class of tenant snapshots in lockstep. - A snapshot is a multi-minute container stage plus
createBackup(~260 s observed) plus encryption. - Second fixed cadence: every tenant Worker runs
*/2 * * * *(apps/fouria/wrangler.jsonc:73), i.e. 720 invocations per tenant per day, for the billable drain.
Impact. Two distinct problems. (1) Synchronized load: at minute 0/15/30/45 every tenant’s container stage, R2 upload and encrypt start together — a self-inflicted thundering herd whose peak is proportional to fleet size. (2) Shared blast radius: a bad snapshot step or a Cloudflare incident in that window hits every tenant at once instead of degrading gradually. Neither is visible in per-tenant metrics.
Fix. Derive a stable per-tenant offset from the slug (e.g. fnv1a(slug) % interval)
and either (a) have the queue consumer delay each message by that offset before
forwardTenantControl, or (b) compute the next due time from
controller.scheduledTime + offset and let the tenant’s interval gate absorb it.
Prefer (a): the queue owns the delay, the tenant stays stateless. Effort: S.
Shipped. fanoutJitterSeconds derives a stable FNV-1a offset from the slug into
a 300 s window (overridable with CONTROL_FANOUT_JITTER_SECONDS, 0 disables,
capped at 86400) and sets it as the message’s delaySeconds. The offset is
deterministic, so a redelivery keeps the same schedule, and it only applies on the
queue path — the direct fallback delivers immediately. Over a 15-minute tick the
fleet now spreads across five minutes instead of one.
The second fixed cadence noted above (*/2 * * * * per tenant Worker for the
billable drain) is a separate trigger and is out of this slice.
F3 — A dashboard GET reconciles, prunes and writes the catalog
Rules: R3 (never do the work in the request), R5 (idempotency / side effects),
R14 (reads should be cheap and observable).
Severity: Medium (read amplification + a GET that deletes objects).
Status: ✅ Shipped — pure read + explicit reconcileBackupCatalog (see below).
Existing ID: new; §S3 covers health/WS polling, not the backups read path.
Evidence.
getBackupHistorycallsreconcileOrphanedBackupson every invocation (persistence.ts:3121). The same reconcile also runs before the interval gate inscheduleBackupIfNeeded(:3567) and inrestoreIfNeeded(:1393).reconcileOrphanedBackups(persistence.ts:956) doesbucket.list({ limit: 1000 })(:970), readsmeta.jsonper candidate archive, deletes over-cap archives on read (:1056-1059→cleanupOldBackups), and may write the handle back (:1072).- The prune fan-out is unbounded:
Promise.all(toRemove.map((entry) => deleteBackupObjects(bucket, entry.id)))(persistence.ts:1988), where each delete is a sequence of up to four R2 deletes (:1952-1965). GET /api/admin/storage/backupsis that read (routes/api/storage.ts:203-205).BackupsPagecalls it on load (client/pages/BackupsPage.tsx:66), and the history panel then calls it every 5 s for up to 5 minutes while an async snapshot/restore is pending (client/components/BackupHistoryPanel.tsx:15-17,:127-139) — ~60 full reconciles per operation.
Impact. A read endpoint performs retention deletes and writes, so an admin
opening a page can change storage. Cost: one R2 list plus N meta reads per poll, per
open tab, with no caching. Correctness: two concurrent reads can both prune and both
write the handle (see F4). Observability: a failed delete surfaces as a failed
GET, which reads like a dashboard bug rather than a storage fault.
Fix. Split the read from the repair: getBackupHistory returns the persisted
catalog (getPersistenceStore) and never mutates. Move reconcile + prune into the
existing durable maintenance/backup step (it already runs on a lease and already has a
cursor), or into an explicit POST /api/admin/storage/reconcile used by the operator
and by the scheduled job. Cache the panel response for the poll window instead of
re-listing on every tick. Effort: M.
Shipped. getBackupHistory is now a pure read — getPersistenceStore +
getStableBackup + isRecoveryHeld in parallel, no list, write or delete. The write
path is explicit: reconcileCatalogState (adoption + retention prune inside the
etag-CAS mutation, reporting the ids it changed) → repairBackupCatalog (sweep +
in-place encryption) → the refreshed history, exposed as reconcileBackupCatalog.
The durable repair step calls it before the gate step of every scheduled
snapshot, so the catalog still self-heals after a killed invocation (issue #344);
scheduleBackupIfNeeded keeps its inline reconcile and restore keeps its own.
Operators get POST /api/admin/storage/reconcile (ADMIN/ROOT) as the manual lever,
and the debug/reprovision guard reconciles before deciding there is nothing
restorable, so a crashed upload cannot be mistaken for an unrecoverable tenant. The
history panel now polls the durable job (getBackupJobStatus) while a snapshot is in
flight instead of re-listing the catalog every 5 s, falling back to the history diff
only when the server returns no job id. R3 stays Partial only because of the
synchronous admin snapshot (F5).
F4 — backup-handle.json is written without compare-and-set
Rules: R8 (interleaving across awaits), R5 (idempotent alarms), R2 (storage is truth).
Severity: Medium (a committed archive can silently disappear from the catalog).
Status: ✅ Shipped — mutatePersistenceStore etag CAS.
Existing ID: new.
Evidence.
savePersistenceStoreis an unconditionalbucket.put(HANDLE_KEY, …)(persistence.ts:611-613). There is noonlyIf, no version, no retry.- It is called from a read path (
:1072) and from every write path: restore signal/latch (:1183,:1546), snapshot commit (:2792,:2878,:2917), and the scheduler timestamp (:3652). - The lock path already does this correctly, with etag CAS and a tombstone
(
persistence.ts:3249,:3252,:3268,:3302).
Impact. Two writers that both read the handle and then put it back race: last
writer wins. The realistic sequence is a polled GET (reconcile) overlapping a
snapshot finalize — the finalize adds backups/<id> and the poll writes back the
pre-snapshot store, so a successfully uploaded archive is no longer in the catalog
(the archive objects remain, so F3’s reconcile can adopt it later, but
stable/config and the retention decision were made against a stale list). During a
restore the same race can drop a restore-config-pending marker.
Fix. put(..., { onlyIf: { etagMatches: read.etag } }) with a small read-retry
loop, mirroring releaseLock (persistence.ts:3302); expose savePersistenceStore
as mutatePersistenceStore(fn) so callers cannot forget. Effort: S.
Shipped. mutatePersistenceStore re-reads the handle and re-applies the caller’s
mutation against the fresh catalog on conflict (5 attempts, then
PersistenceStoreConflictError) — not a blind merge, because prune/removal
mutations must legitimately shrink the catalog. Writes use etagMatches when an
etag exists and etagDoesNotMatch: '*' for a create-only handle; a mutation may
return null to skip the write. Reconcile, snapshot finalization, the retention
prune and the schedule-timestamp commit now all run inside that CAS. The restore
markers were left on their existing paths in this slice.
F5 — The restore/backup lease outlives its renewal mechanism
Rules: R5 (at-least-once, but only one owner), R7 (no long blocking work on the hot path), R10 (budget). Severity: Medium. Status: ✅ Shipped — 12-minute lease, serialized phase/heartbeat renewals, structured renewal events, and durable-by-default admin snapshots. Existing ID: adjacent to J4 (lease/orphan handling) but distinct.
Evidence.
- The former 60 s lease and timer-only heartbeat could expire during long archive
operations; the lease now derives its 12-minute TTL from
STEP_DEADLINE_MSand renews at phase boundaries plus a serialized 3-minute heartbeat. - The operation it protects is much longer:
sandbox.createBackup()was observed at ~260 s (persistence.ts:2619records its duration), and encryption/decryption runs after it while the lease is still held. - The repo has already hit CPU-bound streaming: the restore decrypt path was
rewritten after
Worker exceeded CPU time limit(see theByteQueuenote inAGENTS.md). - Admin snapshot requests enqueue a durable job by default when
ACTIVITYexists;?inline=1is the explicit synchronous path, while?async=1preserves its accepted-then-background behavior. WithoutACTIVITY, default requests run inline.
Impact. A JS heartbeat can still be delayed by synchronous or CPU-saturated work; explicit renewals protect the slow boundaries and the longer TTL bounds ordinary delays. Renewal failures/late renewals are now observable. The longer TTL means a dead holder can block takeover for up to 12 minutes (and the existing bounded waiter can time out first); this is an intentional trade-off against false expiry while a durable step is legitimately active.
Fix. Renewals serialize on a promise chain and finish before release tombstoning;
renew is best-effort and never aborts the operation. Phase renewals cover legacy
stage, createBackup, encryption, config capture, restore decryption and archive apply.
Events: persistence.lock.renewed, persistence.lock.renewal_late (at half-TTL),
and persistence.lock.renewal_failed. Durable-by-default snapshots return 202
with the job id and 409 on an occupied job slot. Effort: S.
F6 — R2 enumeration is single-page everywhere
Rules: R6 (stream/bound), R9 (silent truncation).
Severity: Medium (latent: today’s tenants are far below the caps).
Status: ✅ Shipped — listAllObjects follows truncated/cursor.
Existing ID: new (the companion’s S1 is KV, not R2).
Evidence.
persistence.ts:970and:1093—bucket.list({ prefix: BACKUPS_PREFIX, limit: 1000 }). At ~4 objects per archive (data.sqsh,data.sqsh.enc,meta.json, config), that is ~250 archives; retention is 3 by default, so this only bites after years of orphans or a mis-setmaxVersions.persistence.ts:2162and:3132—bucket.list({ prefix: CONFIG_PREFIX, limit: 500 })while config retention is 30 days and a config version is written on every snapshot — a busy tenant crosses 500 versions in ~0 days only if snapshots are very frequent, but the cap is silently applied after 500 entries and the UI stops showing older versions with no indication that the list was truncated.
Impact. Same class as F1, one layer down: truncation is silent, so “list” looks authoritative when it is not. The config-version list is user-visible, which makes the truncation a wrong answer rather than a slow one.
Fix. One listAll(bucket, prefix) helper that follows truncated/cursor
(R2ListOptions.cursor), used by all four call sites; return the count so the UI can
say “showing the newest N” if a cap is still intentional. Effort: S.
Shipped. listAllObjects(bucket, prefix) follows truncated/cursor with a
forward-progress guard (a truncated page without a new cursor logs and stops rather
than looping). It backs the backup reconcile sweep, the orphan archive sweep,
cleanupOldConfigVersions and listConfigVersions. Default page sizes are
unchanged; only the silent truncation is gone.
F7 — BackupWatch adds a second per-tenant alarm that execs into the container
Rules: R4 (one alarm per object → queue), R1 (coordination atom), R10 (budget). Severity: Medium (latent: gated off by default, but it is the intended default). Existing ID: new.
Evidence.
BackupWatchowns its own alarm (backup-watch/durable-object.ts:31-59), separate fromActivityScheduler— deliberate for flag rollback, but it means two alarms per tenant.- Every tick calls
sandbox.isRuntimeActive()(:41) andscanTenantChanges(:42), which is a containerexecrunningfindover the state roots (change-detection.ts:102-107,buildChangeScanScriptat:76). - Default cadence is 2 minutes (
config.ts:236), and the tick re-arms itself atnow + checkMs(watch.ts:127,:156). - When a snapshot is already active, the tick’s only outcome is
BACKUP_JOB_BUSY(durable-object.ts:47-52), then it re-arms.
Impact. While enabled, that is a container RPC pair every 2 minutes per running
tenant (720/day), i.e. 1,440 control-channel operations per tenant per day purely to
decide “no change” — the exact churn the repo fights elsewhere (Establishing RPC connection, process-count noise). It also keeps a third DO warm per tenant, and a
busy tick is pure waste.
Fix. Before probing, read the ActivityScheduler’s interval/next-due state (a
cheap DO read, no container RPC) and skip the tick when a snapshot is already
scheduled or the floor has not elapsed. Fold change detection into
ActivityScheduler when the flag graduates, so there is one alarm per tenant again.
Effort: S (gate), M (fold).
F8 — Telegram ingress rewrites its whole status object on every webhook
Rules: R14 (per-thread detail, aggregate summaries), R2 (incremental persistence). Severity: Low–Medium. Existing ID: new (the companion does not cover ingress write cost).
Evidence.
recordIngress(telegram-ingress/route.ts:48-60) is called on the forwarded path (:157) and the queued path (:197), and is dispatched withwaitUntil.- It calls
recordEventon the DO (TelegramIngress.ts:34-35), whichrecordIngressEventimplements as a read-modify-write of the full bounded status object (telegram-ingress/status.ts).
Impact. One DO storage write per inbound Telegram update, serialized on a single
object. For a busy tenant this is both a write amplification (the ring of 20 plus
counters are rewritten for each event) and an avoidable DO wake. waitUntil also
means a webhook response can complete while the write is still queued.
Fix. Coalesce: keep counters in memory for the tick and flush at most once every N seconds (or on the last event of a burst), or write one Analytics Engine datapoint per event and keep only the small “last N” ring in the DO. Effort: S.
F9 — SandboxV1 is deployed but unreachable
Rules: R15 (explicit boundaries), R9 (operational surface). Severity: Medium (deploy cost + drift risk), not a correctness bug. Existing ID: new.
Evidence.
wrangler.jsoncdeclares a second container appfouria-sandbox-v1withscheduling_policy: "durable_object"and its own image (:156-168), a DO binding (:176-178), and the same in staging (:337) and production (:486).- The class exists and is exported, but
resolveSandboxRuntime()reportsv1aspendingand no route selects it (sandbox-runtime.ts), as documented in the module header (sandbox-v1.ts:20-31). - A
Dockerfile.v1is maintained alongside the real Dockerfile.
Impact. Every deploy builds/pushes an extra container image and registers an
extra container application that serves no traffic. It is a silent drift surface: a
breaking change in Dockerfile.v1 or sandbox-v1.ts fails a deploy for a component
nobody uses, and the migration ledger (v7) permanently records a class that is not
reachable.
Fix. Either (a) drop the container app + binding from all three environments and
keep the class + migration until the SDK 1.0 cutover actually lands, or (b) finish the
cutover behind SANDBOX_RUNTIME=v1 and delete the legacy path. Do not ship a third
state. Effort: S.
F10 — The skill’s repo map is stale
Rules: documentation supporting R1/R4/R15 (the map is how the next author finds the correct object). Severity: Low (but actively misleading). Status: ✅ Shipped — map corrected (see the committed diff). Existing ID: n/a.
Evidence. skills/cloudflare-durable-agents/SKILL.md:345 maps CustomerRegistry
DO to apps/fouria/src/customer-registry.ts. That file does not exist; migrations
v2/v3 created and then deleted the class (wrangler.jsonc:104-113). BackupWatch
and SandboxV1 are absent from the map even though both are deployed.
Fix. Correct the table in the same change as this audit (done — see the diff), and add a one-line rule that the map must be updated when a DO class is added, renamed, or deleted. Effort: trivial.
3. What already satisfies the rules — do not regress
These are the load-bearing pieces; each maps to a rule and should be treated as a constraint on any refactor.
- Durable boot job. Cold boot (restore → launch) runs on the
ActivityScheduleralarm, not inwaitUntil— R3/R9.activity/boot-job.ts, requested from/api/warmup, non-passive/api/healthand/api/startup/ws. - Combined alarm with a generation guard. Four job families share one alarm; the
minimum due wins, and
nulldeletes the alarm so an idle tenant is silent — R4.activity/scheduler.ts:834-846,activity/alarm.ts:113-133. - Per-step cursor and lease. One step per invocation,
STEP_LEASE_MS = 6 min,STEP_DEADLINE_MS = 12 min,STEP_CONTINUE_DELAY_MS = 1 s— R10/R5.activity/job-steps.ts:29-52. - Deferral does not consume an attempt. A maintenance job blocked by an active
restore is rescheduled instead of failed — R5.
activity/scheduler.ts. - Stop-time idleness re-check.
stopIfIdle()re-reads the idle window and process list at stop time; a tenant that became active isskipped, not stopped — R9.sandbox-do.ts. - Queue-based Telegram ingress. Cold updates are enqueued and the container is woken by the consumer, not by the webhook — R3/R9.
- Streamed backup crypto.
FixedLengthStream+ByteQueuekeep restore/encrypt memory- and CPU-bounded — R6. - Pure catalog reads, explicit catalog writes.
getBackupHistorynever lists, prunes, sweeps or writes; healing lives inreconcileBackupCatalog(durablerepairstep,POST /api/admin/storage/reconcile, restore, reprovision) — R3/R5. - Non-waking liveness.
/healthzdoes oneisRuntimeActive()read and never starts a container — R9. - Tenant resolution from the Worker env.
sandbox-tenant.tsavoids the fixed container id, so the R2 bridges cannot list another tenant — R15. - The synchronous
checkpointcontrol action is intentional. It must fail closed before a rollout, so it is the one accepted exception to R3 — do not “fix” it into a queue without a verified-completion barrier.
4. Order of work
Ties into the companion’s roadmap instead of replacing it. “Now” means the next PR-sized slice.
| When | Work | Finding | Companion ID | Status |
|---|---|---|---|---|
| Now | Paginate KV + queue fan-out with bounded concurrency | F1 | P0 #2 / S1 | Shipped |
| Now | Per-tenant jitter on the fan-out | F2 | new | Shipped |
| Now | Etag CAS on the catalog handle | F4 | new | Shipped |
| Now | listAll for R2 (backups + configs) | F6 | new | Shipped |
| Now | Fix the skill’s repo map | F10 | n/a | Shipped |
| Next | Make GET /storage/backups read-only; move reconcile/prune to a job | F3 | new | Shipped |
| Next | Renew the lease at phase boundaries; TTL from STEP_DEADLINE_MS | F5 | J4 adjacent | Next |
| Next | Gate / fold BackupWatch before enabling it by default | F7 | new | Next |
| Next | Coalesce ingress status writes | F8 | new | Next |
| Next | Resolve SandboxV1: remove the deploy surface or finish the cutover | F9 | new | Next |
| Later | Checkpointed steps, DLQ, pushed telemetry, durable billing ledger | — | P0 #3, P1, P2 | Later |
Rationale for the order: F1 and F2 are the only findings that combine silent correctness loss with fleet-wide blast radius; F4/F6 are small diffs that remove the “last writer wins” and “silent truncation” classes before they are exercised; F3 shipped by splitting the catalog read from an explicit, CAS-guarded reconcile; F5 is the remaining place where a timer takes on write authority and wants a design note before code.
5. Reconciliation with the companion audit
| This doc | Companion ID | Relationship |
|---|---|---|
| F1 | §S2 / S1 / P0 #2 | Same defect; this adds R9 framing + the queue consumer shape. |
| F2 | — | New: jitter/blast radius. Adjacent to P1 #9 (cadence), different fix. |
| F3 | — | New: the backups read path; §S3 is health/startup polling. |
| F4 | — | New: catalog handle CAS. J4 covers lease/orphan recovery, not this write. |
| F5 | J4 adjacent | New mechanism: timer-only renewal vs a 60 s TTL. |
| F6 | — | New: R2 pagination (S1 is KV). |
| F7 | — | New: BackupWatch cadence and container RPCs. |
| F8 | — | New: ingress write amplification; R14/A5 cover fleet aggregates. |
| F9 | — | New: unused deploy surface. |
| F10 | — | New: documentation. |
Nothing in the companion is contradicted here. Where both touch the same code (F1, F5) the companion’s roadmap entry stays authoritative for scheduling; this doc only adds the rule citation and the implementation shape.
6. Verification delivered for the P0 slice
Per the repository covenant, each change needed its corpus, tests and docs in the same PR. What this slice actually ships:
- Corpus:
test/business-rules/operational-modes.txt(fan-out pagination, queue delivery/retry, jitter, catalog CAS) andtenant-isolation.txt(the sweep resolves each tenant at delivery time and never crosses slugs). - Unit tests:
workers/fouria-dispatch/test/unit/control-queue.test.ts(KV pagination, batching, jitter bounds and determinism, live re-resolution, ack/retry, fallback concurrency) and the extendedcron-fanout.test.ts; inapps/fouria,src/persistence.test.tscovers CAS conflicts/retry exhaustion, mutation-shaped prune, create-only handles and paginated R2 listing. - Docs:
workers/fouria-dispatch/README.md(sweep semantics, queue, jitter and theCONTROL_QUEUEbinding),apps/fouria/docs/WORKER-DEVELOPMENT.md(CAS catalog writes andlistAllObjects). - Changeset: both touched packages (
fouria,fouria-dispatch) are CalVer andignored by Changesets, so the PR carries an empty changeset whose body records the change.
Remaining findings keep the obligations recorded for their own PRs: reconcile ownership
in apps/fouria/docs/OPERATOR-GUIDE.md, lease renewal tests under src/activity/, and
docs/TESTING.md if a new E2E covers the fan-out.
7. Explicit non-recommendations
- Do not move the boot back into
waitUntil()— that regression is documented and was the cause of a truncated restore. - Do not add a global scheduler DO to own per-tenant alarms. It would recreate the singleton bottleneck R1 forbids; pagination + a queue is the correct fix for the fleet sweep.
- Do not make every alarm a
Workflow. WFP does not support Workflows; the existing step cursor already gives resumability without a second execution model. - Do not add multi-region container replicas. The OpenClaw state DB is single-writer; availability is an RTO problem (companion §3.2).
- Do not remove the
checkpointcontrol action’s synchronous contract. It is the one place where waiting for a verified archive is the point.