Durable jobs, availability and scalability audit
On this page
Date: 2026-10-22
Scope: apps/fouria (tenant Worker), workers/fouria-dispatch, and the Lerma touchpoints that drive them.
Method: code read of the Durable Object job model, the gateway/sandbox lifecycle, the queue/cron entry points, and the dispatch routing path; cross-checked against current Cloudflare platform limits (Durable Objects, Workflows).
This is a findings + roadmap document, not an implementation plan approval. Nothing here has been measured on a live tenant unless stated.
Implementation status (2026-10-22): findings J1/J2/J4 and P0 item 4 are implemented. Durable jobs now run one bounded step per alarm with a persisted cursor (
apps/fouria/src/activity/job-steps.ts, phasedprepareSnapshot/archiveSnapshot/finalizeSnapshot), andBACKUP_INTERVAL_MINUTESis a floor stretched to2 × the last observed snapshot duration(resolveEffectiveIntervalMs). The activity alarm now enqueues a scheduled snapshot instead of snapshotting inline. The remaining overrun risk (the unboundedarchivestep) is evaluated in snapshot-step-budget.md. Still open: P0 1–3 (routing stale fallback, KV pagination/queue fan-out, durable billing ledger), and the P1/P2 items.
1. Architecture as built today
Browser ──► Dispatch Worker (auth, tenant resolve) ──► per-tenant User Worker (WFP script)
│
├── Sandbox DO (legacy SDK, idle sleep override)
│ └── container: openclaw gateway (single owner)
├── ActivityScheduler DO (SQLite)
│ ├── automation clock (single alarm)
│ ├── boot job ┐
│ ├── backup job ├─ one shared alarm,
│ └── maintenance job ┘ executed inline
├── BackupWatch DO (opt-in change scan)
└── TelegramIngress DO (rate limit + status)
Queues: per-tenant telegram-ingress (tenant is its own consumer), fouria-tasks, fouria-billing
Crons: tenant worker */2 (billable drain) + 0 */8 * * 1-5; dispatch */15 (backup fan-out)
Storage: R2 BACKUP_BUCKET (archives, configs, markers, advisory lease)
What is already good (keep it):
- The boot is a durable job, not a
waitUntil()continuation. This was the right diagnosis for the cancelled-restore class of bug. - Long jobs are committed before the request is acknowledged (
enqueueBackupJob/requestBoot/enqueueMaintenanceJob), with leases and idempotency keys. - The gateway is a single-owner process guarded by a container-wide
flock; the alarm is deliberately separated from the SDKSandboxDO so it cannot stompsleepAfter. - Restores are fenced by a maintenance window + advisory lease, and only durable/background owners may run them.
- Per-tenant queues are correctly per-tenant (
Naming.telegram_ingress_queue_name/2), the tenant script is its own consumer. - Structured events (
boot-job.*,maintenance-job.*,gateway.start.*) plus lerma OTLP ingestion already give a good incident trail.
The recommendations below are about the shape of that job model and the failure domains around it, not about rewriting it.
2. Long-running jobs
2.1 One DO, one alarm, four job classes executed inline — the core constraint
ActivityScheduler.alarm() (apps/fouria/src/activity/scheduler.ts:243-306) runs, sequentially, in a single invocation:
- the boot job (
restoreIfNeeded→launchRestoredState), - the backup job (stage →
createBackup→ config capture → prune; or restore → launch), - the maintenance job (
doctor --fix, SQLite integrity), - the automation wake/refresh.
Platform budget for that invocation (verified against current docs):
| Limit | Value |
|---|---|
| DO alarm handler wall clock | 15 minutes |
| DO CPU per invocation | 30 s default, 5 min if limits.cpu_ms is raised |
| Cron trigger / queue consumer wall clock | 15 minutes |
And the tenant’s own bounds:
| Constant | Value | Source |
|---|---|---|
STARTUP_TIMEOUT_MS | 300 s | src/config.ts:11 |
BOOT_JOB_LEASE_MS | 14 min | src/activity/boot-job.ts:23 |
BACKUP_JOB_LEASE_MS | 14 min | src/activity/backup-jobs.ts:16 |
MAINTENANCE_JOB_LEASE_MS | 14 min | src/activity/maintenance-jobs.ts:19 |
RESTORE_BACKUP_TIMEOUT_MS | 180 s | src/config.ts:109 |
DEFAULT_LEGACY_STAGE_TIMEOUT_MS | 600 s | src/persistence.ts:226 |
BACKUP_INTERVAL_MINUTES | 5 | wrangler.jsonc:236 |
Finding J1 (P0). A single alarm can exceed its own 15-minute wall limit.
A scheduled snapshot on a large tenant is stage (≤10 min) + createBackup + config capture + prune. Add the automation wake (which itself calls scheduleBackupIfNeeded) and a doctor-repair queued behind it, and the invocation can pass 15 minutes. The platform kills the alarm; the job stays running until its 14-minute lease expires, then retries from the beginning (re-stage, re-upload). That matches the orphaned-archive incident that motivated reconcileOrphanedBackups — the reconciliation is treating a symptom of the execution model.
Finding J2 (P0). Head-of-line blocking between unrelated jobs.
Because boot → backup → maintenance are sequential on one alarm, a 14-minute boot lease delays a queued doctor-repair by up to 14 minutes (this is exactly the incident already documented in docs/plans and in maintenance-jobs). Generation-guarded arming fixed the stale arm bug, but the serialization is structural.
Finding J3 (P1). No in-job checkpointing.
Every job is “all-or-nothing per attempt” with a bounded attempt count (3). A cold boot that legitimately takes ~14 minutes (documented on staging-client-dev) can burn leases and attempts, ending in status: failed with no partial progress. The only progress that survives is the R2 artifact and restore-target.json; there is no “phase cursor”.
Finding J4 (P1). Backup cadence is shorter than the backup’s worst-case duration.
BACKUP_INTERVAL_MINUTES=5 with a 10-minute stage means a large tenant is snapshotting almost continuously. Two consequences: (a) automation wake windows are starved because the alarm is busy; (b) scheduleBackupIfNeeded mostly returns interval-not-elapsed / lock-busy, so the configured cadence is fiction. The interval should be derived from the observed stage duration (e.g. max(BACKUP_INTERVAL_MINUTES, 3 × p95 stage)), or the cadence should move to a change-driven model (which BackupWatch already implements) with the interval as a floor.
Finding J5 (P2). The schedule mirror is a 10–15 s CLI fork every 5 minutes.
readAutomationSchedule() shells openclaw automations list --json (src/activity/refresh.ts:27-38) on every warm path, throttled to 5 minutes. It spawns a process inside the container each time, contributes to the process-count noise the repo already fights, and adds latency to /api/health. The gateway exposes automation/cron state over the same admin HTTP RPC the rest of the dashboard migrated to; the mirror should use that (or be event-driven: the automation mutation path already knows when the schedule changed).
2.2 Recommended job-execution model
Option A — checkpointed job steps (low risk, stays inside WFP). Keep the DO as the durable queue and authority, but make each alarm invocation execute at most one bounded step, persisting a cursor:
type JobStep = 'stage' | 'archive' | 'encrypt' | 'commit' | 'capture-config' | 'prune';
// each alarm: run one step, persist `nextStep`, arm the alarm immediately
// each step gets its own timeout budget, well under the 15-minute alarm wall
- Bounded alarms (each step ≤ ~5 min) → no platform kill, no lease-expiry retry from zero.
- Resumable across DO eviction, deploy cutover, and
OperationInterruptedError. - Per-step durations become a first-class observability signal (the
boot-job.*/maintenance-job.*structured events can carrystep+durationMs). - No platform capability needed; works in a Workers-for-Platforms namespace.
- Also lets J2 be fixed properly: the alarm runs one step of one job, then re-arms with the earliest of (next step, other job’s due time) — a fair round-robin instead of strict priority.
Option B — control-plane Workflows executor (strategic).
Workflows give native per-step retries/backoff/timeouts, step.sleep, per-instance state, and an operations dashboard. Constraint that decides the design: Workflows cannot be deployed to Workers-for-Platforms namespaces, and every tenant is a WFP script. So Workflows can only be used by a shared, non-WFP control-plane Worker that drives tenant work over the dispatch binding:
Control-plane Worker (normal script, not WFP)
└── Workflow instance id = tenant slug + job id (idempotent)
step: POST /control { op: 'snapshot', phase: 'stage' } → tenant Worker → sandbox
step: POST /control { op: 'snapshot', phase: 'archive' }
step: POST /control { op: 'snapshot', phase: 'commit' }
This is attractive because the dispatch Worker is already that control plane (cron fan-out, freeze/unfreeze, backup actions), it removes the 15-minute cap, and it gives retry semantics without hand-rolled leases. Cost: one extra hop per step, a new deployable, and the tenant still needs a local claim/lease so a stray manual call cannot double-run a phase.
Recommendation: A now (it is a contained refactor of activity/*-jobs.ts + persistence.ts step boundaries), B as the target once there is more than a handful of tenants, because B also fixes the dispatch cron fan-out (see S2) and the “one alarm per tenant” ceiling.
Do not adopt the Agents SDK merely for runFiber()/queue here: the durable-execution pieces are DO-based and would duplicate the state machines already written and tested, while the parts that would actually help (Workflows, hibernatable sockets) are separable.
3. Availability
3.1 Dispatch routing depends on Supabase on a 60-second cache
resolveTenant() (workers/fouria-dispatch/src/index.ts:110-158) reads KV, and on a miss queries Supabase, then caches for expirationTtl: 60. A Supabase error returns null → the request 404s/denies.
Finding A1 (P0). A Supabase blip becomes a full routing outage within 60 seconds. Every tenant’s KV entry expires a minute after it was written. There is no stale-while-revalidate, no negative cache, and no “last known good” fallback.
Fix, in order of value:
- Serve stale on failure. Two keys per tenant:
slug:<slug>(short TTL, refreshed) andslug:<slug>:last-good(no TTL, written on every successful resolve). On miss/error, fall back to last-good and log arouting.stale-fallbackevent. - Lengthen the hot TTL (5–15 min) and refresh in
waitUntil()so latency is off the request path. - Negative-cache 404s briefly (10–30 s) so a typo’d or unknown host does not hammer Supabase.
- Longer term, make the tenant directory authoritative in a DO (or D1) written by the lerma provisioner on every state change, with KV/Supabase as the rebuild source. Routing should not need a relational query on the hot path at all.
3.2 Single container, single region, restore-dominated RTO
containers.constraints.regions: ["WEUR"](prodmax_instances: 5,instance_type: basic). A single tenant has exactly one container; there is no replica, and the OpenClaw state DB is single-writer by design, so this is not fixable with horizontal redundancy — availability has to be expressed as RTO, not uptime of a redundant pair.- RTO is dominated by
sandbox.restoreBackup()from R2 (a 350 MB archive is the observed order of magnitude). Everything else (container admission ≈110 s, setup, gateway bootstrap) is bounded and already instrumented.
Actions that materially reduce RTO:
- Publish an RTO budget per tenant class and alert on breach (boot job duration p95, restore duration p95). Today the numbers exist in events but not as an SLO.
- Do not restore on every cold start.
restoreIfNeededalready short-circuits when the container still holds tenant state — protect that with tests, because it is the single biggest RTO lever. - Keep the archive small. The legacy-bridge symlink exclusion and the
node:sqliteonline backup were the right moves; add an explicit archive-size metric and a soft ceiling in the snapshot path. - Warm the recovery point. After a restore, optionally re-hydrate the next likely archive location/metadata so a second restore is not cold in R2.
- For paid plans, consider a larger
instance_typefor heavy tenants: restore and stage are CPU/IO bound in the container too.
3.3 Deploys interrupt in-flight durable work
A script version update resets DO↔container control sessions (the repo already classifies OperationInterruptedError / RuntimeControlProtocolError). Today this is mitigated by resilience (waitForPortResilient, transport-drop reattach, stale-container recovery) but not by placement:
Finding A2 (P1). Rollouts are N per-tenant deploys with no gradual rollout or drain.
Recommendation: deploy via the dispatch namespace’s gradual rollout percentage, and add a deploy-window guard: while a boot/restore/deprovision job is running, defer the tenant’s script update (lerma already sequences per-tenant steps, so this is a check, not new machinery). Combined with Option A checkpointing, an interrupted job resumes instead of restarting.
3.4 Billing data can be lost (and double-counted) across isolates
storeNormalizedTraces keeps buffered OTel traces in a module-level Map (src/otel-traces.ts:157, cap 5000), drained by the */2 cron. emitBillableActions dedupes with a module-level Set (src/billable-actions.ts:163). Both are per-isolate and vanish on eviction/deploy.
Finding A3 (P0, revenue correctness).
- A trace buffered at 12:01 and evicted at 12:02 is lost → under-billing.
- Two isolates draining overlapping batches each have an empty
emittedActionIds, so both claim the sameaction_idand both enqueue it toBILLING_QUEUE→ potential duplicate rows. Reconciliation runs before the claim, so it cannot see a sibling isolate’s claim. Verify whether the lerma-side consumer dedupes onaction_id; do not assume it does.
Fix: persist the buffer and the dedup ledger durably (the tenant ActivityScheduler DO already exists; a small billing table, or a KV/D1 ledger keyed by action_id with TTL). Emit to BILLING_QUEUE with the action_id as the idempotency key and make the consumer dedupe on it, rather than relying on process memory for the claim.
3.5 Queues have no dead-letter path
wrangler.jsonc consumers set max_retries (5 tasks / 10 telegram) with no dead_letter_queue. A Telegram update that fails 10 times is silently dropped; the dashboard records the last 20 events but the message is unrecoverable.
Finding A4 (P1). Add a DLQ per queue (or one shared), surface DLQ depth in lerma, and make TelegramIngress status report dead-lettered distinctly from failed.
3.6 Lerma’s health polling is the availability observability, and it is a poller
Lerma probes every tenant (?passive=1), one HTTP round trip each. That is fine at 10 tenants, a scaling and single-point problem at 1000, and it is now redundant: issue #337 already lands tenant OTLP spans in lerma with an uptime endpoint.
Finding A5 (P1). Treat polling as a fallback and derive availability from pushed telemetry (invocation success + a lightweight heartbeat event), keeping active probes for tenants that are silent. That both removes a fan-out bottleneck and makes uptime survive a lerma restart.
4. Scalability
S1. ActivityScheduler is a per-tenant singleton, and that is fine — until jobs are long
One DO per tenant scales horizontally by tenant count (DOs are designed for that). The ceiling is within a tenant: one alarm, one job at a time, 15 minutes. Fixing J1/J2/J3 (bounded, checkpointed steps) removes the practical ceiling without changing the topology.
S2. Dispatch cron fan-out does not paginate and is unbounded
workers/fouria-dispatch/src/index.ts:869-903:
const tenantList = await env.TENANT_INDEX.list({ prefix: 'slug:' }); // ← first page only
const tenantEntries = await Promise.all(tenantList.keys.map(...)); // ← one get() per tenant
for (...) ctx.waitUntil(forwardTenantControl(env.DISPATCHER, entry, 'backup'));
Finding S1 (P0 for scale, and a silent correctness bug today).
KV list returns at most 1000 keys per page; there is no cursor loop. Past 1000 tenants, the remainder never get a scheduled backup — and nothing errors. Additionally, Promise.all of one KV get per tenant plus one waitUntil fetch per tenant has no concurrency bound, and the cron’s 15-minute wall will be exhausted long before that completes at scale.
Fix: page the KV list (cursor), then fan out through a Queue (one message per tenant, consumer with max_concurrency and retries). The tenant’s durable job queue already makes each delivery idempotent, so at-least-once fan-out is safe. This also gives a natural place for the per-tenant back-pressure the current design lacks.
S3. Health/WS streams are polling loops in the Worker
/api/health-ws, /api/startup/ws, and the maintenance/telegram status sockets each hold a Worker WebSocket open and re-probe container state on an interval (src/routes/public.ts:620-750). Every tick is a sandbox exec/listProcesses RPC — the documented control-channel churn source. Billing is duration-based while the socket is open.
Finding S2 (P1). Push instead of poll: have the DO that owns the state (boot job / restore progress / maintenance job) push on change. A hibernatable WebSocket (DO WebSocket hibernation, or an Agent-style connection) would remove both the polling RPC traffic and most of the duration cost. The startup stream is the best first candidate because it already reads state the DO owns (restoreProgress, bootJob).
S4. Cost/observability ingress is per-tenant, but the readers are central
Analytics Engine writes are fine. The readers (/api/admin/costs/v2/summary per tenant via AE SQL, lerma’s aggregation) will need caching/rollup as tenant count grows; per-tenant AE SQL on every dashboard load is a rate-limit risk. Recommend a shared rollup table in lerma (it already has observability_events + a pruner) rather than N live queries.
5. Prioritized roadmap
P0 — correctness / outage vectors (do first, small diffs)
| # | Change | Why | Where |
|---|---|---|---|
| 1 | KV last-good + stale fallback + longer TTL + negative cache on resolveTenant | Supabase blip currently breaks all routing in ≤60 s | workers/fouria-dispatch/src/index.ts:110-158 |
| 2 | Paginate KV list + move cron fan-out to a Queue with bounded concurrency | Tenants >1000 silently get no backups | workers/fouria-dispatch/src/index.ts:869-903 |
| 3 | Persist billing trace buffer + dedup ledger (DO/D1/KV with TTL) | In-memory buffer loses and double-counts revenue events | src/otel-traces.ts:157, src/billable-actions.ts:163 |
| 4 | Bound the alarm: one step per invocation, or assert every job path fits <15 min and fail loudly if not | Alarm kills are the root cause of orphaned archives/leases | src/activity/scheduler.ts:243-306 |
P1 — resilience and fairness
| # | Change | Why |
|---|---|---|
| 5 | Checkpointed job steps (cursor persisted per job) | Resume across eviction/deploy instead of retry-from-zero; kills head-of-line blocking |
| 6 | DLQ for every queue + surfaced depth | Silent message loss today |
| 7 | Derive availability from pushed telemetry; keep polling as fallback | Removes lerma’s O(N) hot-path dependency |
| 8 | Deploy-window guard + gradual rollout for tenant scripts | In-flight jobs survive code updates |
| 9 | Align BACKUP_INTERVAL_MINUTES with observed stage p95 (or make change-watch the primary trigger) | The 5-minute cadence cannot be met on large tenants and starves automation wakes |
| 10 | Boot/job failure alerting to lerma (boot.status=failed after 3 attempts is currently only visible on the loading page) | A failed boot is silent at the platform level |
P2 — scale-out and control plane
| # | Change | Why |
|---|---|---|
| 11 | Control-plane Workflows executor (non-WFP) driving tenant phase endpoints | Removes the 15-min cap, gives native step retries/observability; also hosts fan-out |
| 12 | Push-based state streams (hibernatable WS) for startup/health/maintenance | Cuts polling RPC churn and WS duration billing |
| 13 | Replace the 5-minute automations list CLI fork with admin RPC or event-driven mirror | Latency, process noise, control-channel load |
| 14 | Durable tenant directory (DO/D1) as routing authority, Supabase as rebuild source | Removes relational query from the hot path permanently |
| 15 | Per-tenant AE cost rollups instead of live AE SQL per dashboard load | Rate limits and latency at tenant scale |
6. Concrete design sketch: checkpointed job steps
This is the change that unlocks items 4, 5, 8 and most of J1–J3.
// src/activity/job-runner.ts
export interface JobCursor {
jobId: string;
operation: 'snapshot' | 'restore' | 'maintenance' | 'boot';
step: string; // 'stage' | 'archive' | 'commit' | 'capture' | 'prune' | ...
stepStartedAt: number;
steps: Record<string, { startedAt: number; finishedAt?: number; error?: string }>;
}
alarm()reads the active job, runs exactly one step with its own timeout, persists the cursor, and re-arms:setAlarm(min(nextStepEta, earliestOtherJobDue)).- A step that throws is retried at the step level (
attemptsper step, not per job); the job only fails when a step exhausts its budget. - The existing
BackupJobRecordkeeps its public shape (status,detail,resultId) so the dashboard and websocket job views keep working;cursoris additive. - Lease semantics simplify: the lease now covers one step, so a stale
runningrecord resumes at the cursor instead of replaying the operation. - Because each step is short, the 15-minute alarm wall stops being a design constraint and
BOOT_JOB_LEASE_MS = 14 mincan drop to something meaningful. - Tests:
src/activity/*-jobs.test.tsalready inject storage/clock — add “killed between steps resumes at cursor”, “step retry does not increment job attempts”, “two jobs round-robin instead of blocking”. Corpus:test/business-rules/operational-modes.txt.
Exit criteria for this being “done”: on staging, force-kill the DO mid-archive and confirm the next alarm resumes at encrypt/commit (no second upload); enqueue a doctor-repair behind a 10-minute snapshot and confirm it starts within one alarm tick of the snapshot’s final step.
7. Explicit non-recommendations
- Do not add multi-region container failover. The OpenClaw state DB is single-writer; a second writer is a corruption risk, and Sandbox does not offer hot replicas. Invest in RTO instead.
- Do not move the boot into
waitUntil()again. That regression class is well documented; the DO job is correct. - Do not replace the DO queues with Workflows per tenant. WFP does not support Workflows; any Workflows design must live in a shared control plane.
- Do not introduce the Agents SDK wholesale. The useful pieces (Workflows, hibernatable sockets) are available without adopting an agent runtime that duplicates the tested state machines.
8. Verification / documentation obligations
If these become PRs, the repo covenant applies:
test/business-rules/operational-modes.txt(job lifecycle, deferral, stale-recovery),tenant-isolation.txt(routing/directory),dev-expense.txt(billing emission).- Tests:
src/activity/*,src/routes/public.test.ts,workers/fouria-dispatch/test/unit/*,apps/lerma/test/lerma/provisioning/*. - Docs:
apps/fouria/docs/WORKER-DEVELOPMENT.md,apps/fouria/docs/OPERATOR-GUIDE.md,docs/ENV-VARIABLES.md,workers/fouria-dispatch/README.md. - A changeset in every PR (
bun run changeset/--emptyfor docs-only).