F FourIA latest 2026.10.29 GitHub ↗

Legacy Sandbox: retain tenant state across container replacement

On this page

Status: in progress — Phase 1 delivered; live quiesce/snapshot validated on staging dev, same-archive restore after replacement pending. This plan hardens the serving legacy runtime independently of the parallel SandboxV1 migration.

  • Delivered (Phase 1): (a) fail-closed restore — an initialized tenant with a recorded recovery point never boots empty; (b) the cross-isolate restore skip now probes the container’s actual state instead of isRuntimeActive(), so a fresh/replaced container always restores rather than booting empty (containerHasState in apps/fouria/src/persistence.ts).
  • Delivered (Phase 2, partial step 2): a synchronous, verified pre-replacement checkpoint (X-FourIA-Control: checkpoint) wired across Worker → dispatch → Lerma. Both upgrade paths checkpoint the running tenant before the image rollout. The checkpoint step itself is best-effort in Lerma (a control-plane failure is logged and the upgrade proceeds, so it can never brick a valid upgrade); the snapshot archive verification inside the Worker stays fail-closed. The shared control-plane coordination record (prerequisite 2a) and full cross-plane fencing for concurrent/out-of-band replacement remain gated.
  • Deployed and live-validated (Phase 3, quiesce): the checkpoint now quiesces the tenant first — a durable, cross-isolate maintenance window (src/maintenance.ts) plus a gateway stop before snapshotting, so the archive is not raced by the gateway’s own writes; ensureGateway refuses to restart the gateway while the window is open, and the window self-heals on expiry. Service is always restored in a finally. This does not yet drain through the rollout (writes between checkpoint and replacement remain outside the archive). The window/gateway-stop guards are best-effort: only the snapshot itself is fail-closed, so a broken guard can never block an upgrade (hardened after a staging incident — see PR #408/#409/#412). On 2026-10-09 at 11:06 UTC, a direct checkpoint against the running tenant-client-dev returned backupId=7d0b2ae8-ea9d-4286-9770-883e2b355a8a, quiesced: true, and maintenanceWindow: true. Observability confirmed killGateway completed, a 28.45 MB F4E1-encrypted archive was created, and the Gateway health event returned ok: true afterward. This validates the live quiesce/snapshot path, but the direct checkpoint did not replace the container, so restoration of this same archive after replacement is still pending.
  • Delivered (control-plane reachability): the Lerma→dispatch internal path (/_control/*, /_invalidate/*) is reachable past Cloudflare Access via path-scoped bypass apps (apps/fouria/iac/access.tf, same posture as the Lerma /api/* CI bypass), and the Lerma deploy sends INTERNAL_DISPATCH_INVALIDATE_TOKEN (PR #413). Without both, the checkpoint step could never run on staging — this was the actual cause of the staging promotion failures, not the quiesce.
  • Other staging evidence (2026-10-09): the control-plane Access bypass and bearer fix allowed tenant-client* to converge on unstable-310e519 and unstable-85bd702; the 0a235513 archive was restored successfully. The 10:43 promotion skipped its checkpoint for both instances because their runtimes were not active (skipped: not-running), which is expected and non-waking. The earlier 0f2f467f archive was created and encrypted; the observed restore was for 0a235513, not 0f2f467f.
  • Pending, gated: the shared maintenance/replacement fence (step 2, needs the control-plane coordination record), durable primary storage (step 3, feasibility gate), migration (step 4), recover-before-run across all launch paths (step 5), supervision/observability (step 6) and destructive acceptance (step 7).

No dependency, infrastructure, deployment, or tenant data has been changed by the pending steps. Full compliance requires a synchronous durability boundary; the Phase 1 guard prevents silent empty boots but does not by itself make every acknowledged write durable across unexpected container loss.

Outcome and non-negotiable acceptance criteria

Containers, gateway connections, and in-flight sessions may stop. Previously saved tenant state must not disappear when an image is upgraded, a container sleeps, is frozen, is replaced, or becomes unavailable.

  1. Planned replacement: persist every completed mutation before replacing the container. Block replacement if preservation cannot be proved.
  2. Unexpected container loss: every mutation acknowledged as saved must already be durable outside that container (zero recovery-point loss).
  3. Unavailable storage: stop accepting mutations rather than acknowledge writes into an ephemeral fallback directory.
  4. Recovery: an existing tenant never silently starts empty or selects an older recovery generation. Report an error and keep it unavailable instead.
  5. Single owner: at most one gateway may mutate a tenant’s state. A timed-out RPC or an expired lease alone is not proof that the old gateway stopped.
  6. Isolation and encryption: preserve tenant boundaries and existing application-level encryption requirements, not merely provider encryption.
  7. Migration independence: keep ACTIVE_SANDBOX_RUNTIME = 'legacy', the matched legacy SDK/base-image versions, and the v1 scaffold unchanged. Application-image changes are expected for these fixes; an OpenClaw fork or upstream persistence change requires separate approval, not an SDK cutover.

“Saved” must mean a successfully completed durable file operation or database transaction, not merely an in-memory change. Persisted conversation history, sessions, agents, credentials, pairing, schedules, memory, custom plugins, and workspace artifacts are included. Live sockets, running processes, and unfinished writes are not. A process killed halfway through a mutation may leave that mutation uncommitted, but must not erase previously committed state. Audit OpenClaw’s actual success responses: calling a local write() or returning a SQLite commit with asynchronous durability is not enough. If the application acknowledges before persistence, change that boundary rather than redefining those acknowledged writes as expendable.

Evidence from the current repository

SurfaceFindingConsequence
Root package.json catalog; apps/fouria/DockerfileServing alias is @cloudflare/sandbox-legacy -> 0.13.0-next.776.1, paired with the same image. The separate package is @cloudflare/sandbox@1.0.0.Although called legacy/v0.x, this is a preview API: do not copy stable string-exec/startProcess examples.
apps/fouria/src/sandbox-runtime.tsAll tenants still select legacy.Fixes cannot depend on an unfinished v1 cutover.
apps/lerma/lib/lerma/provisioning/tenant_provisioner.ex, @upgrade_pipelineFetch -> deploy -> mark updated; no persistence preparation step.Image replacement is not fenced by a final preservation barrier.
apps/lerma/lib/lerma/instances.ex, do_upgradeAnother upgrade entry point also directly deploys.Fixing only the provisioning pipeline leaves a bypass.
apps/lerma/lib/lerma/cloudflare/containers.exImage/type changes and restarts use container rollouts.All replacement paths need the same preservation policy.
apps/fouria/src/index.ts, freeze controlCalls sandbox.destroy() without a final checkpoint.Recent state can be lost on an intentional stop.
apps/fouria/src/sandbox-do.ts, idle expiryCalls this.stop() without a final checkpoint.Non-billable tenants can lose recent state on sleep.
apps/fouria/src/gateway/sandbox-recovery.tsRecovery destroys a runtime judged stale.Unexpected loss cannot be repaired by copying data after it is gone.
apps/fouria/src/persistence.ts, restoreIfNeededRuntime liveness is used as a cross-isolate restore-completion signal. Empty/unrestorable catalogs may return a clean start; older candidates may be tried.A newly started SDK container is not evidence that tenant state was recovered, and fallback can silently discard newer state.
apps/fouria/Dockerfile; persistence.tsReal legacy roots /root/openclaw/.openclaw and /root/clawd are outside /home/openclaw, requiring a stage/materialization bridge.Enumerate real paths; a mount or backup of only /home/openclaw is insufficient.
persistence.ts, legacy stageDatabase images precede tree copying; existing stage is overwritten rather than replaced.Individual SQLite backups are consistent, but cross-file consistency and deletion fidelity need explicit tests.

These are source-level risks, not a claim that every reported tenant loss has been reproduced or that live Cloudflare configuration has been inspected.

What the requested Cloudflare guides establish

  • Sandbox lifecycle: a stable sandbox ID/DO does not make its container filesystem durable. Sleep and destruction discard that filesystem.
  • Storage API and persistent-storage tutorial: mount an external data directory early and remount it on each fresh container; data already persisted there survives destruction. Mounts overlay existing image contents rather than merging them.
  • File watching: watchers are streams, end on sleep/shutdown, and must be re-established. The guide restricts watched paths to /workspace. A watcher is not a durable journal and cannot save a change after the container has already disappeared.
  • Background processes: a container-owned monitor can outlive an individual HTTP request, but not the container. On our pin launch it through startShell/argv exec, keep it separate from the child-free setup/post-start waits, and supervise it.
  • Mount-buckets guide: credential-less R2 binding mounts require ContainerProxy, already exported by this Worker. Local localBucket uses periodic sync, unlike production mounts, so local tests alone cannot establish production durability.
  • Cloudflare FUSE example and s3fs limitations: an object-store mount does not provide full POSIX semantics; s3fs has neither atomic renames nor coordination between mounting clients. Do not put the entire live OpenClaw state/SQLite tree on s3fs and call it safe.

The installed legacy types confirm mountBucket, watch, createBackup, and argv exec returning a process handle. They do not expose the stable guide’s startProcess API. Use apps/fouria/src/gateway/sandbox-exec.ts for execution.

One active gateway, durable storage as the primary source of truth, snapshots as disaster-recovery history. More replicas are not needed for this criterion.

One coordination record shared by both control planes

Use the existing shared PostgreSQL source of truth: propose a tenant_runtime_operations table keyed by immutable instance ID, with operation ID, monotonically increasing owner epoch, phase, source runtime/image, required recovery generation and target storage mode. Reserve/advance operations using transactions and conditional updates (UPDATE ... WHERE owner_epoch = ... AND phase = ... RETURNING ...), with one active operation per instance. R2 holds immutable exports/manifests; PostgreSQL publishes the selected generation only after its objects are verified. Pin an operation’s objects against retention from upload through publication/recovery acknowledgement, and reverify after an interrupted publication. Do not imply a transaction spans R2 and SQL.

Both Lerma and the tenant Worker must consult this record before intentional stop/replace or launch; use a reviewed least-privilege Worker PostgreSQL access path, never Lerma’s HTTP API or database credentials inside the container. Inventory each entry point against this gate. Persist cloud API intent and operation identity before issuing a rollout; reconcile ambiguous responses and completion rather than letting a lease timeout authorize a competing operation. An owner epoch coordinates the controller but only fences actual writes if the storage backend rejects stale owners, or the old runtime is proved terminated.

Separate three classes of storage:

ClassPlacementRule
Tenant runtime state, SQLite databases/journals, transcripts, custom files, workspace and repository metadataA qualified durable filesystem, or synchronous application-level storesAll saved changes reach remote durability before success. No local-only authoritative copy.
External documents, uploads, completed artifacts with object semanticsTenant-scoped R2 through existing bridges or a qualified mountBucket pathTreat remote object commit, not a queued copy or local cache write, as completion. Publish multi-object changes through immutable generations plus a committed manifest if needed.
Image-provided dependencies, rebuildable indexes/caches, temporary execution files, live socketsContainer/imageExplicitly classified as rebuildable/transient; exclude only with evidence.

Storage feasibility gate: do not pretend s3fs solves SQLite

The smallest application-change candidate to evaluate, not automatically adopt, is a POSIX-oriented FUSE filesystem such as JuiceFS, backed by R2 data and a separately durable PostgreSQL metadata store. Cloudflare supports FUSE; the filesystem adds atomic metadata operations and locking absent from s3fs.

Relevant upstream evidence:

  • JuiceFS cache/write semantics: without client --writeback, data is uploaded before metadata publication; fsync/fdatasync/close trigger flush. A successful write alone still does not prove persistence.
  • POSIX compatibility: atomic metadata operations, mmap, flock, and traditional record locks are documented. This is not proof that our OpenClaw/SQLite configuration is safe on Cloudflare’s actual kernel and runtime.
  • Metadata engines: PostgreSQL is supported. Metadata is part of the authoritative data, not a disposable cache; its durability/failover guarantees matter as much as R2.

Do not enable delayed uploads, client writeback, or use an ephemeral metadata database. Audit every SQLite connection’s journal and synchronous settings, including plugins: changing a PRAGMA in an unrelated CLI connection does not change the gateway’s connection. Prove WAL/shared-memory behavior with one owning container, or select and validate another supported journal mode.

This candidate adds infrastructure, latency, credential handling, licensing, and encryption questions. Approval and a successful production-like spike are required before adoption. Never give the agent platform PostgreSQL credentials, use a shared metadata namespace with tenant-supplied paths, or silently expose keys previously held only in the Worker. Qualify a supported credential proxy or an explicitly approved tenant-isolated access model; do not assume the SDK’s s3fs proxy transparently works for another client or PostgreSQL TCP. Application-level encryption of live data needs a design of its own: the F4E1 post-snapshot transform does not encrypt live filesystem writes. Candidate qualification must produce a tenant-keyed live-data encryption and key-custody design compatible with the current policy, or the candidate fails the gate. Specify metadata topology too: isolated volumes/catalogs and database roles per tenant, or a reviewed alternative with enforceable isolation and explicit corruption blast radius. A directory prefix alone is not metadata isolation.

If this gate fails: use synchronous application-level persistence for OpenClaw state (durable SQL/DO records and R2 objects, with changes to its write paths), or host the gateway on qualified persistent storage. Those are explicit architectural decisions, not silently included migrations. The supplied Sandbox APIs alone cannot guarantee zero loss for unmodified arbitrary local filesystem/SQLite writes. Do not replace the hard criterion with asynchronous file watching, rsync, or WAL shipping.

The application-level alternative may require an upstream OpenClaw change or a maintained fork. That is a significant, separately approved scope expansion with its own compatibility/release policy; it is not a ready-made off-ramp that can be implemented without changing the current application image.

Construction sequence (seven reviewable changes)

Actionable near-term proposal: 1-2. Feasibility study/decision: 3. Conditional implementation, not approved scope: 4-7, only if 3 passes and the resulting backend/application changes are approved. Split large steps into smaller PRs as needed, preserving their acceptance and activation gates.

Use small branches/PRs from develop; do not deploy automatically as part of planning. Each implementation PR updates its corresponding documentation and business-rules corpus. “Default” and “strongest” below are review/implementation complexity tiers, not a request to change a specific model.

1. Inventory state and reproduce loss

Context: legacy OpenClaw uses real /root roots plus /home aliases; snapshots stage them, and several replacement entry points exist. No primary durable filesystem is wired today.

Depends on: none. Tier: default implementation, strongest review.

  • Write tests first that save unique sentinels through real application paths: config, paired device, credentials, agent, conversation/transcript, automation, SQLite rows, workspace binaries, repositories, rename and deletion.
  • Inventory realpaths, all databases/journals, plugin-owned state, custom workspace paths, permissions, symlinks and cross-root references. Separate image dependencies from user-installed/custom plugin content.
  • Record every stop/replace path: both Lerma upgrade paths, image/type/restart rollouts, promotion/rescue scripts, freeze, idle expiry, debug restart/ reprovision, watchdog recovery, and out-of-band platform termination. Map every managed entry point to the shared PostgreSQL coordination gate.
  • Audit acknowledgement and flush semantics for API, WS, external channels, scheduled jobs, shell tools and plugins. A Worker-only response gate misses gateway-owned writes.
  • Record baseline hashes, DB queries, file ownership/modes and image identity; persist test expectations outside the disposable container.

Exit: failing regression(s) demonstrate the gap; a versioned storage inventory assigns every tenant state path to an authoritative store. No unclassified tenant state is treated as a cache.

Verification: targeted Vitest/ExUnit tests plus a disposable remote tenant reproduction using real R2, then the shared checks below.

Rollback: test/docs-only; no customer mutation.

2. Immediately fence planned destructive operations

Context: @upgrade_pipeline, Instances.do_upgrade, freeze and idle sleep currently lack a final preservation barrier. Existing async backup acceptance is not proof of completion.

Depends on: 1. Tier: strongest.

  • Prerequisite 2a, separately reviewed before enabling the guard: provision the new least-privilege Worker-to-PostgreSQL access path, coordination table, tenant-scoped grants and binding/network configuration. Direct Worker DB access is not wired today; do not present it as an existing facility. Exit this prerequisite only when both planes can conditionally reserve/read the same record, cross-tenant access is denied, and a DB outage fails closed. It requires infrastructure approval and its own tests/docs. Until it lands, operationally suspend unattended managed replacements rather than claim the guard exists; this containment does not prevent platform-originated loss.
  • Add a persisted per-tenant maintenance/replacement operation with ID, owner epoch, source image/runtime identity, phase and required recovery generation. Implement the shared PostgreSQL record and conditional-update protocol above, not module-local flags. Include cancellation and reconciliation after controller restart, and enforce the same record in both control planes.
  • Stop admission from proxy/admin mutations, new WS commands, queues, activity wake/heal and scheduled jobs. Quiesce external channel adapters too; closing web UI sockets alone is insufficient. Stop gateway/child writers while retaining the container. Verify they exited before exporting state.
  • Resolve in-flight mutations before certifying the export: drain or safely abort unfinished work, flush all previously acknowledged mutations, including open append-only transcripts and database journals, and verify writer exit. A forced termination that cannot prove preservation must fail the barrier; previously acknowledged writes cannot be reclassified as unfinished work.
  • Force a final complete, consistent export of all inventoried state. Verify archive bytes/checksums, manifest, decryption and DB integrity by restoring into an isolated validator that cannot connect channels or run automations. Include deletion fidelity, not just existence checks.
  • Permit rollout/destroy only after durable completion is recorded for this operation. Do not accept 202, an interval-skipped snapshot, a metadata HEAD, or any older restorable archive as the barrier.
  • Require the replacement to restore this exact verified generation before any gateway/CLI writer starts. Implement that minimal recover-before-run guard now, not only in step 5; empty boot, active-runtime shortcuts and older-backup fallback must not turn a safe export into a lossy upgrade.
  • Centralize policy across both upgrade paths, resize/restart, freeze, idle sleep and operator recovery. On preservation failure leave the old container intact and visibly unavailable; resume it only through a reconciled cancel.
  • Bootstrap this control release without replacing the old container: use an explicitly verified code-only deployment with unchanged legacy SDK/image and resolved resources. Use a Worker-code-only path that does not touch the container application, or explicitly suppress create_image_rollout only after proving the resolved image reference/digest and resources are unchanged. A reused mutable tag alone is not proof of identical image contents. Cloudflare may return resource numbers rather than an instance_type name; avoid an accidental rollout from comparing that absent string. Validate on a disposable tenant first. If code-only bootstrap cannot be proved, require an operator-controlled quiesced export before deployment.

Exit: no intentional replacement can execute without a fresh verified barrier; write/job races and operation retries are covered. This step protects planned upgrades, not unexpected loss before the final export.

Verification: targeted replacement/control/idle tests and old-image -> new image remote E2E, including export failure -> zero rollout API calls.

Rollback: keep the preservation guard; cancel maintenance explicitly. Do not roll back to an unguarded destroy path or older tenant state.

3. Qualify the synchronous primary storage backend

Context: periodic snapshots and watcher-driven copies have a loss window; plain R2/s3fs does not supply the full state-tree semantics OpenClaw needs. JuiceFS + R2 + durable PostgreSQL is only an evaluated candidate.

Depends on: 1; can run alongside 2 with disjoint spike/test files. Tier: strongest architecture/security/database review.

  • Build a disposable, production-like legacy-image spike. Do not change the production default SDK/image or select SandboxV1.
  • Prove mount readiness, atomic replace, file/directory persistence, symlink behavior, locks, concurrent gateway/CLI access, SQLite journal recovery, successful close/fsync durability and errors during storage outages.
  • Destroy the container immediately after successful saved operations without running a snapshot or a shutdown handler; remount with an empty local cache and verify every acknowledged operation.
  • Kill the mount daemon, sever R2/metadata access, revoke credentials, and test metadata restart/failover. Prevent successful writes onto an underlying local directory after unmount/mount failure. Enforce writer fencing at the storage layer or prove the old container cannot write before any ownership transfer.
  • Establish how every real OpenClaw connection/writer reaches remote flush before its own success response, including open append-only transcripts. If changes are required upstream, specify them and version/pin them; do not assume a filesystem mount can repair premature application acknowledgements.
  • Review isolation, credentials, key custody, live-data encryption, dependency licenses, metadata backups/HA, costs, region/network restrictions, startup latency and acceptable operation latency. Select the metadata namespace topology and failure blast radius explicitly; test tenant-level access denial. Approve tenant-keyed encryption of live writes before considering the backend qualified, not after migration.

Exit: approved storage ADR with measured durability evidence, supported configuration and a concrete integration/credential/tenant-keyed-encryption design, metadata topology/HA and storage-enforced ownership strategy. If no candidate passes, stop and choose an explicit architectural alternative before step 4. Do not declare the criterion achievable with snapshots alone.

Verification: repeat crash-after-ack tests against real production-style storage with external operation IDs; test interrupted operations for atomicity.

Rollback: remove only disposable spike resources through scoped cleanup; no live tenant cutover.

4. Integrate durable storage and migrate legacy state safely

Context: 3 selects a backend that supplies the needed durability semantics; 2 provides a safe pause/export protocol. Existing state is still in legacy roots and possibly encrypted legacy archives.

Depends on: 2 and successful 3. Tier: strongest. Ship adapter code behind an inactive tenant mode; activation also requires the step-5 lifecycle barrier.

  • Introduce a legacy storage adapter with prepare, verify, flush, and recover responsibilities; use the qualified backend’s actual APIs. Add separate live-data/metadata namespaces, not backup-retention-managed keys.
  • Mount/prepare real authoritative directories before onboard, config patch, plugin writes or gateway launch. Preserve OpenClaw’s real-parent-directory requirement; do not introduce symlinked atomic-replace parents.
  • Keep immutable image dependencies separate. Preserve user-created plugins, skills, custom workspaces and repository metadata from the inventory.
  • Migrate one tenant under maintenance: export source before mounting over it, import into an isolated destination, compare manifests/hashes/permissions, validate all databases and representative application reads, flush, then atomically commit the tenant’s storage generation/mode through the shared PostgreSQL conditional update. The activation transaction requires step 5’s barrier/ownership capability to be live; never flip a tenant after 4 alone.
  • A previously initialized but empty/corrupt/inaccessible destination is fatal, not permission to run onboarding. First-ever initialization requires an explicit provisioner-owned marker and is idempotent.
  • Once live primary storage is enabled, skip automatic legacy snapshot overlay restoration over it. Keep legacy archives read-only as migration evidence and rescue history; never let automatic retention remove the only retained pre-migration recovery source.
  • Add binding/secret provisioning and schema changes only after approval; update all env/secret docs and generated types via their normal generators.

Exit: inactive adapter and resumable import are verified. A canary may read/write the durable primary store only after 4 and 5 are both live for it; then image upgrade keeps all state and existing tenants cannot bootstrap empty. Migration is idempotent across crashes at every recorded phase.

Verification: same SDK/image pair, pre-mount export test, interrupted import, missing key/mount failure tests, legacy encrypted-import test and canary remote image upgrade.

Rollback: before new writes, return to the untouched source. After cutover writes, pause and reverse-export the latest durable state or keep the old code on the same compatible primary store; never restore the stale migration baseline. Block old code if storage/schema compatibility cannot be proved.

5. Enforce recover-before-run and unified lifecycle policy

Context: restoreIfNeeded currently uses runtime liveness and local caches; stale recovery, maintenance jobs and sleep decisions can bypass state readiness.

Depends on: 4. Tier: strongest.

Deploy this barrier before activating step 4’s tenant storage mode. Adapter preparation can precede it; customer cutover cannot. Extend step 2’s exact-export recovery guard rather than introducing a second incompatible controller.

  • Use durable tenant initialization and state-generation records plus a container-incarnation-scoped readiness receipt. Runtime active != mounted, verified and recovered. Do not rely on an SDK process ID surviving replacement.
  • On every fresh incarnation: acquire ownership -> prepare storage -> verify tenant/generation/integrity -> recover journal -> apply validated config -> launch one gateway -> expose readiness. No channels/automations before this.
  • Route all launch paths (public traffic, health/warmup, queues, scheduler, admin/CLI and watchdog) through the same barrier, with non-waking probes staying non-waking. Explicitly handle “unknown liveness” as unknown.
  • Keep intentional replacement drain/flush barriers even after primary storage exists. Unexpected replacement recovers from primary storage, not the last scheduled snapshot. Storage failure closes mutation admission and stops all independent writers, not just public HTTP routes.
  • Do not automatically fall back to an older state generation, delete recovery evidence on a transport 404, or erase required generation markers. Older recovery is a separately authorized disaster-recovery operation with a visible loss assessment, not a successful normal upgrade.
  • Record maintenance/recovery separately from ordinary health-derived error status, preserving the existing ROOT/operator access and non-waking health conventions without allowing operators to bypass the data-readiness barrier.

Exit: every fresh container is tied to validated tenant state before any writer starts; missing/failed recovery cannot lead to onboarding or stale reads.

Verification: concurrent isolates/start requests, controller restart, maintenance cancellation, unknown liveness, corrupt state and mount-loss tests.

Rollback: retain the barrier and durable state; no empty/fallback boot.

6. Add supervision and persistence observability (not another sync guarantee)

Context: primary storage is now durable. File watches/background processes are useful for audits and health, but never the write-commit mechanism.

Depends on: 4; may run alongside 5 in separate observability files. Tier: default implementation, strongest security review.

  • Supervise any storage daemon/monitor using the installed process-handle API and existing startShell helper, not the guide’s unavailable startProcess. Reattach on Worker reconnect; restart on a fresh container. Do not place a never-ending child inside setup/post-start waits or call output on a live process. Update idle classification so monitoring alone cannot pin dev/staging tenants awake, while a pending durable flush prevents a stop.
  • Optionally use supported /workspace watches for audit/index invalidation; confirm pinned path restrictions and mount-event limitations. Do not assume /root aliases escape the watch root safely. Re-establish watches after restart, handle overflow/disconnect and reconcile with full inventories.
  • Emit metadata-only events for storage readiness, generation, owner epoch, failed flush, recovery refusal, migration verification and replacement phase. Do not log credential values, tenant file contents or transcripts.
  • Replace UI claims that “R2 configured” or “last snapshot exists” means state is safe. Display primary-store readiness, verified generation and backup history as separate facts. Alert on failed preservation and empty-boot attempts.

Exit: storage failure is visible and prevents misleading success; watcher or monitor loss does not affect correctness or auto-sleep accounting.

Verification: supervision reconnect/sleep tests, log-redaction tests, UI states and mount-down canary.

Rollback: disable optional watches/metrics, never primary storage/barriers.

7. Destructive durability qualification and staged production rollout

Context: 4-6 provide primary storage, guarded lifecycle and observability. Snapshots remain recovery history, not the durability authority.

Depends on: 5 and 6. Tier: strongest acceptance review.

  • Run against disposable remote tenants with actual FUSE/R2, not only the local SDK’s periodic binding sync. Never fault-inject an existing customer.
  • Immediately after each acknowledged state mutation, replace/kill the container without taking a snapshot; recreate from another image and an empty cache. Verify the external expected operation IDs, file bytes, DB rows, deletions, rename results and application-visible state.
  • Include freeze/unfreeze, idle sleep/wake, resize, restart, both Lerma upgrade paths, debug reprovision, watchdog recovery and image rollback. Interrupt the controller after each maintenance/migration phase and resume.
  • Test SDK/container reconnects, storage outages, failed final export, corrupt latest generation, orphaned uploads, metadata failure/failover, duplicate owner attempts, custom plugin state and encrypted-state recovery.
  • Verify every DB with integrity checks and application reads; verify cross-file references too. No silent fallback, resurrection of deleted files, duplicate writer, plaintext-policy bypass or success-before-durability.
  • Promote canary -> internal dev/staging -> explicitly selected production tenant -> batches. Preserve source archives until migration evidence and the agreed rollback window permit normal retention. A failure stops promotion.

Exit: zero lost acknowledged mutations under the declared container-loss failure model, zero unintended empty boots, and all intentional replacement paths enforce the barrier. Only then label this release compliant.

Verification: shared checks plus remote destructive suite; retain evidence for source/target versions and expected/recovered generations.

Rollback: pause affected tenants and preserve the latest primary state. Roll back compatible application code, not data. Incompatible schema rollback requires a reviewed forward/reverse migration; do not start an older image just because it exists.

Dependency order and safe parallelism

1 inventory/reproduction
  -> 2 planned-replacement barriers --+
  -> 3 storage qualification ---------+-> 4 integration/migration
                                          -> 5 lifecycle enforcement --+
                                          -> 6 supervision/visibility -+-> 7 acceptance/rollout

2 and 3 can proceed independently after the inventory; 5 and 6 can use separate files after inactive adapter integration. Tenant activation requires both 4 and 5, never an intermediate release with only 4. Storage adoption and tenant migration are serial approval gates. If a step becomes too large, split it without changing the durability contract or skipping its exit criteria. A failed feasibility gate requires revising the plan, not silently shipping a best-effort substitute.

Shared verification and documentation requirements

For implementation changes, use tests-first and the existing repo commands:

bun run format:check && bun run lint && bun run typecheck && bun run test
cctr apps/fouria/test/business-rules/
bun run --cwd apps/fouria test:e2e:cli
bun run --cwd apps/fouria test:e2e:local:cli
bun run --cwd apps/fouria test:e2e:backup

Remote E2E runs require an explicitly approved disposable environment and real production-style mount behavior. Existing backup E2E commands do not yet constitute the new destructive durability suite: add that suite before claiming the criterion is verified.

For Elixir changes also run from the repository root:

mix deps.get && mix workspace.run -t deps.get
mix check && mix license_audit
mix credo --strict
mix precommit

Update docs/TESTING.md, docs/CLOUD-ARCHITECTURE.md, apps/fouria/docs/WORKER-DEVELOPMENT.md, apps/fouria/docs/OPERATOR-GUIDE.md, onboarding and affected env/secret references alongside implementation. Extend the operational-modes, tenant-isolation and dashboard-API-contract business-rules corpora for the new behavior. New routes need their normal high-coverage tests. None of these proposed runtime behaviors should be documented as already deployed.

Release claim and outstanding decisions

Immediate priority is 1-2: stop avoidable state loss during managed updates and stops. In parallel qualify 3; 1-2 alone are not the user’s complete guarantee. The complete acceptance gate is 7 on the serving legacy runtime.

Required decisions before implementation of 4: approved primary storage backend, metadata isolation/HA, credential custody, live-data encryption, application acknowledgement/flush changes, latency budget, and storage/schema rollback compatibility. No cloud resource creation or customer cutover is authorized by writing this plan.