# Legacy Sandbox: retain tenant state across container replacement

> Status: **in progress — Phase 1 delivered; live quiesce/snapshot validated on staging dev, same-archive restore after replacement pending**.
> This plan hardens the serving legacy runtime independently of the parallel
> SandboxV1 migration.
>
> - **Delivered (Phase 1):** (a) fail-closed restore — an initialized tenant
>   with a recorded recovery point never boots empty; (b) the cross-isolate
>   restore skip now probes the container's actual state instead of
>   `isRuntimeActive()`, so a fresh/replaced container always restores rather
>   than booting empty (`containerHasState` in
>   `apps/fouria/src/persistence.ts`).
> - **Delivered (Phase 2, partial step 2):** a synchronous, verified
>   pre-replacement checkpoint (`X-FourIA-Control: checkpoint`) wired across
>   Worker → dispatch → Lerma. Both upgrade paths checkpoint the running tenant
>   before the image rollout. The checkpoint step itself is **best-effort** in
>   Lerma (a control-plane failure is logged and the upgrade proceeds, so it can
>   never brick a valid upgrade); the snapshot archive verification inside the
>   Worker stays fail-closed. The shared control-plane coordination record
>   (prerequisite 2a) and full cross-plane fencing for concurrent/out-of-band
>   replacement remain gated.
> - **Deployed and live-validated (Phase 3, quiesce):** the checkpoint now **quiesces** the tenant
>   first — a durable, cross-isolate maintenance window (`src/maintenance.ts`)
>   plus a gateway stop before snapshotting, so the archive is not raced by the
>   gateway's own writes; `ensureGateway` refuses to restart the gateway while the
>   window is open, and the window self-heals on expiry. Service is always
>   restored in a `finally`. This does **not** yet drain through the rollout
>   (writes between checkpoint and replacement remain outside the archive).
>   The window/gateway-stop guards are **best-effort**: only the snapshot itself
>   is fail-closed, so a broken guard can never block an upgrade (hardened after
>   a staging incident — see PR #408/#409/#412). On 2026-10-09 at 11:06 UTC, a
>   direct checkpoint against the running `tenant-client-dev` returned
>   `backupId=7d0b2ae8-ea9d-4286-9770-883e2b355a8a`, `quiesced: true`, and
>   `maintenanceWindow: true`. Observability confirmed `killGateway` completed,
>   a 28.45 MB F4E1-encrypted archive was created, and the Gateway health event
>   returned `ok: true` afterward. This validates the live quiesce/snapshot path,
>   but the direct checkpoint did not replace the container, so restoration of
>   this same archive after replacement is still pending.
> - **Delivered (control-plane reachability):** the Lerma→dispatch internal path
>   (`/_control/*`, `/_invalidate/*`) is reachable past Cloudflare Access via
>   path-scoped bypass apps (`apps/fouria/iac/access.tf`, same posture as the
>   Lerma `/api/*` CI bypass), and the Lerma deploy sends
>   `INTERNAL_DISPATCH_INVALIDATE_TOKEN` (PR #413). Without both, the checkpoint
>   step could never run on staging — this was the actual cause of the staging
>   promotion failures, not the quiesce.
> - **Other staging evidence (2026-10-09):** the control-plane Access bypass and
>   bearer fix allowed `tenant-client*` to converge on `unstable-310e519` and
>   `unstable-85bd702`; the `0a235513` archive was restored successfully. The
>   10:43 promotion skipped its checkpoint for both instances because their
>   runtimes were not active (`skipped: not-running`), which is expected and
>   non-waking. The earlier `0f2f467f` archive was created and encrypted; the
>   observed restore was for `0a235513`, not `0f2f467f`.
> - **Pending, gated:** the shared maintenance/replacement fence (step 2, needs
>   the control-plane coordination record), durable primary storage (step 3,
>   feasibility gate), migration (step 4), recover-before-run across all launch
>   paths (step 5), supervision/observability (step 6) and destructive
>   acceptance (step 7).
>
> No dependency, infrastructure, deployment, or tenant data has been changed by
> the pending steps. Full compliance requires a synchronous durability boundary;
> the Phase 1 guard prevents _silent_ empty boots but does **not** by itself make
> every acknowledged write durable across unexpected container loss.

## Outcome and non-negotiable acceptance criteria

Containers, gateway connections, and in-flight sessions may stop. Previously
saved tenant state must not disappear when an image is upgraded, a container
sleeps, is frozen, is replaced, or becomes unavailable.

1. **Planned replacement:** persist every completed mutation before replacing
   the container. Block replacement if preservation cannot be proved.
2. **Unexpected container loss:** every mutation acknowledged as saved must
   already be durable outside that container (zero recovery-point loss).
3. **Unavailable storage:** stop accepting mutations rather than acknowledge
   writes into an ephemeral fallback directory.
4. **Recovery:** an existing tenant never silently starts empty or selects an
   older recovery generation. Report an error and keep it unavailable instead.
5. **Single owner:** at most one gateway may mutate a tenant's state. A timed-out
   RPC or an expired lease alone is not proof that the old gateway stopped.
6. **Isolation and encryption:** preserve tenant boundaries and existing
   application-level encryption requirements, not merely provider encryption.
7. **Migration independence:** keep `ACTIVE_SANDBOX_RUNTIME = 'legacy'`, the
   matched legacy SDK/base-image versions, and the v1 scaffold unchanged.
   Application-image changes are expected for these fixes; an OpenClaw fork or
   upstream persistence change requires separate approval, not an SDK cutover.

"Saved" must mean a successfully completed durable file operation or database
transaction, not merely an in-memory change. Persisted conversation history,
sessions, agents, credentials, pairing, schedules, memory, custom plugins, and
workspace artifacts are included. Live sockets, running processes, and
unfinished writes are not. A process killed halfway through a mutation may
leave that mutation uncommitted, but must not erase previously committed state.
Audit OpenClaw's actual success responses: calling a local `write()` or returning
a SQLite commit with asynchronous durability is not enough. If the application
acknowledges before persistence, change that boundary rather than redefining
those acknowledged writes as expendable.

## Evidence from the current repository

| Surface                                                                        | Finding                                                                                                                                                     | Consequence                                                                                                                   |
| ------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| Root `package.json` catalog; `apps/fouria/Dockerfile`                          | Serving alias is `@cloudflare/sandbox-legacy` -> `0.13.0-next.776.1`, paired with the same image. The separate package is `@cloudflare/sandbox@1.0.0`.      | Although called legacy/v0.x, this is a preview API: do not copy stable string-exec/startProcess examples.                     |
| `apps/fouria/src/sandbox-runtime.ts`                                           | All tenants still select legacy.                                                                                                                            | Fixes cannot depend on an unfinished v1 cutover.                                                                              |
| `apps/lerma/lib/lerma/provisioning/tenant_provisioner.ex`, `@upgrade_pipeline` | Fetch -> deploy -> mark updated; no persistence preparation step.                                                                                           | Image replacement is not fenced by a final preservation barrier.                                                              |
| `apps/lerma/lib/lerma/instances.ex`, `do_upgrade`                              | Another upgrade entry point also directly deploys.                                                                                                          | Fixing only the provisioning pipeline leaves a bypass.                                                                        |
| `apps/lerma/lib/lerma/cloudflare/containers.ex`                                | Image/type changes and restarts use container rollouts.                                                                                                     | All replacement paths need the same preservation policy.                                                                      |
| `apps/fouria/src/index.ts`, freeze control                                     | Calls `sandbox.destroy()` without a final checkpoint.                                                                                                       | Recent state can be lost on an intentional stop.                                                                              |
| `apps/fouria/src/sandbox-do.ts`, idle expiry                                   | Calls `this.stop()` without a final checkpoint.                                                                                                             | Non-billable tenants can lose recent state on sleep.                                                                          |
| `apps/fouria/src/gateway/sandbox-recovery.ts`                                  | Recovery destroys a runtime judged stale.                                                                                                                   | Unexpected loss cannot be repaired by copying data after it is gone.                                                          |
| `apps/fouria/src/persistence.ts`, `restoreIfNeeded`                            | Runtime liveness is used as a cross-isolate restore-completion signal. Empty/unrestorable catalogs may return a clean start; older candidates may be tried. | A newly started SDK container is not evidence that tenant state was recovered, and fallback can silently discard newer state. |
| `apps/fouria/Dockerfile`; `persistence.ts`                                     | Real legacy roots `/root/openclaw/.openclaw` and `/root/clawd` are outside `/home/openclaw`, requiring a stage/materialization bridge.                      | Enumerate real paths; a mount or backup of only `/home/openclaw` is insufficient.                                             |
| `persistence.ts`, legacy stage                                                 | Database images precede tree copying; existing stage is overwritten rather than replaced.                                                                   | Individual SQLite backups are consistent, but cross-file consistency and deletion fidelity need explicit tests.               |

These are source-level risks, not a claim that every reported tenant loss has
been reproduced or that live Cloudflare configuration has been inspected.

### What the requested Cloudflare guides establish

- [Sandbox lifecycle](https://developers.cloudflare.com/sandbox/sdk/concepts/sandboxes/):
  a stable sandbox ID/DO does **not** make its container filesystem durable.
  Sleep and destruction discard that filesystem.
- [Storage API](https://developers.cloudflare.com/sandbox/sdk/api/storage/) and
  [persistent-storage tutorial](https://developers.cloudflare.com/sandbox/sdk/tutorials/persistent-storage/):
  mount an external data directory early and remount it on each fresh container;
  data already persisted there survives destruction. Mounts overlay existing
  image contents rather than merging them.
- [File watching](https://developers.cloudflare.com/sandbox/sdk/guides/file-watching/):
  watchers are streams, end on sleep/shutdown, and must be re-established. The
  guide restricts watched paths to `/workspace`. A watcher is not a durable
  journal and cannot save a change after the container has already disappeared.
- [Background processes](https://developers.cloudflare.com/sandbox/sdk/guides/background-processes/):
  a container-owned monitor can outlive an individual HTTP request, but not the
  container. On our pin launch it through `startShell`/argv `exec`, keep it
  separate from the child-free setup/post-start waits, and supervise it.
- [Mount-buckets guide](https://developers.cloudflare.com/sandbox/sdk/guides/mount-buckets/):
  credential-less R2 binding mounts require `ContainerProxy`, already exported
  by this Worker. Local `localBucket` uses periodic sync, unlike production
  mounts, so local tests alone cannot establish production durability.
- [Cloudflare FUSE example](https://developers.cloudflare.com/containers/examples/r2-fuse-mount/)
  and [s3fs limitations](https://github.com/s3fs-fuse/s3fs-fuse#limitations):
  an object-store mount does not provide full POSIX semantics; s3fs has neither
  atomic renames nor coordination between mounting clients. Do not put the
  entire live OpenClaw state/SQLite tree on s3fs and call it safe.

The installed legacy types confirm `mountBucket`, `watch`, `createBackup`, and
argv `exec` returning a process handle. They do not expose the stable guide's
`startProcess` API. Use `apps/fouria/src/gateway/sandbox-exec.ts` for execution.

## Recommended design

**One active gateway, durable storage as the primary source of truth, snapshots
as disaster-recovery history.** More replicas are not needed for this criterion.

### One coordination record shared by both control planes

Use the existing shared PostgreSQL source of truth: propose a
`tenant_runtime_operations` table keyed by immutable instance ID, with operation
ID, monotonically increasing owner epoch, phase, source runtime/image, required
recovery generation and target storage mode. Reserve/advance operations using
transactions and conditional updates (`UPDATE ... WHERE owner_epoch = ... AND
phase = ... RETURNING ...`), with one active operation per instance. R2 holds
immutable exports/manifests; PostgreSQL publishes the selected generation only
after its objects are verified. Pin an operation's objects against retention
from upload through publication/recovery acknowledgement, and reverify after an
interrupted publication. Do not imply a transaction spans R2 and SQL.

Both Lerma and the tenant Worker must consult this record before intentional
stop/replace or launch; use a reviewed least-privilege Worker PostgreSQL access
path, never Lerma's HTTP API or database credentials inside the container.
Inventory each entry point against this gate. Persist cloud API intent and
operation identity before issuing a rollout; reconcile ambiguous responses and
completion rather than letting a lease timeout authorize a competing operation.
An owner epoch coordinates the controller but only fences actual writes if the
storage backend rejects stale owners, or the old runtime is proved terminated.

Separate three classes of storage:

| Class                                                                                                         | Placement                                                                   | Rule                                                                                                                                                                               |
| ------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Tenant runtime state, SQLite databases/journals, transcripts, custom files, workspace and repository metadata | A qualified durable filesystem, or synchronous application-level stores     | All saved changes reach remote durability before success. No local-only authoritative copy.                                                                                        |
| External documents, uploads, completed artifacts with object semantics                                        | Tenant-scoped R2 through existing bridges or a qualified `mountBucket` path | Treat remote object commit, not a queued copy or local cache write, as completion. Publish multi-object changes through immutable generations plus a committed manifest if needed. |
| Image-provided dependencies, rebuildable indexes/caches, temporary execution files, live sockets              | Container/image                                                             | Explicitly classified as rebuildable/transient; exclude only with evidence.                                                                                                        |

### Storage feasibility gate: do not pretend s3fs solves SQLite

The smallest application-change candidate to **evaluate**, not automatically
adopt, is a POSIX-oriented FUSE filesystem such as JuiceFS, backed by R2 data
and a separately durable PostgreSQL metadata store. Cloudflare supports FUSE;
the filesystem adds atomic metadata operations and locking absent from s3fs.

Relevant upstream evidence:

- [JuiceFS cache/write semantics](https://juicefs.com/docs/community/guide/cache/):
  without client `--writeback`, data is uploaded before metadata publication;
  `fsync`/`fdatasync`/`close` trigger flush. A successful `write` alone still
  does not prove persistence.
- [POSIX compatibility](https://juicefs.com/docs/community/posix_compatibility/):
  atomic metadata operations, mmap, flock, and traditional record locks are
  documented. This is **not** proof that our OpenClaw/SQLite configuration is
  safe on Cloudflare's actual kernel and runtime.
- [Metadata engines](https://juicefs.com/docs/community/databases_for_metadata/):
  PostgreSQL is supported. Metadata is part of the authoritative data, not a
  disposable cache; its durability/failover guarantees matter as much as R2.

Do not enable delayed uploads, client writeback, or use an ephemeral metadata
database. Audit every SQLite connection's journal and synchronous settings,
including plugins: changing a PRAGMA in an unrelated CLI connection does not
change the gateway's connection. Prove WAL/shared-memory behavior with one
owning container, or select and validate another supported journal mode.

This candidate adds infrastructure, latency, credential handling, licensing,
and encryption questions. Approval and a successful production-like spike are
required before adoption. Never give the agent platform PostgreSQL credentials,
use a shared metadata namespace with tenant-supplied paths, or silently expose
keys previously held only in the Worker. Qualify a supported credential proxy
or an explicitly approved tenant-isolated access model; do not assume the
SDK's s3fs proxy transparently works for another client or PostgreSQL TCP.
Application-level encryption of live data needs a design of its own: the F4E1
post-snapshot transform does not encrypt live filesystem writes. Candidate
qualification must produce a tenant-keyed live-data encryption and key-custody
design compatible with the current policy, or the candidate fails the gate.
Specify metadata topology too: isolated volumes/catalogs and database roles per
tenant, or a reviewed alternative with enforceable isolation and explicit
corruption blast radius. A directory prefix alone is not metadata isolation.

**If this gate fails:** use synchronous application-level persistence for
OpenClaw state (durable SQL/DO records and R2 objects, with changes to its write
paths), or host the gateway on qualified persistent storage. Those are explicit
architectural decisions, not silently included migrations. The supplied
Sandbox APIs alone cannot guarantee zero loss for unmodified arbitrary local
filesystem/SQLite writes. Do not replace the hard criterion with asynchronous
file watching, rsync, or WAL shipping.

The application-level alternative may require an upstream OpenClaw change or a
maintained fork. That is a significant, separately approved scope expansion
with its own compatibility/release policy; it is not a ready-made off-ramp that
can be implemented without changing the current application image.

## Construction sequence (seven reviewable changes)

**Actionable near-term proposal:** 1-2. **Feasibility study/decision:** 3.
**Conditional implementation, not approved scope:** 4-7, only if 3 passes and
the resulting backend/application changes are approved. Split large steps into
smaller PRs as needed, preserving their acceptance and activation gates.

Use small branches/PRs from `develop`; do not deploy automatically as part of
planning. Each implementation PR updates its corresponding documentation and
business-rules corpus. "Default" and "strongest" below are review/implementation
complexity tiers, not a request to change a specific model.

### 1. Inventory state and reproduce loss

**Context:** legacy OpenClaw uses real `/root` roots plus `/home` aliases;
snapshots stage them, and several replacement entry points exist. No primary
durable filesystem is wired today.

**Depends on:** none. **Tier:** default implementation, strongest review.

- Write tests first that save unique sentinels through real application paths:
  config, paired device, credentials, agent, conversation/transcript, automation,
  SQLite rows, workspace binaries, repositories, rename and deletion.
- Inventory realpaths, all databases/journals, plugin-owned state, custom
  workspace paths, permissions, symlinks and cross-root references. Separate
  image dependencies from user-installed/custom plugin content.
- Record every stop/replace path: both Lerma upgrade paths, image/type/restart
  rollouts, promotion/rescue scripts, freeze, idle expiry, debug restart/
  reprovision, watchdog recovery, and out-of-band platform termination.
  Map every managed entry point to the shared PostgreSQL coordination gate.
- Audit acknowledgement and flush semantics for API, WS, external channels,
  scheduled jobs, shell tools and plugins. A Worker-only response gate misses
  gateway-owned writes.
- Record baseline hashes, DB queries, file ownership/modes and image identity;
  persist test expectations outside the disposable container.

**Exit:** failing regression(s) demonstrate the gap; a versioned storage
inventory assigns every tenant state path to an authoritative store. No
unclassified tenant state is treated as a cache.

**Verification:** targeted Vitest/ExUnit tests plus a disposable remote tenant
reproduction using real R2, then the shared checks below.

**Rollback:** test/docs-only; no customer mutation.

### 2. Immediately fence planned destructive operations

**Context:** `@upgrade_pipeline`, `Instances.do_upgrade`, freeze and idle sleep
currently lack a final preservation barrier. Existing async backup acceptance
is not proof of completion.

**Depends on:** 1. **Tier:** strongest.

- **Prerequisite 2a, separately reviewed before enabling the guard:** provision
  the new least-privilege Worker-to-PostgreSQL access path, coordination table,
  tenant-scoped grants and binding/network configuration. Direct Worker DB
  access is not wired today; do not present it as an existing facility. Exit
  this prerequisite only when both planes can conditionally reserve/read the
  same record, cross-tenant access is denied, and a DB outage fails closed.
  It requires infrastructure approval and its own tests/docs. Until it lands,
  operationally suspend unattended managed replacements rather than claim the
  guard exists; this containment does not prevent platform-originated loss.
- Add a persisted per-tenant maintenance/replacement operation with ID, owner
  epoch, source image/runtime identity, phase and required recovery generation.
  Implement the shared PostgreSQL record and conditional-update protocol above,
  not module-local flags. Include cancellation and reconciliation after
  controller restart, and enforce the same record in both control planes.
- Stop admission from proxy/admin mutations, new WS commands, queues, activity
  wake/heal and scheduled jobs. Quiesce external channel adapters too; closing
  web UI sockets alone is insufficient. Stop gateway/child writers while
  retaining the container. Verify they exited before exporting state.
- Resolve in-flight mutations before certifying the export: drain or safely
  abort unfinished work, flush all previously acknowledged mutations, including
  open append-only transcripts and database journals, and verify writer exit.
  A forced termination that cannot prove preservation must fail the barrier;
  previously acknowledged writes cannot be reclassified as unfinished work.
- Force a final complete, consistent export of all inventoried state. Verify
  archive bytes/checksums, manifest, decryption and DB integrity by restoring
  into an isolated validator that cannot connect channels or run automations.
  Include deletion fidelity, not just existence checks.
- Permit rollout/destroy only after durable completion is recorded for this
  operation. Do not accept `202`, an interval-skipped snapshot, a metadata HEAD,
  or any older restorable archive as the barrier.
- Require the replacement to restore this exact verified generation before any
  gateway/CLI writer starts. Implement that minimal recover-before-run guard
  now, not only in step 5; empty boot, active-runtime shortcuts and older-backup
  fallback must not turn a safe export into a lossy upgrade.
- Centralize policy across both upgrade paths, resize/restart, freeze, idle
  sleep and operator recovery. On preservation failure leave the old container
  intact and visibly unavailable; resume it only through a reconciled cancel.
- Bootstrap this control release without replacing the old container: use an
  explicitly verified code-only deployment with unchanged legacy SDK/image and
  resolved resources. Use a Worker-code-only path that does not touch the
  container application, or explicitly suppress `create_image_rollout` only
  after proving the resolved image reference/digest and resources are unchanged.
  A reused mutable tag alone is not proof of identical image contents.
  Cloudflare may return resource numbers rather than an `instance_type` name;
  avoid an accidental rollout from comparing that absent string. Validate on a
  disposable tenant first. If code-only bootstrap cannot be proved, require an
  operator-controlled quiesced export before deployment.

**Exit:** no intentional replacement can execute without a fresh verified
barrier; write/job races and operation retries are covered. This step protects
planned upgrades, **not** unexpected loss before the final export.

**Verification:** targeted replacement/control/idle tests and old-image -> new
image remote E2E, including export failure -> zero rollout API calls.

**Rollback:** keep the preservation guard; cancel maintenance explicitly. Do
not roll back to an unguarded destroy path or older tenant state.

### 3. Qualify the synchronous primary storage backend

**Context:** periodic snapshots and watcher-driven copies have a loss window;
plain R2/s3fs does not supply the full state-tree semantics OpenClaw needs.
JuiceFS + R2 + durable PostgreSQL is only an evaluated candidate.

**Depends on:** 1; can run alongside 2 with disjoint spike/test files.
**Tier:** strongest architecture/security/database review.

- Build a disposable, production-like legacy-image spike. Do not change the
  production default SDK/image or select SandboxV1.
- Prove mount readiness, atomic replace, file/directory persistence, symlink
  behavior, locks, concurrent gateway/CLI access, SQLite journal recovery,
  successful close/fsync durability and errors during storage outages.
- Destroy the container immediately after successful saved operations without
  running a snapshot or a shutdown handler; remount with an empty local cache
  and verify every acknowledged operation.
- Kill the mount daemon, sever R2/metadata access, revoke credentials, and test
  metadata restart/failover. Prevent successful writes onto an underlying local
  directory after unmount/mount failure. Enforce writer fencing at the storage
  layer or prove the old container cannot write before any ownership transfer.
- Establish how every real OpenClaw connection/writer reaches remote flush
  before its own success response, including open append-only transcripts.
  If changes are required upstream, specify them and version/pin them; do not
  assume a filesystem mount can repair premature application acknowledgements.
- Review isolation, credentials, key custody, live-data encryption, dependency
  licenses, metadata backups/HA, costs, region/network restrictions, startup
  latency and acceptable operation latency.
  Select the metadata namespace topology and failure blast radius explicitly;
  test tenant-level access denial. Approve tenant-keyed encryption of live
  writes before considering the backend qualified, not after migration.

**Exit:** approved storage ADR with measured durability evidence, supported
configuration and a concrete integration/credential/tenant-keyed-encryption
design, metadata topology/HA and storage-enforced ownership strategy. If no
candidate passes, stop and choose an explicit architectural alternative before
step 4. Do not declare the criterion achievable with snapshots alone.

**Verification:** repeat crash-after-ack tests against real production-style
storage with external operation IDs; test interrupted operations for atomicity.

**Rollback:** remove only disposable spike resources through scoped cleanup;
no live tenant cutover.

### 4. Integrate durable storage and migrate legacy state safely

**Context:** 3 selects a backend that supplies the needed durability semantics;
2 provides a safe pause/export protocol. Existing state is still in legacy
roots and possibly encrypted legacy archives.

**Depends on:** 2 and successful 3. **Tier:** strongest. Ship adapter code behind
an inactive tenant mode; activation also requires the step-5 lifecycle barrier.

- Introduce a legacy storage adapter with `prepare`, `verify`, `flush`, and
  `recover` responsibilities; use the qualified backend's actual APIs. Add
  separate live-data/metadata namespaces, not backup-retention-managed keys.
- Mount/prepare real authoritative directories before onboard, config patch,
  plugin writes or gateway launch. Preserve OpenClaw's real-parent-directory
  requirement; do not introduce symlinked atomic-replace parents.
- Keep immutable image dependencies separate. Preserve user-created plugins,
  skills, custom workspaces and repository metadata from the inventory.
- Migrate one tenant under maintenance: export source before mounting over it,
  import into an isolated destination, compare manifests/hashes/permissions,
  validate all databases and representative application reads, flush, then
  atomically commit the tenant's storage generation/mode through the shared
  PostgreSQL conditional update. The activation transaction requires step 5's
  barrier/ownership capability to be live; never flip a tenant after 4 alone.
- A previously initialized but empty/corrupt/inaccessible destination is fatal,
  not permission to run onboarding. First-ever initialization requires an
  explicit provisioner-owned marker and is idempotent.
- Once live primary storage is enabled, skip automatic legacy snapshot overlay
  restoration over it. Keep legacy archives read-only as migration evidence
  and rescue history; never let automatic retention remove the only retained
  pre-migration recovery source.
- Add binding/secret provisioning and schema changes only after approval;
  update all env/secret docs and generated types via their normal generators.

**Exit:** inactive adapter and resumable import are verified. A canary may
read/write the durable primary store only after 4 and 5 are both live for it;
then image upgrade keeps all state and existing tenants cannot bootstrap empty.
Migration is idempotent across crashes at every recorded phase.

**Verification:** same SDK/image pair, pre-mount export test, interrupted import,
missing key/mount failure tests, legacy encrypted-import test and canary remote
image upgrade.

**Rollback:** before new writes, return to the untouched source. After cutover
writes, pause and reverse-export the latest durable state or keep the old code
on the same compatible primary store; never restore the stale migration
baseline. Block old code if storage/schema compatibility cannot be proved.

### 5. Enforce recover-before-run and unified lifecycle policy

**Context:** `restoreIfNeeded` currently uses runtime liveness and local caches;
stale recovery, maintenance jobs and sleep decisions can bypass state readiness.

**Depends on:** 4. **Tier:** strongest.

Deploy this barrier before activating step 4's tenant storage mode. Adapter
preparation can precede it; customer cutover cannot. Extend step 2's exact-export
recovery guard rather than introducing a second incompatible controller.

- Use durable tenant initialization and state-generation records plus a
  container-incarnation-scoped readiness receipt. Runtime active != mounted,
  verified and recovered. Do not rely on an SDK process ID surviving replacement.
- On every fresh incarnation: acquire ownership -> prepare storage -> verify
  tenant/generation/integrity -> recover journal -> apply validated config ->
  launch one gateway -> expose readiness. No channels/automations before this.
- Route all launch paths (public traffic, health/warmup, queues, scheduler,
  admin/CLI and watchdog) through the same barrier, with non-waking probes
  staying non-waking. Explicitly handle "unknown liveness" as unknown.
- Keep intentional replacement drain/flush barriers even after primary storage
  exists. Unexpected replacement recovers from primary storage, not the last
  scheduled snapshot. Storage failure closes mutation admission and stops all
  independent writers, not just public HTTP routes.
- Do not automatically fall back to an older state generation, delete recovery
  evidence on a transport 404, or erase required generation markers. Older
  recovery is a separately authorized disaster-recovery operation with a
  visible loss assessment, not a successful normal upgrade.
- Record maintenance/recovery separately from ordinary health-derived error
  status, preserving the existing ROOT/operator access and non-waking health
  conventions without allowing operators to bypass the data-readiness barrier.

**Exit:** every fresh container is tied to validated tenant state before any
writer starts; missing/failed recovery cannot lead to onboarding or stale reads.

**Verification:** concurrent isolates/start requests, controller restart,
maintenance cancellation, unknown liveness, corrupt state and mount-loss tests.

**Rollback:** retain the barrier and durable state; no empty/fallback boot.

### 6. Add supervision and persistence observability (not another sync guarantee)

**Context:** primary storage is now durable. File watches/background processes
are useful for audits and health, but never the write-commit mechanism.

**Depends on:** 4; may run alongside 5 in separate observability files.
**Tier:** default implementation, strongest security review.

- Supervise any storage daemon/monitor using the installed process-handle API
  and existing `startShell` helper, not the guide's unavailable `startProcess`.
  Reattach on Worker reconnect; restart on a fresh container. Do not place a
  never-ending child inside setup/post-start waits or call output on a live
  process. Update idle classification so monitoring alone cannot pin dev/staging
  tenants awake, while a pending durable flush prevents a stop.
- Optionally use supported `/workspace` watches for audit/index invalidation;
  confirm pinned path restrictions and mount-event limitations. Do not assume
  `/root` aliases escape the watch root safely. Re-establish watches after
  restart, handle overflow/disconnect and reconcile with full inventories.
- Emit metadata-only events for storage readiness, generation, owner epoch,
  failed flush, recovery refusal, migration verification and replacement phase.
  Do not log credential values, tenant file contents or transcripts.
- Replace UI claims that "R2 configured" or "last snapshot exists" means state
  is safe. Display primary-store readiness, verified generation and backup
  history as separate facts. Alert on failed preservation and empty-boot attempts.

**Exit:** storage failure is visible and prevents misleading success; watcher
or monitor loss does not affect correctness or auto-sleep accounting.

**Verification:** supervision reconnect/sleep tests, log-redaction tests, UI
states and mount-down canary.

**Rollback:** disable optional watches/metrics, never primary storage/barriers.

### 7. Destructive durability qualification and staged production rollout

**Context:** 4-6 provide primary storage, guarded lifecycle and observability.
Snapshots remain recovery history, not the durability authority.

**Depends on:** 5 and 6. **Tier:** strongest acceptance review.

- Run against disposable remote tenants with actual FUSE/R2, not only the
  local SDK's periodic binding sync. Never fault-inject an existing customer.
- Immediately after each acknowledged state mutation, replace/kill the
  container without taking a snapshot; recreate from another image and an empty
  cache. Verify the external expected operation IDs, file bytes, DB rows,
  deletions, rename results and application-visible state.
- Include freeze/unfreeze, idle sleep/wake, resize, restart, both Lerma upgrade
  paths, debug reprovision, watchdog recovery and image rollback. Interrupt the
  controller after each maintenance/migration phase and resume.
- Test SDK/container reconnects, storage outages, failed final export, corrupt
  latest generation, orphaned uploads, metadata failure/failover, duplicate
  owner attempts, custom plugin state and encrypted-state recovery.
- Verify every DB with integrity checks and application reads; verify
  cross-file references too. No silent fallback, resurrection of deleted files,
  duplicate writer, plaintext-policy bypass or success-before-durability.
- Promote canary -> internal dev/staging -> explicitly selected production
  tenant -> batches. Preserve source archives until migration evidence and the
  agreed rollback window permit normal retention. A failure stops promotion.

**Exit:** zero lost acknowledged mutations under the declared container-loss
failure model, zero unintended empty boots, and all intentional replacement
paths enforce the barrier. Only then label this release compliant.

**Verification:** shared checks plus remote destructive suite; retain evidence
for source/target versions and expected/recovered generations.

**Rollback:** pause affected tenants and preserve the latest primary state.
Roll back compatible application code, not data. Incompatible schema rollback
requires a reviewed forward/reverse migration; do not start an older image just
because it exists.

## Dependency order and safe parallelism

```text
1 inventory/reproduction
  -> 2 planned-replacement barriers --+
  -> 3 storage qualification ---------+-> 4 integration/migration
                                          -> 5 lifecycle enforcement --+
                                          -> 6 supervision/visibility -+-> 7 acceptance/rollout
```

2 and 3 can proceed independently after the inventory; 5 and 6 can use separate
files after inactive adapter integration. **Tenant activation requires both 4
and 5, never an intermediate release with only 4.** Storage adoption and tenant
migration are serial approval gates. If a step becomes too large, split it without changing the
durability contract or skipping its exit criteria. A failed feasibility gate
requires revising the plan, not silently shipping a best-effort substitute.

## Shared verification and documentation requirements

For implementation changes, use tests-first and the existing repo commands:

```bash
bun run format:check && bun run lint && bun run typecheck && bun run test
cctr apps/fouria/test/business-rules/
bun run --cwd apps/fouria test:e2e:cli
bun run --cwd apps/fouria test:e2e:local:cli
bun run --cwd apps/fouria test:e2e:backup
```

Remote E2E runs require an explicitly approved disposable environment and real
production-style mount behavior. Existing backup E2E commands do not yet
constitute the new destructive durability suite: add that suite before claiming
the criterion is verified.

For Elixir changes also run from the repository root:

```bash
mix deps.get && mix workspace.run -t deps.get
mix check && mix license_audit
mix credo --strict
mix precommit
```

Update `docs/TESTING.md`, `docs/CLOUD-ARCHITECTURE.md`,
`apps/fouria/docs/WORKER-DEVELOPMENT.md`,
`apps/fouria/docs/OPERATOR-GUIDE.md`, onboarding and affected env/secret
references alongside implementation. Extend the operational-modes,
tenant-isolation and dashboard-API-contract business-rules corpora for the new
behavior. New routes need their normal high-coverage tests. None of these
proposed runtime behaviors should be documented as already deployed.

## Release claim and outstanding decisions

Immediate priority is 1-2: stop avoidable state loss during managed updates and
stops. In parallel qualify 3; **1-2 alone are not the user's complete guarantee**.
The complete acceptance gate is 7 on the serving legacy runtime.

Required decisions before implementation of 4: approved primary storage
backend, metadata isolation/HA, credential custody, live-data encryption,
application acknowledgement/flush changes, latency budget, and storage/schema
rollback compatibility. No cloud resource creation or customer cutover is
authorized by writing this plan.
