# Sandbox v1 — staging observation record

Append-only record of the **step 5** observation for each milestone on the
`v1-sandbox` track. The target is Lerma client `client-sandbox-v1` (id 20) and
its two instances (`tenant-client-sandbox-v1-dev`, `tenant-client-sandbox-v1`)
in the Paso4 **staging** account, dispatch namespace `fouria-tenants-staging`.

Nothing here is a production observation: no production tenant has been
converted, and the migration authority (`WORKER_PLATFORM === "staging"`) is not
present in production.

## How to reproduce

```bash
# Instance state + health (staging Lerma)
#   docker exec -i lerma-pre sh -s <<'EOF'
#   ... Application.get_env(:lerma, :provisioning)[:dispatch_namespace] ...
/app/bin/lerma rpc 'Lerma.Instances.list_all_for_client(20)'

# Deployed script identity, bindings and bundle content (Cloudflare API)
#   GET /accounts/<acct>/workers/dispatch/namespaces/fouria-tenants-staging/scripts
#   GET .../scripts/<name>/settings
#   GET .../scripts/<name>/content        # grep for SANDBOX_V1_* markers
#   GET /accounts/<acct>/containers/applications

# Worker telemetry (Lerma ingest of the Observability destinations)
#   from(e in "observability_events", where: e.script_name in ^names, ...) |> Repo.all()
```

## S1 — Authority, identity and platform gate (`unstable-a90015f5...`)

- Both instances report the version; Lerma health `:ok`.
- Health events for production: `down` (503, 7.5 s) at 20:00:52 → `up` at
  20:03:37 during the rollout window.
- No migration opportunity was exercised (there is no conversion yet).

## S2 — Conversion intent, lease and DO conversion (`unstable-486b2938...`)

- Both instances report the version, status `sleeping`, health `:ok`.
- `Dockerfile.v1` includes `squashfs-tools`.
- No intent exists, so both tenants remain legacy. No conversion was triggered.

## S3 — Serving parity: proxy, health and admin RPC (`unstable-41be4409...`)

### Verified

| Check                                          | Result                                                                                                                                                       |
| ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Tenant scripts modified at the promotion times | ✅ `tenant-client-sandbox-v1` 08:20:47Z, `…-dev` 08:22:41Z                                                                                                   |
| Bundle contains the S3 code                    | ✅ `SANDBOX_V1_RUNTIME_UNKNOWN`, `SANDBOX_V1_CONTROL_UNSUPPORTED`, `SANDBOX_V1_SLEEPING`, `sandbox.runtime.selected`, `__fouriaSandboxV1`, `v1EnsureGateway` |
| Same bundle on both instances                  | ✅ identical script `etag`                                                                                                                                   |
| Container image                                | ✅ `fouria:unstable-41be4409…` on both legacy tenant apps                                                                                                    |
| Container health                               | ✅ 3 healthy / 0 active per app (idle-slept)                                                                                                                 |
| Lerma instance state                           | ✅ status `sleeping`, `deployed_version` = `unstable-41be4409…`, `HealthMonitor.check_instance_now/1` → `:ok`                                                |
| Worker telemetry after the deploy              | ✅ spans `200` (80 / 67), no `error` outcome, no exception class after 2026-11-04T20:00Z                                                                     |
| `503` spans                                    | all 13 + 13 from **2026-11-04 20:02–20:03Z** (first cold boot), none after the S3 deploy                                                                     |
| Latest activity                                | ✅ 2026-11-05T08:57Z                                                                                                                                         |
| Hosted `/healthz`                              | ✅ `302` (Cloudflare Access), no 5xx                                                                                                                         |

### Blocking finding: the v1 path is deployed but cannot be selected

The live tenant script bindings (read from the dispatch-namespace settings) are:

`ACTIVITY`, `ANALYTICS`, `BACKUP_BUCKET`, `BACKUP_BUCKET_NAME`,
`BACKUP_CHANGE_DETECTION`, `BACKUP_CHANGE_WATCH`,
`BACKUP_ENCRYPTION_KEY`, `BACKUP_ENCRYPTION_REQUIRED`, `BROWSER`, `CDP_SECRET`,
`CF_ACCESS_*`, `CF_AI_GATEWAY_*`, `CLOUDFLARE_*`, `DEBUG_ROUTES`, `LEADS_BUCKET`,
`MOLTBOT_GATEWAY_TOKEN`, `OPENCODE_API_KEY`, `PASO4_TELEGRAM_GROUP_ID`,
`R2_ACCESS_KEY_ID`, `R2_SECRET_ACCESS_KEY`, **`Sandbox`**,
`TELEGRAM_INGRESS`, `TELEGRAM_INGRESS_HOST`, `TELEGRAM_INGRESS_QUEUE`,
`WORKER_URL`.

Missing for the v1 track:

1. **`WORKER_PLATFORM`** (and `SANDBOX_V1_CUTOVER`). `resolveRuntimeIntent`
   requires the exact trusted value `staging`; without the binding it always
   returns `legacy`. The S3 runtime selection therefore never activates.
2. **`SandboxV1` Durable Object binding** (and its class migration). The bundle
   exports `SandboxV1`, but with no namespace binding `acquireSandboxV1()` would
   throw `SandboxV1 binding is not configured` if selection ever reached it.
3. **A v1 container application** (`scheduling_policy: "durable_object"`, v1
   image). The account has only the per-tenant legacy applications
   (`tenant-<slug>-sandbox-staging`, image `fouria:unstable-…`). The
   `fouria-sandbox-v1-staging` application declared in `wrangler.jsonc` is
   created by a standalone `wrangler deploy`, not by Lerma's dispatch-namespace
   upload, so it does not exist for tenants.

### Root cause

Tenant scripts are uploaded by **Lerma** into `fouria-tenants-staging`, not by
`wrangler deploy` with this repo's `wrangler.jsonc` environment. The two Lerma
paths differ:

- `Lerma.Provisioning.TenantProvisioner.required_var_bindings/2` adds
  `WORKER_PLATFORM` / `SANDBOX_V1_CUTOVER` (S1 change).
- `Lerma.Instances.instance_bindings/2`, used by the **upgrade/promotion**
  redeploy, does not add them (and neither path adds `SandboxV1`). The promotion
  therefore re-uploads the tenant with the legacy binding set.
- The deployed staging Lerma runs `develop`, which does not contain the S1
  provisioning change at all, so instances created before it also lack the
  bindings.

### Verdict

- S3 is **deployed** (bundle identity proven) and **inert** (runtime selection
  cannot choose v1; the v1 runtime is not provisioned). This is a truthful
  "deployed, not activatable" state, not a runtime failure: legacy serving is
  unchanged, with no post-deploy errors.
- Activation belongs to **OCM-4 (provisioning, rollout, promotion)** with the
  conversion orchestration in **S5 / PER-3**:
  1. add `WORKER_PLATFORM`, `SANDBOX_V1_CUTOVER` and the `SandboxV1` DO binding
     (plus its class migration) to `Instances.instance_bindings/2`;
  2. provision/declare a per-tenant v1 container application with the v1 image
     and `scheduling_policy: "durable_object"` (naming + rollout policy);
  3. redeploy the staging Lerma with the v1-branch changes so an upgrade carries
     the bindings;
  4. then run the conversion job and repeat this observation against the v1
     runtime (HTTP/WS proxy, passive health, single owner).

## Next observation

Repeat S3 once the binding/application gap above is closed: the same API reads
must show `SandboxV1` in the tenant bindings and a per-tenant v1 container
application, and the observability stream must show v1-path traffic
(`sandbox.runtime.selected: v1`) with no proxy/WS errors.

## OCM-4a — staging provisioning of the v1 surface (code landed, not yet deployed)

Closing the gap above is provisioning work, implemented on `v1-sandbox`:

- **`Lerma.Cloudflare.SandboxV1`** is the single selection point:
  `enabled?/0` is true only for `cf_environment == "staging"`;
  `image_enabled?/1` additionally requires an `unstable-*` validation image
  (the publisher pushes the matching `v1-unstable-*` tag); `v1_image/1`,
  `binding/0`, `container_config/3` and `drop_binding/1` keep the binding, the
  class migration and the container application keyed on the same predicate.
- **Provisioning and upgrade bindings** both add `WORKER_PLATFORM` and
  `SANDBOX_V1_CUTOVER`; the staging platform also gets the `SandboxV1` DO
  binding. The upgrade path (`Instances.instance_bindings/2`) previously lacked
  the platform bindings entirely — that was the root cause above.
- **`DispatchNamespace`** derives the `new_sqlite_classes` set from the
  declared container entries, so `SandboxV1` is migrated only when it is also
  bound (production stays legacy-only). `namespace_id_for_class/3` resolves the
  per-class namespace.
- **`Containers.deploy_script_and_container/1`** declares both container
  entries on staging/unstable images, prepares the v1 image artifact, and
  creates the per-tenant `…-sandbox-v1-<platform>` application with
  `scheduling_policy: "durable_object"`. It drops the `SandboxV1` binding when
  the image has no v1 tag, so an upload can never bind an unmigrated class.
- Deprovisioning removes both applications.

Verified locally (unit + full Lerma suite); the staging effect is **not yet
observed** because the running staging Lerma predates this change. The next
step is to deploy the branch Lerma to staging (`:latest-pre` lane), promote a
tenant, and repeat the checks above. Note the manual fallback: a promotion from
`develop` Lerma would re-upload the legacy binding set, so the branch Lerma
must be the one driving the upgrade.

## OCM-4a live validation on staging (2026-11-05)

The branch Lerma was temporarily deployed to the staging host under a unique
local image tag (transferred with `docker save`/`load`, compose repointed, then
restored to `:latest-pre` afterwards), a fresh validation build was published
(`unstable-8ce99552…`), and both tenants were upgraded through that Lerma.

### Verified live

| Check                    | Result                                                                                                                                                          |
| ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Worker bundle            | both tenant scripts at `unstable-8ce99552…`, identical script etag                                                                                              |
| Platform bindings        | `WORKER_PLATFORM = staging` and `SANDBOX_V1_CUTOVER = false` on both                                                                                            |
| v1 DO binding            | `SandboxV1` bound (prod namespace `c1b83745…`, dev `f50564f8…`)                                                                                                 |
| v1 container apps        | `tenant-client-sandbox-v1{,-dev}-sandbox-v1-staging`, `scheduling_policy: "durable_object"`, empty application config (image comes from `ctx.container.images`) |
| Named image              | script metadata carries `images.sandbox` as a digest-pinned Cloudflare Registry reference (`…/fouria@sha256:35b931…`)                                           |
| Legacy apps              | both on `fouria:unstable-8ce99552…`, 3/3 healthy, no errors                                                                                                     |
| Instance state           | both `running`, `HealthMonitor.check_instance_now/1` → `:ok`; `/healthz` → `302` (Access)                                                                       |
| Telemetry after recovery | no non-`ok` spans or exception classes; latest activity current                                                                                                 |
| Runtime selection        | still **legacy** — no conversion intent exists, so the v1 surface is inert as designed                                                                          |

### Incident during the first validation attempt (both scripts deleted)

The first two upgrade attempts used an earlier branch image whose v1
container-application body still contained `configuration.image` /
`instance_type` (a `durable_object` application rejects them). The API
rejected the app creation **after** the script upload, and the atomic-deploy
rollback deleted the just-uploaded script — for a live tenant that also removed
its routing, leaving both instances in `error` (the containers themselves stayed
healthy). Recovery was a re-run of `upgrade_instance` once the code was fixed.

Two code changes came out of it:

1. **`preserve_do_namespaces` now covers the script.** An upgrade rollback must
   never delete a live tenant's script (it is what routes the tenant); the
   newly uploaded script and its DO namespaces are kept and the failure is
   reported for a retry. Fresh provisions still clean up.
2. **The upgrade binding set is self-sufficient.** `Instances.instance_bindings/2`
   previously relied on `keep_bindings` for the per-instance/platform bindings,
   so a rebuild after a deleted script left the tenant without the gateway token,
   CDP secret, AI Gateway credential, Access AUD, agent roster, Telegram queue,
   and other vars. It now re-asserts the full set: `MOLTBOT_GATEWAY_TOKEN`
   (stored), a deterministically derived `CDP_SECRET`, AI Gateway/API token,
   `OPENCODE_API_KEY` (dev), the per-instance Access AUD (resolved by the
   tenant's own hostname), agent/scope flags, account/team vars, Telegram
   ingress queue/host, change-detection vars and `DEBUG_ROUTES`. Verified after
   recovery: 32 bindings on dev, 31 on production (only `DEBUG_ROUTES` is
   dev-only).

### Operational notes

- The staging Lerma is back on `:latest-pre`; the branch image was a temporary
  validation deployment. A later promotion from `develop` Lerma would again
  drop the v1 bindings, so the branch Lerma must drive the upgrade until this
  lands.
- The v1 container applications are idle (no instances) and start only when a
  converted tenant's DO calls `container.start()`.
- Next: S5 / PER-3 conversion orchestration (intent + `convertToV1` through the
  control plane) so a tenant can be converted and served on `SandboxV1`.

## S5 / PER-3 — legacy → v1 conversion, live (2026-11-05)

Bundle `unstable-e7d193fd…` promoted to `client-sandbox-v1` (id 20) from the
branch Lerma (`ghcr.io/0xcab0/lerma:v1-sandbox-02327309`, deployed temporarily).

| Check                          | Result                                                                                                                                                                         |
| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `POST /_control/:slug/migrate` | `200 { state: "committed", recordId: "77cadb66-cc11-4561-a6b5-edc1f1a52248" }` on `client-sandbox-v1-dev`                                                                      |
| Conversion record              | `sandbox-migration/client-sandbox-v1/conversion.json` — v1 `DirectoryBackup`, `dir=/home/openclaw`, 22.5 MB `tar+zstd/1`, source `f19e007a…`                                   |
| C8 barrier                     | `sandbox-migration/client-sandbox-v1/stable.json` points at the record (committed 12:04:26)                                                                                    |
| Runtime gate                   | `backup` on the converted instance → `503 SANDBOX_V1_CONTROL_UNSUPPORTED` (v1)                                                                                                 |
| Gate is per-tenant             | `backup` on `client-sandbox-v1` (still legacy) → `200` (no conversion intent)                                                                                                  |
| v1 container                   | `tenant-client-sandbox-v1-dev-sandbox-v1-staging` / instance `openclaw` `running`, image digest `sha256:bd925b80…`                                                             |
| **Served end-to-end**          | `GET /api/health` (health-monitor service token) → `200`, `status: "running"`, `startup.ok: true`, `readiness.ok: true`, `aiGateway`/`container`/`durableObject`/`r2` all `up` |
| Instance records               | both instances `running` on `unstable-e7d193fd…`                                                                                                                               |

### Defects found and fixed while making the conversion work

1. **A queued conversion stayed `queued`** — at the time this read as "the DO
   alarm never fires in the tenant runtime" (no alarm span, no error), so the
   control plane briefly awaited `v1ConvertNow()` inline (the legacy
   checkpoint's long-request pattern). **Corrected:** alarms do fire on this
   lane — the idle-snapshot cadence alarm later committed record `4437c278-…`
   with no operator action — so the inline path was removed. `migrate` is now
   queue-only (`202` + poll) and the conversion runs on the DO alarm, with the
   durable marker re-arm (`ensureConversionArmed`) covering a lost alarm. See
   `WORKER-DEVELOPMENT.md` § Sandbox v1 conversion orchestration (S5).
2. **`exec` stdin silently truncated the upload.** Passing a `ReadableStream` as
   `exec` stdin is accepted but did not deliver every byte (`unsquashfs` read a
   partial file and exited 2, while the R2 object was a valid SQUASHFS 4.0 lz4
   image, verified locally and in the sandbox base image). Uploads now use
   `stdin: 'pipe'`, close the pipe before awaiting the exit code, and surface the
   extractor's stderr.
3. **The archive is not a copy of `/home/openclaw`.** The legacy bridge stages
   the roots under `.fouria-legacy-stage/{openclaw,clawd}` and leaves
   `/home/openclaw/.openclaw` + `/home/openclaw/clawd` as symlinks into `/root`.
   The v1 image has real directories (a `DirectoryBackup` does not follow
   symlinks), so extraction now materializes the staged roots as real v1
   directories and relocates the absolute legacy paths (`/root/openclaw` →
   `/home/openclaw`, length preserving and applied in place to the state SQLite
   with a `node:sqlite` `quick_check`; `/root/clawd` → `/home/openclaw/clawd` in
   text files only). A snapshot without a complete legacy stage fails closed.
4. **Serving before the barrier committed.** A present intent used to resolve as
   `v1` immediately, so a failed conversion could wake a fresh empty v1 container
   and onboard over the legacy recovery point. The gate now requires the record +
   pointer (C8) and answers `503 SANDBOX_V1_CONVERSION_PENDING` until then.
5. **A gateway launch failure retracted a committed conversion.** `convertToV1`
   no longer launches: the commit is the data artifact and a launch failure is
   logged (`sandbox.v1.conversion.launch_failed`) and retried by the next
   request's v1 boot, instead of reporting `failed` and re-running the
   destructive conversion on retry.
6. **Lerma marked a healthy rollout as `error`.** `await_container_healthy`
   required `active_rollout_id` to clear, but the platform keeps reporting it
   after every instance serves the new image (both tenants: 3/3 healthy for the
   whole 10-minute budget). Completion is now "no active rollout **or** the
   application image carries the target version".

### Pre-flight proof of the materialization

`materializeLegacyArchive` was driven against the real stable archive inside the
published v1 image (a `docker exec`-backed fake container): the extracted state
boots `start-openclaw.sh start` with `OPENCLAW_RESTORED=true`, `/health` and the
Control UI answer `200`, the agent workspace is present, and the state database
reports `quick_check: ok` with zero `/root/openclaw` references.

### Known gaps (next slices)

- The **Control UI** (`GET /`) sits behind the tenant _host_ Access application,
  which rejects the health-monitor service token (`service_token_status: false`
  in the login meta) and needs an interactive identity, so the browser surface
  could not be exercised headlessly. The health path is a separate, token-
  accepting Access app and is what proved readiness (`status: running`,
  `startup`/`readiness` ok, all components up).
- v1 control actions other than the idempotent `migrate` still answer
  `503 SANDBOX_V1_CONTROL_UNSUPPORTED` (checkpoint/backup/resume on v1).
- Only the dev-env instance was converted; the production-env staging instance
  remains legacy until the S6 fleet decision.

### Deployment state left behind

- Staging Lerma (`root@46.225.138.152`, container `lerma-pre`) runs the branch
  image `ghcr.io/0xcab0/lerma:v1-sandbox-02327309`; the original compose file is
  kept at `/opt/lerma/docker-compose.yml.bak-v1`. Roll back with
  `/tmp/rollback-lerma.sh` (or restore that backup and `docker compose up -d
  --no-deps --force-recreate app`).
- The staging Dispatch (`dev.fouria.io`) is the branch deploy, so it still
  forwards `migrate` and echoes the conversion state. A develop deploy replaces
  it and drops that forwarding.
- Both are deliberately left in place while the v1 lane continues (the
  converted tenant needs the v1-aware upgrade path); they are staging-only and
  must be reverted before any production promotion.

## PER-2 — v1 snapshots, recovery point and restore drill (2026-11-05)

Bundles `unstable-6795827b…` (catalog + snapshots + freeze) and
`unstable-473c46b0…` (pre-catalog adoption) promoted to `client-sandbox-v1`
(id 20); the drills ran on `client-sandbox-v1-dev`.

| Check                         | Result                                                                                                                                                            |
| ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `POST /_control/:slug/backup` | `200`; `sandbox.v1.snapshot.committed` (record `79165e52…`, 22.5 MB) — served by the DO, no `.sqsh` written                                                       |
| Catalog + pointer             | `sandbox-migration/client-sandbox-v1/records/<id>.json`; `stable.json` advanced to the new record                                                                 |
| Encryption at rest            | `sandbox-v1/openclaw/<id>.tar.zst.enc` (22.5 MB) + `<id>.meta.json` (`encrypted: true`, plaintext size/sha256) — **no plaintext** object                          |
| `POST /_control/:slug/freeze` | `200`; snapshot committed **then** the container was destroyed (`status: stopped`)                                                                                |
| Cold boot restore             | one non-passive `GET /api/health` (health-monitor token) → new container instance `started 17:18:17` → `200`, `running`, `startup.ok`/`readiness.ok` within ~15 s |
| Retention                     | 4 records → 3; the oldest catalogued record's entry **and** 22.5 MB ciphertext removed, the new record protected                                                  |
| Pre-catalog adoption          | a conversion record written before the catalog existed was adopted on the next snapshot (fix `473c46b0`)                                                          |

Notes:

- A `durable_object` container application is an **instance application**: the
  rollout API rejects it (`Instance applications do not support rollouts`), so a
  container replacement is either the platform's own instance churn or the DO's
  `freeze` → next-request boot. `freeze` therefore doubles as the restore drill
  lever.
- A non-passive `/api/health` boot is **detached**: the first probe answers `503
  stopped` in ~0.5 s while the registered boot runs, and the next probe is
  `200 running`. The loading page's startup WebSocket drives the same boot.
- Retention only manages catalogued records; a record committed before the
  catalog existed needs adoption, which is why a snapshot now adopts
  `conversion.json` before listing (found live: the pruned record's payload was
  deleted correctly, the un-catalogued conversion archive was not).

## OCM-2 — durable lanes, boot guards and repair (2026-11-05)

Bundle `unstable-889ec3a2…` promoted to `client-sandbox-v1` (id 20); drills on
`client-sandbox-v1-dev`.

| Check                        | Result                                                                                                                                             |
| ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| Scheduler runtime resolution | the alarm snapshot after the promotion wrote **v1** record `2c97395d…` (catalog + pointer), not a legacy archive                                   |
| Legacy lane untouched        | legacy app instance stays `inactive`; `backups/f19e007a…` and `backup-handle.json` still date from 2026-11-04                                      |
| Freeze on the new bundle     | `200`, `backupId: 8910f736…`; pointer advanced to it                                                                                               |
| Cold boot after the freeze   | instance `started 17:50:25`; `503 stopped` ×3 (~36 s, detached boot) → `200 running` with `startup.ok`/`readiness.ok`                              |
| Launch guards (unit)         | non-zero early exit fails in seconds (not after the 300 s budget); a corruption signature queues `doctor-repair` (`gateway-launch-state-database`) |

Notes:

- A v1 upgrade redeploy runs `checkpoint` (v1 snapshot), not `checkpoint-park`:
  park is still legacy-shaped and belongs to SEC-3, so the rollout is not
  write-blocked on v1 yet.
- The scale-to-zero path is the platform inactivity stop (no DO idle callback),
  so "stop without re-archiving on idle expiry" remains open (PER-2 note).
- The corrupt-state-DB repair is covered by unit tests only: forcing real SQLite
  corruption on the staging tenant would risk the tenant for no extra signal.

## SEC-3 — v1 rollout park and resume (2026-11-06)

Bundle `unstable-548cb408…` promoted to `client-sandbox-v1` (id 20); the drill ran
on `client-sandbox-v1-dev`.

| Check                    | Result                                                                                                       |
| ------------------------ | ------------------------------------------------------------------------------------------------------------ |
| `checkpoint-park`        | `200 { parked: true, backupId: 2455d8c8…, sessionsPatched: 0, sessionsDrained: false }` (no live sessions)   |
| Recovery point committed | the pointer advanced to the park record before the write lock armed                                          |
| Write while parked       | `POST /_control/:slug/backup` → `409 TENANT_PARKED` (echoed by the dispatch as `code`)                       |
| `resume`                 | `200 { resumed: true }`; the park record was cleared                                                         |
| Readiness after resume   | health probe `200 running` on the first attempt (park never destroys the container, so no restore is needed) |
| Unparked resume          | no-op that never wakes the tenant (unit-tested)                                                              |

Notes:

- The Lerma promotion of this bundle used `checkpoint` (the direct-upgrade path),
  not `checkpoint-park`, and logged `Pre-replacement checkpoint unconfirmed` plus
  a skipped legacy job: the v1 payload carried `recordId` but the control
  contract names the committed archive `backupId`, so the dispatch did not echo
  the evidence. `fdb1f639` returns both.
  **Follow-up (closed):** right after the promotion the runtime still answered a
  plain `checkpoint` without `backupId` (the `checkpoint-park` branch from
  `548cb408` was already serving), so a bundle-blackend question was recorded.
  A re-deploy of the same bundle settled it: `POST …/checkpoint` now returns
  `{ ok, action: "checkpoint", backupId: "dd0b1be3-7116-4a7d-b970-21f54b08cc5d" }`
  and the deployed content carries the alias. The transferable finding is the
  **preflight asymmetry**: `upgrade_all_instances_async` skips an instance when
  the _live container image_ already matches the requested version, so a script
  upload that did not take effect cannot be repaired by re-promoting. Recorded
  as a follow-up (verify the deployed bundle, not only the image) rather than
  fixed in this slice.
- Park is intentionally a **write-block only**: the container is replaced by the
  rollout, not by the park, so `resume` needs no restore. `killGateway` stays
  process-scoped.
- The session drain found nothing to patch on this tenant; the read-only drain and
  the durable mode map are exercised by unit tests (`session-drain.test.ts`,
  `session-mode-restore.test.ts`).

## SEC-3 (Lerma side) — the direct bundle upgrade parks and resumes (2026-11-06)

`Instances.upgrade_instance/3` (the path the staging validation promotion API
uses) re-uploaded the bundle and rolled the container without write-blocking the
tenant. It now mirrors `@upgrade_pipeline`: parked checkpoint first (fail-closed
on a failed or non-parked response), released after the rollout settles and on
every error path, sleeping instances skipped without a resume request.

Verified live against the branch Lerma image `v1-sandbox-e5dd5278` on
`client-sandbox-v1-dev`:

| Check                      | Result                                                                                                                                  |
| -------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| `Containers.checkpoint/2`  | `{:ok, %{backup_id: "545f7844-…", parked: true, checkpointed: true}}` (direct probe)                                                    |
| `Containers.resume/1`      | `{:ok, %{resumed: true}}`                                                                                                               |
| Upgrade parks              | a full `upgrade_instance` moved the v1 recovery point to record `d1c9f143-1aba-4bc1-b8e0-31522497eb06` (committed inside the upgrade)   |
| Upgrade releases           | `POST …/backup` → `200` (not `409 TENANT_PARKED`) and `/api/health` `200 running` right after                                           |
| Missing park (not running) | the first drill ran while the container was idle: `skipped: not-running` was accepted, no resume was sent, and the pointer did not move |

Note: logs emitted inside `lerna rpc` probes are not visible in `docker logs`
(the RPC process does not own the console), so this drill is evidenced by the
durable v1 pointer + the post-upgrade write acceptance rather than by Lerma log
lines. A promotion-driven run (Task in the app) is the way to read those lines.

## Rollout park through the real promotion lane (2026-11-06)

Bundle `unstable-fd05f0c3…` (the v1 outbound-bridge fix) was promoted through
`Lerma`'s promotion API, i.e. the same `Task`-driven `upgrade_instance/3` path a
release uses. This is the first end-to-end park → replacement → resume run:

| Phase            | Evidence                                                                                                                       |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| Park             | `16:06:41 Pre-replacement checkpoint parked for tenant-client-sandbox-v1-dev: backup_id="d1c02116-…" parked=true`              |
| Recovery point   | `sandbox-migration/client-sandbox-v1/stable.json` → `d1c02116-5363-4ab8-bb3e-388a776ce03d` (committed before the upload)       |
| Script + rollout | script `PUT …/scripts/tenant-client-sandbox-v1-dev`, then the container application rolled to `unstable-fd05f0c3…`             |
| Settle           | `Waiting for … container health … healthy 3`, `rollout_active: false`                                                          |
| Resume           | `16:09:21 Resume after container replacement for tenant-client-sandbox-v1-dev: resumed=true`                                   |
| After            | `POST …/backup` → `200` (park cleared); `/api/health` → `200 running`, `readiness.ok: true`; instance `running` on `fd05f0c3…` |

### Defects found by the drill

1. **Premature convergence.** The promotion gate read the DB only:
   `upgrade_all_instances_async` sets `deployed_version` to the requested version
   **before** deploying, and `HealthMonitor.mark_healthy/3` flipped the in-flight
   `upgrading` instance back to `running` (the container being replaced still
   answers `200`). The gate therefore declared success ~30 s into a rollout that
   took ~2m40s. Fixed: the status endpoint reports `serving_version` (the live
   container-application image tag), `promote-validation.sh` requires it to match,
   and the health monitor no longer transitions `upgrading` instances.
2. **Stuck rollout carried across promotions.** `tenant-client-sandbox-v1`
   (prod-env) still had `rollout_active: true` on the **previous** image
   (`unstable-548cb408…`) from the earlier promotion, so the fd05 promotion timed
   out settling it and marked the instance `error` for hours; because the gate
   ignores `error` instances, the promotion still reported success. Re-running
   `upgrade_instance` superseded the stuck rollout and the instance is now
   `running`/healthy on `fd05f0c3…`. Follow-up worth doing: for a converted
   tenant, gate on the **`SandboxV1` application** (or skip the legacy app), since
   the legacy app is unused by the v1 runtime and can wedge independently.
3. **`upgrade_instance` reliability verified** for the Lerma-side park: sleeping
   containers report `skipped: not-running` (no park, no resume), a failed or
   plain (`parked: false`) checkpoint aborts before the upload, and the park is
   released on the error path too (all covered by
   `test/lerma/instances_upgrade_test.exs`).

### Still open for this bundle

The v1 **outbound virtual hosts** fix (`.github`-published `src/sandbox-outbound.ts`

- `SandboxV1` intercept registration) is unit-verified and live on both instances
  (`fd05f0c3…`), but its runtime effect (the container reaching `otel.internal`)
  has no operator-visible signal in the v1 lane yet: the tenant was idle, so no
  model spans were exported and no `[otel] Received …` log line was produced.
  Confirm on the next agent turn (or when an operator checks the tenant's
  `diagnostics-otel` health).

## PER-2 — restore by record id on v1 (2026-11-06)

`POST /api/admin/storage/restore/:id` on a converted tenant used to fall into the
legacy `.sqsh` path (unsupported on v1), so an operator could not apply an older
v1 recovery point. The route now branches on the runtime:

| Step               | Behaviour                                                                                                                                    |
| ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------- |
| Maintenance window | opened (`restore`) **before** the DO touches the container; the v1 `ensureGateway` branch honors it, so no request can reboot under the swap |
| Stop writers       | DO kills the single gateway owner, then waits (≤5 s) for `/health` to fail so the SDK swap does not hit `EBUSY`                              |
| Apply the record   | `DirectoryBackup.restore(record, { dir: HOME_DIR })` — the SDK swaps the directory atomically                                                |
| Recovery point     | by default the stable pointer advances to the restored record (`?setCurrent=0` applies it **without** promoting)                             |
| Fail-closed        | unknown/unreadable record → `404`, container untouched; a restore already in progress is refused by an in-DO flag                            |
| Boot               | Worker boots the restored state and reports `gatewayReady`; a boot failure never undoes the restore (next request retries)                   |

v1 restores are synchronous (there is no legacy job queue for the record format).

**Live drill status:** the code path is unit- and corpus-verified
(`sandbox-v1.test.ts`, `storage.test.ts`, `business-rules/sandbox-v1-persistence.txt`),
but the live drill is **blocked by Access**: `/api/admin/*` on a tenant host is
behind the host Cloudflare Access app, and the only service token available to
this lane (the health monitor) is accepted by the health app, not the host app.
An operator session (or a browser login) is required to exercise the route on
`client-sandbox-v1-dev`; the bundle `unstable-9952203a…` ships the code.

## Promotion gate fix verified live (2026-11-06)

`unstable-9952203a…` (v1 restore-by-record) was promoted through the validation
lane after redeploying the branch Lerma that carries the new gate:

- the branch Lerma build `v1-sandbox-9952203a` exposes `serving_version` in
  `GET /api/provisioning/upgrade/:slug`;
- the promotion **waited for the real rollout**: `Lerma accepted …` at the start,
  then convergence only after both instances reported
  `serving_version == unstable-9952203a…` (`rollout_active: false`, healthy 3);
  the pre-fix run had declared success ~30 s in while the rollout was mid-flight;
- park/resume evidence on the dev instance: recovery point advanced to
  `26c78dac-ffcb-4574-a4b8-5b82c4be792b` (committed at 21:57:22, before the image
  rollout), and `POST /_control/…/backup` afterwards returns `200` (the park was
  released by `resume`);
- final state: both instances `deployed_version = serving_version =
  unstable-9952203a…`, container health 3/3, `tenant-client-sandbox-v1-dev`
  `running`, `tenant-client-sandbox-v1` `sleeping` (idle container).

Operational note: the promotion gate requires `serving_version`, so the Lerma
image running on staging must include that change; an older Lerma image makes a
promotion hang until its timeout (it never sees the field). The staging Lerma was
redeployed from `v1-sandbox-9952203a` (compose backup:
`/opt/lerma/docker-compose.yml.bak-latest-pre`, which still points at
`latest-pre`).

## Combined promotion: promotion-gate repair + v1 idle snapshot cadence (2026-11-07)

Two slices landed together and shipped as **one** validation bundle, so the
staging lane picked up the cadence and the promotion-gate repair in a single
rollout:

| Commit     | Slice                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `4389a6e4` | **Lerma promotion repair + truthful gate.** `POST /api/provisioning/upgrade/:slug` also retries `error` instances whose `deployed_version` is not the requested version (an instance already declaring the target is excluded, since re-driving a broken container at the target only loops). `promote-validation.sh` `is_converged` now requires **zero** `error` instances, so a promotion can no longer report success while a tenant stays down at the old image.                                                                                                                                                                                                                                                                                                                                                                          |
| `855810b9` | **v1 idle snapshot cadence (`PER-2`).** A converted tenant is owned by `SandboxV1`, and raw `ctx.container` has no eviction hook, so the platform's inactivity stop silently dropped writes since the last committed record. The DO now arms a durable marker + `ctx.storage.setAlarm` on `startContainer()` and, on `alarm()`, commits a `backup` snapshot then re-arms. It never wakes a stopped container, skips while a restore is in flight, and skips when `stable.json` is newer than the interval (deduping with the `ActivityScheduler`). `resolveSnapshotIntervalMs` resolves `SANDBOX_V1_SNAPSHOT_INTERVAL_MS` → `BACKUP_INTERVAL_MINUTES` → 1 h, capped at **half** the inactivity timeout (staging 7.5 m, dev 1 h). `selectRecordsToPrune` breaks a same-millisecond `createdAt` tie by record id, so retention is deterministic. |
| `795f1dab` | **Test hygiene (unrelated to the track).** The `ensure_container_application/5` keyless dev-SSH test now pins its own `:provisioning[:dev_instance_ssh_public_key]`; `config/runtime.exs` loads the developer-local `apps/lerma/.env` (which sets `DEV_INSTANCE_SSH_PUBLIC_KEY`), so the test only passed on a machine without that entry.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |

### Autogenerated-types covenant (checked, no fix needed)

`855810b9` adds `SANDBOX_V1_SNAPSHOT_INTERVAL_MS?: string` to
`apps/fouria/src/types.ts`. `apps/fouria/AGENTS.md` calls that file
autogenerated, but `bun run types` (`wrangler types`) writes
`apps/fouria/worker-configuration.d.ts` only — `src/types.ts` is the
hand-maintained `OpenClawEnv`, and the call site already narrows it with its own
`SnapshotCadenceEnv` (`{ SANDBOX_V1_SNAPSHOT_INTERVAL_MS?, BACKUP_INTERVAL_MINUTES? }`).
The addition therefore stands, no call-site cast was needed, and the regenerated
`worker-configuration.d.ts` (unrelated workerd/binding churn) was reverted rather
than committed.

### Deployment

- Bundle + both runtime images: `unstable-795f1dab0d6b2d4bce7b6a98c9b5afb3be089e1f`
  (`fouria:unstable-…` + `fouria:v1-unstable-…`, R2
  `fouria-bundles/unstable-…/fouria.mjs` and `container-manifest.json`).
- Staging Lerma image: `ghcr.io/0xcab0/lerma:v1-sandbox-795f1dab`, built from the
  repo root and shipped by `docker save | ssh … docker load` (no ghcr push), compose
  backup `/opt/lerma/docker-compose.yml.bak-pre-795f1dab`, container `lerma-pre`
  `healthy` and `/health` `200`.
- One promotion for everything: `Lerma accepted unstable-795f1dab…: 2 running
  instance(s)` → `client-sandbox-v1 is running unstable-795f1dab…`. The direct
  bundle-only path (`Instances.upgrade_instance/3`) completes park → script
  upload → container rollout → resume **inside** the POST, so the gate converged
  on its first poll (park/resume visible in the Lerma log below).

### Live drills (2026-11-07, `unstable-795f1dab…`)

| Drill                      | Result                                                                                                                                                                                |
| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Status gate                | both instances `deployed_version = serving_version = unstable-795f1dab…`, `status` `running` / `sleeping`, `container_health.errors: 0`, `healthy: 3` / `2` (`rollout_active: false`) |
| `checkpoint-park` (dev)    | `200` `parked: true`, `backupId: 63f79239-9d43-4824-96b0-83f37e34b71e`, `sessionsPatched: 0` (no live sessions)                                                                       |
| `resume` (dev)             | `200` `resumed: true`                                                                                                                                                                 |
| `backup` after resume      | `200` (`ok: true`) — proves the park guard was released (a parked tenant answers `409 TENANT_PARKED`); the dispatch projection only echoes `backupId` for park/freeze                 |
| Recovery point (R2)        | `sandbox-migration/client-sandbox-v1/stable.json` → `recordId 1a169e07-cb50-444c-8ae5-5976bfe72a20`, `committedAt 2026-11-07T08:50:12Z` (written by the drills)                       |
| Lerma park/resume evidence | `Pre-replacement checkpoint parked for tenant-client-sandbox-v1-dev … parked=true` (08:46:35Z) and `Resume after container replacement … resumed=true` (08:48:39Z)                    |
| Idle cadence               | **unit-verified, live observation pending the next hourly tick** (dev cadence is capped at 1 h); it is not waited on                                                                  |

Verification for this bundle: `bun run format:check && bun run lint &&
bun run typecheck && bun run test` (3974 passed, 8 skipped), `cctr
apps/fouria/test/business-rules/` (582/582), `scripts/staging-validation.test.ts`
(83/83), `bun run plan:sandbox-v1:check`, `mix check`, `mix license_audit`, and
the full Lerma suite (`1242 passed, 4 excluded`).

## CDP bridge merge fix + HEAD redeploy (2026-11-07)

The develop merge (`806a2580`) brought the CDP bridge (#563) in, but the merge
resolution left `cdp.internal` out of the shared outbound host map: the
v1-side refactor had moved `TenantSandbox.outboundByHost` into
`src/sandbox-outbound.ts`, so `handleCdpRequest` existed while no route
referenced it. Two `apps/fouria/src/index.test.ts` node-pool suites failed on the
merged tree (`4020 passed, 8 skipped, 2 failed`) — `format:check`, `lint` and
`typecheck` stayed green because the map is generic, so this was a
merge-resolution regression rather than a type error, and because both runtimes
consume the same map v1 was missing the intercept too.

| Commit     | Slice                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `e56ac9c8` | **`cdp.internal` joins the shared host map.** `fix(fouria): register the cdp.internal bridge on the shared outbound host map` adds `[CDP_INTERNAL_HOST]: handleCdpRequest` to `SANDBOX_OUTBOUND_ROUTES` (6 hosts: `releases`/`docs`/`leads`/`software.r2.internal`, `otel.internal`, `cdp.internal`), drops the bridge imports left dead in `src/index.ts`, and pins the host in `sandbox-outbound.test.ts` / `sandbox-v1.test.ts`; the two node-pool suites now stub `@cloudflare/sandbox-legacy`, which the CDP bridge pulls in transitively. |

### Deployment

- Bundle + both runtime images: `unstable-e56ac9c8d79f062ee6b66bfed1f221e1e82f102f`
  (`fouria:unstable-…` + `fouria:v1-unstable-…` on `registry.cloudflare.com`, R2
  `fouria-bundles/unstable-…/{fouria.mjs,container-manifest.json}`).
- Staging Lerma image: `ghcr.io/0xcab0/lerma:v1-sandbox-e56ac9c8`, built from the
  repo root and shipped by `docker save | ssh … docker load` (no ghcr push),
  compose backup `/opt/lerma/docker-compose.yml.bak-pre-e56ac9c8`, container
  `lerma-pre` `healthy` and `/health` `200`.
- One promotion for everything: `Lerma accepted unstable-e56ac9c8…: 2 running
  instance(s)` → `client-sandbox-v1 is running unstable-e56ac9c8…`. The bundle-only
  path parked both instances (`v1-dev` 20:02:50Z, `v1` 20:07:39Z), replaced the
  container and resumed each (`v1-dev` 20:07:16Z, `v1` 20:11:01Z).

### Live drills (2026-11-07, `unstable-e56ac9c8…`)

| Drill                      | Result                                                                                                                                                                                                         |
| -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Status gate                | both instances `deployed_version = serving_version = unstable-e56ac9c8…`, `status` `running`, `container_health.errors: 0`, `healthy: 2` / `3`, `rollout_active: false`                                        |
| `checkpoint-park` (dev)    | `200` `parked: true`, `backupId 8f4e88bc-368a-48c5-888e-30317895f6f8`, `sessionsPatched: 0` (no live sessions)                                                                                                 |
| `resume` (dev)             | `200` `resumed: true`                                                                                                                                                                                          |
| `backup` after resume      | `200` (`ok: true`) — the park guard is released (a parked tenant answers `409 TENANT_PARKED`)                                                                                                                  |
| Recovery point (R2)        | `sandbox-migration/client-sandbox-v1/stable.json` → `recordId 0e426f0e-577b-40a1-a4a6-c0abcafaa20e`, `committedAt 2026-11-07T20:12:55Z`, `trigger: "backup"` (the park's checkpoint record is `8f4e88bc-…`)    |
| `cdp.internal` (deployed)  | both tenant scripts in `fouria-tenants-staging` (10,767,930 bytes, same content) contain `cdp.internal`, and `SANDBOX_OUTBOUND_ROUTES` lists all **6** hosts including `[CDP_INTERNAL_HOST]: handleCdpRequest` |
| Lerma park/resume evidence | `Pre-replacement checkpoint parked for tenant-client-sandbox-v1-dev … parked=true` (20:02:50Z) / `Resume … resumed=true` (20:07:16Z); same pair for `tenant-client-sandbox-v1` (20:07:39Z / 20:11:01Z)         |
| Idle cadence               | still `[suite]` only — **not** waited on for this deploy (dev cadence is capped at 1 h)                                                                                                                        |

**What is still unmeasured.** The `cdp.internal` _registration_ is now observed in
the deployed bundle on both runtimes; the end-to-end browser round-trip (a
converted tenant driving Browser Rendering through the container's
`cdp-proxy.mjs`) was **not** exercised — `test/e2e/r_browser_remote_profile.txt`
is cloud-profile only and was deliberately not run.

Verification for this bundle: `bun run format:check && bun run lint &&
bun run typecheck && bun run test` (4022 passed, 8 skipped, 199 files), `cctr
apps/fouria/test/business-rules/` (593/593), `scripts/staging-validation.test.ts`
(83/83), `bun run plan:sandbox-v1:check`, `mix check`, `mix license_audit`, and
the full Lerma suite (1259 passed, 4 excluded).

## Operator surface: Lerma migration + ROOT debug tools, and the `already_converted` signal (2026-11-07)

Two slices that make a conversion reachable by an operator instead of by a
hand-rolled control call, plus the correctness fix that came out of validating
them live. Nothing here converts a tenant.

| Commit     | Slice                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `c01d00b1` | **ROOT debug tools work on Sandbox v1.** Every `/debug/*` route was audited against the client a converted tenant actually serves with (`V1SandboxClient`): a shared `readFinishedProcessLogs()` (legacy `logs()` → v1 `output()`) fixes `/processes?logs=true` and `/logs?id=` for a finished process; `/gateway-logs` returns `path` + `runtime`; `/restart-openclaw` gained a v1 branch (no legacy checkpoint, no `restore-needed`); `/reprovision` checkpoints with `v1CreateSnapshot` and validates restorability with the C8 `verifyRecoveryPoint`; `/backup/:mode` snapshots through the DO; `/destroy-container` clears the startup cache and the isolate start dedupe; `/env` reports `sandbox_runtime_actual`; `/debug/sleep` answers `503 SANDBOX_V1_SLEEP_UNSUPPORTED` (SDK 1.0 has no stop primitive). Suite-verified only (see the drill table). |
| `804c512e` | **Lerma can migrate a legacy tenant.** `Containers.migrate/2` posts the `migrate` control action with a 30-minute budget and normalises `{state, record_id, duration_ms}`; `SandboxV1Migration` computes eligibility (staging platform, never the reserved `client` slug, a client's `dev` environment only, active instance, committed legacy recovery point) and the dry-run report; `POST /api/provisioning/migrate/:slug` serves `?dry_run=1` plus real runs (`200`/`403`/`409`/`502`); `mix lerma.sandbox_v1.migrate` prints the same table and requires `--yes` off a TTY.                                                                                                                                                                                                                                                                               |
| `6602f1fc` | **The dry-run stops calling an already-converted tenant eligible.** A conversion leaves the legacy archive in R2 (it is the conversion's source), so the legacy recovery point alone cannot tell: `Containers.v1_recovery_point/1` reads the v1 stable pointer (`sandbox-migration/<canonical tenant>/stable.json`, in the same per-instance bucket the legacy recovery point resolves), `R2.object_body/2` supplies the record id, and `SandboxV1Migration` emits `already_converted` (`true`/`false`/`nil` when unreadable) + `v1_record_id` per instance and at the top level, as both the blocking reason and a warning. A real run on a converted tenant answers `200 {state: "committed", already: true, record_id}` **without** a control call, so a no-op never occupies the 30-minute request path.                                                   |

### Deployment (2026-11-07)

- Bundle + both runtime images: `unstable-6602f1fc98fc41692678ea05718adde8505f6b56`
  (`fouria:unstable-…` + `fouria:v1-unstable-…` on `registry.cloudflare.com`, R2
  `fouria-bundles/unstable-…/{fouria.mjs,container-manifest.json}`).
- Staging Lerma image: `ghcr.io/0xcab0/lerma:v1-sandbox-6602f1fc`, built from the
  repo root and shipped by `docker save | ssh … docker load` (no ghcr push);
  compose backup `/opt/lerma/docker-compose.yml.bak-pre-6602f1fc`; `lerma-pre`
  `healthy` and `/health` `200`.
- One promotion for everything: `Lerma accepted unstable-6602f1fc…: 2 running
  instance(s)` → `client-sandbox-v1 is running unstable-6602f1fc…`. The park
  checkpoints were `556d5773-…` (dev, parked 21:34:57Z → resumed 21:40:02Z) and
  `5eb28f82-…` (production-env, parked 21:40:31Z → resumed 21:44:10Z).
- Status gate afterwards: both instances `status: running`,
  `deployed_version = serving_version = unstable-6602f1fc…`,
  `container_health.healthy` 3 (dev) / 2, zero errors.

### Idle cadence — first live tick (observed, not injected)

`sandbox-migration/client-sandbox-v1/stable.json` advanced to
`4437c278-baf2-4ec2-af7f-fa9598f5eef7` at `2026-11-07T21:18:55.381Z` with
`records/4437c278-….json` recording `trigger: "backup"` — a scheduled v1
snapshot committed with no operator action and no promotion in flight (the
`trigger: "checkpoint"` records are the promotion parks). This is the live tick
the earlier entries listed as pending; the ~1 h dev interval matches
`resolveSnapshotIntervalMs`.

### Live drills (2026-11-07, `unstable-6602f1fc…` + `v1-sandbox-6602f1fc`)

All requests used the provisioning application key with
`X-FourIA-Application: fouria`; the key is never printed.

| Drill                                           | Result                                                                                                                                                                                                                                                                                                                                            |
| ----------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Dry-run, converted tenant (`client-sandbox-v1`) | `200`, `eligible: false`, `reasons: ["already_converted"]`, `warnings: ["already_converted"]`, top-level `already_converted: true`, `v1_record_id 556d5773-5ea2-4fff-b777-3221234dcaf1`. Per instance: `tenant-client-sandbox-v1-dev` `already_converted: true`; `tenant-client-sandbox-v1` `already_converted: false` + `production_environment` |
| Dry-run, production-environment instance        | `?instance=tenant-client-sandbox-v1` → `eligible: false`, `reasons: ["production_environment"]`, `already_converted: false` (its own bucket has no pointer — it is still legacy), warning `checkpoint_recommended_before_conversion`                                                                                                              |
| Dry-run, reserved slug (`client`)               | `409`, `code: MIGRATION_NOT_ELIGIBLE`, `reasons: ["reserved_client"]`, no control call                                                                                                                                                                                                                                                            |
| Dry-run, non-existent client (`client-sandbox`) | `404` — the slug briefed for the negative gate does not exist on staging; the production-environment refusal was recorded on `client-sandbox-v1`'s own production instance instead                                                                                                                                                                |
| **Idempotency drill** (real run, no `dry_run`)  | `200` in **1.16 s**: `{state: "committed", already: true, record_id: 556d5773-…, duration_ms: 0}` — no control call, so no 30-minute no-op path                                                                                                                                                                                                   |
| Recovery point unchanged by that run            | `stable.json` `556d5773-…` @ `21:34:56.087Z` **before and after**; `conversion.json` still `77cadb66-cc11-4561-a6b5-edc1f1a52248` @ `2026-11-05T12:04:26Z` (no new conversion record)                                                                                                                                                             |
| Tenant serving after the promotion              | `GET /api/health` on `staging-client-sandbox-v1.fouria.io` and `…-dev.fouria.io` (health-monitor service token): `200`, `status: running`, `startup.ok` + `readiness.ok` true, `container`/`durableObject`/`r2`/`aiGateway` up                                                                                                                    |
| ROOT debug tools                                | **Not verified live:** `GET /debug/version` and `/debug/gateway-logs` on the tenant host answer `302` to `paso4.cloudflareaccess.com` (`service_token_status: false`), so a headless call cannot reach them. Unchanged from the earlier entries — suite-verified only                                                                             |

**What this bundle did not do.** No tenant was converted. The production-env
instance of `client-sandbox-v1` is still legacy (`already_converted: false`), and
policy keeps it that way: only a client's `dev` environment tenant is eligible
(contract C2/S7). No `migrate` control action was sent during these drills.

Verification for this bundle: `bun run format:check && bun run lint &&
bun run typecheck && bun run test` (4037 passed, 8 skipped, 199 files), `cctr
apps/fouria/test/business-rules/` (596/596), `scripts/staging-validation.test.ts`
(83/83), `bun run plan:sandbox-v1:check`, `mix check`, `mix license_audit`, the
full Lerma suite (1305 passed, 4 excluded) and `workers/fouria-dispatch`
`--project unit` (127 passed).

## Durable migration job + read-only status, live-validated (2026-11-08)

The `migrate` control action stopped being a synchronous conversion. It is now a
**queue-only enqueue** of a durable job on the tenant's `SandboxV1` Durable
Object, and polling moved to a **separate read-only action** so that asking for
the state can never convert a tenant. Nothing here converts a tenant: the
converted instance in this drill was converted on 2026-11-05.

| Commit     | Slice                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `be255fdd` | **The conversion runs as a durable Durable Object job.** `migrate` calls `v1RequestConversion()` and answers immediately (`202` `queued`/`running`, `200` `committed` + `recordId`, `502` `failed`, `503 SANDBOX_V1_UNAVAILABLE` when the binding/DO is unreachable — nothing recorded, safe to retry). The conversion itself streams the legacy squashfs, extracts it in the v1 container and uploads a `DirectoryBackup` on the DO `alarm()` (conversion first, cadence second, each wrapped), is durable (`migration:convert-request` marker + `migration:convert-status` + 1-based `attempt`) and idempotent, `alarm()` orders `running` → `committed` **before** dropping the marker and before the launch (a launch failure never retracts the commit), a failure records `failed` with a bounded message instead of throwing, `ensureConversionArmed()` re-arms a lost alarm from `conversionState()`/`requestConversion()`/`status()`/`ensureGateway()` without waking the container, and `#conversionRun` is the single owner so overlapping alarm turns cannot start a second destructive conversion. `v1ConvertNow` was removed from the DO, the RPC surface and `V1SandboxClient`. |
| `92a21080` | **`migration-status` is a read-only action.** `X-FourIA-Control: migration-status` calls **only** `v1ConversionState()` — never `v1RequestConversion()` — so it cannot enqueue, start work or wake the container; a `legacy` tenant answers `200 {state: "absent"}` without addressing a v1 DO. It never answers `202`. The Dispatch Worker forwards it (`CONTROL_ACTIONS`, control echo of `state`/`recordId`, no snapshot CPU budget). Lerma's `Containers.migration_status/2` normalises `absent`/`queued`/`running`/`committed`/`failed`. `a3b3e162` is the Lerma half (enqueue + poll: `GET /api/provisioning/migrate/:slug` reads status, the Mix task enqueues and then polls through `migration-status`, and the old edge/524 caveat no longer applies to the trigger). Its commit message reads `feat(fouria): …` because two sessions raced; the content is the Lerma change.                                                                                                                                                                                                                                                                                                        |

The alarm-driven contract is the reason the trigger no longer needs a long
request budget: `POST` returns in ~1 s, and the operator polls the read-only
action with a budget they own (`:migrate_wait_timeout_ms`, 30 min).

### Deployment (2026-11-08, one combined rollout)

- Bundle + both runtime images: `unstable-92a21080233dca606cc4e6f91a99eb22397968cf`
  (`fouria:unstable-…` @ `sha256:7d984068b1269b4a2ed32c2cdd6dd7afdb5b48ec68fd9f4f3ff48b0ed2dd7801`
  and `fouria:v1-unstable-…` on `registry.cloudflare.com`, R2
  `fouria-bundles/unstable-…/{fouria.mjs,container-manifest.json}`).
- **Dispatch Worker (staging) redeployed** — required by the contract and easy to
  miss: without it the tenant-facing action exists but `/_control/:slug/migration-status`
  answers `400 Unknown control action` (observed before the deploy), so a Lerma
  `GET` that reaches the control plane can only fail. Deployed from this branch
  (`bun run --cwd workers/fouria-dispatch deploy:staging`, version
  `8da2700f-f886-4d68-a923-8022c947953d`); `workers/fouria-dispatch` has no
  develop-only commits, so the deploy is an additive superset.
- Staging Lerma image: `ghcr.io/0xcab0/lerma:v1-sandbox-92a21080`, built from the
  repo root and shipped by `docker save | ssh … docker load` (no ghcr push);
  compose backup `/opt/lerma/docker-compose.yml.bak-pre-92a21080`; `lerma-pre`
  `healthy`, `/health` `200`.
- One promotion: `Lerma accepted unstable-92a21080…: 2 running instance(s)` →
  `client-sandbox-v1 is running unstable-92a21080…` (both instances rolled with
  the same image + the digest-pinned `SandboxV1` sandbox image).

### Live validation of the async contract (2026-11-08)

Provisioning application key + `X-FourIA-Application: fouria`; the key is never
printed. Read-only calls only — no `POST` on a legacy tenant.

| Drill                                                                                                    | Result                                                                                                                                                                                                                                                                                                                                                 |
| -------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `GET` converted client (`client-sandbox-v1`)                                                             | `200` in 1.59 s: `{ok: true, state: "committed", already: true, record_id: "19753f1a-2554-4f39-b021-73602719af06", reasons: ["already_converted"], v1_record_id: "19753f1a-…", duration_ms: 0}` — the `already_converted` short-circuit, so **no control call at all** (Lerma logs show only the two pointer `GET`s)                                   |
| `POST` converted client (no `dry_run`)                                                                   | `200` in 1.58 s: identical body (`state: committed`, `already: true`, `record_id: 19753f1a-…`, `duration_ms: 0`) — **no enqueue and no control call**                                                                                                                                                                                                  |
| `GET` reserved slug (`client`)                                                                           | `409`, `code: MIGRATION_NOT_ELIGIBLE`, `reasons: ["reserved_client"]`, no control call                                                                                                                                                                                                                                                                 |
| `GET` client with no instances (`tarjetero`)                                                             | `409`, `code: MIGRATION_NOT_ELIGIBLE`, `reasons: ["no_active_instance"]` — no control call                                                                                                                                                                                                                                                             |
| Read-only action, **converted** tenant (deployed Lerma → dispatch → tenant Worker → `v1ConversionState`) | `Containers.migration_status(%{cf_worker_name: "tenant-client-sandbox-v1-dev"})` → `{:ok, %{state: :committed, record_id: "77cadb66-cc11-4561-a6b5-edc1f1a52248", duration_ms: 355}}` (the conversion record, not the pointer)                                                                                                                         |
| Read-only action, **legacy** tenant on the new bundle (`tenant-client-sandbox-v1`)                       | `{:ok, %{state: :absent, record_id: nil, duration_ms: 282}}` — the legacy branch answers `absent` without addressing a v1 DO, and nothing was enqueued                                                                                                                                                                                                 |
| Same action through the dispatch edge (`POST /_control/client-sandbox-v1-dev/migration-status`)          | `200 {ok: true, status: 200, state: "committed", recordId: "77cadb66-…"}`; `…/client-sandbox-v1/migration-status` → `200 {state: "absent"}`                                                                                                                                                                                                            |
| No conversion started by any of the above                                                                | `sandbox-migration/client-sandbox-v1/stable.json` **identical before and after** (`{"recordId":"19753f1a-…","committedAt":"2026-11-08T12:20:51.098Z"}`); `conversion.json` identical (`77cadb66-…`, `createdAt 2026-11-05T12:04:26.260Z`); no new `records/<id>.json`; Lerma logs show **no** `migration queued/committed/failed` line for these calls |
| Park → resume → backup (`client-sandbox-v1-dev`, v1)                                                     | `200` / `200` / `200`: `checkpoint-park` `{backupId: 6b1f7d89-9e5b-459a-87b1-a1068deadc6f, parked: true, sessionsPatched: 0, sessionsDrained: false}` (5.5 s), `resume` `{resumed: true}` (1.6 s), `backup` (4.5 s)                                                                                                                                    |
| Recovery point after the backup                                                                          | `stable.json` advanced to `54eac3a0-d489-441e-ac99-8f65e96af4d9` @ `2026-11-08T12:33:53.317Z`; retention pruned to 3 records (`19753f1a-…`, `6b1f7d89-…`, `54eac3a0-…`)                                                                                                                                                                                |
| `/api/health` on the converted dev tenant                                                                | `200`, `status: running`, `startup.ok` + `readiness.ok` true, `container`/`durableObject`/`r2`/`aiGateway` up, 1.35 s                                                                                                                                                                                                                                  |
| `/api/health` on the production-env legacy tenant                                                        | `503`, `status: stopped`, `components.container: sleeping` — the idle-slept legacy container is **not** woken by a headless probe (by design); `POST /api/warmup` with the health-monitor token answers `302` (host Access app), so a wake needs an operator browser session. Same state the earlier entries recorded as `sleeping`                    |
| Status gate (`GET /api/provisioning/upgrade/client-sandbox-v1`)                                          | both instances `status: running`, `deployed_version = serving_version = unstable-92a21080…`, `container_health.errors: 0`, `healthy` 2 (production-env) / 3 (dev), `rollout_active: false`                                                                                                                                                             |

**What could not be exercised live.** The Worker's `202 {state: "queued"}`
enqueue path (and the `migration-status` reads `queued`/`running`) require an
**eligible legacy tenant**: on staging the only legacy client is the reserved
`client` slug (`409` by contract C2) and `tarjetero` has no instances, so a real
enqueue would have meant converting a tenant — explicitly out of scope. That
path is covered by `src/index.test.ts` (202 queued/running, 200 committed, 502
failed, 503 unreachable), `src/sandbox-v1.test.ts` (queue/arm, alarm commit,
cadence-after-conversion, overlapping alarms, lost-alarm re-arm) and
`workers/fouria-dispatch/test/unit/control.test.ts` (status forwarding + echo).

Two follow-ups worth knowing: (1) a tenant still on a **pre-change bundle**
answers `400 Unknown control action` for `migration-status` (observed on
`tenant-client-dev`) — the status action only exists on instances promoted with
this bundle, which is also the bundle that makes `migrate` durable; (2) the
instance `image_tag` column still shows the image recorded at provisioning
(`unstable-e7c70d3e…`) while `deployed_version`/`serving_version` and the
container's actual `image` are `unstable-92a21080…` — a pre-existing field that
no path in this change writes.

Verification for this bundle: `bun run format:check && bun run lint &&
bun run typecheck && bun run test` (4050 passed, 8 skipped, 199 files), `cctr
apps/fouria/test/business-rules/` (600/600), `scripts/staging-validation.test.ts`
(83/83), `bun run plan:sandbox-v1:check`, `mix check`, `mix license_audit`, the
full Lerma suite (1327 passed, 4 excluded) and `workers/fouria-dispatch`
`--project unit` (129 passed). The dispatch `integration` project fails 11/27
with `asyncLocalStorage.disable() is not implemented` in this environment
(Node 22.23.2 / workerd); it is environmental and predates the change.

## Provisioning a tenant directly on Sandbox v1 ("born v1"), live-validated (2026-11-08)

A tenant no longer has to be converted: a **fresh** instance can be provisioned
straight onto the `SandboxV1` runtime. The only durable evidence of that choice is
the runtime marker Lerma writes before the script deploy, which the Worker reads
(tri-state, fail-closed) instead of the conversion intent.

| Commit     | Slice                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `99ed1d0d` | **Worker contract.** `src/sandbox-migration/runtime-marker.ts` (strict `runtime.json` shape + tri-state read, the marker's `tenant` must match the scope it was read from), `resolveRuntimeIntent` reads the conversion intent **and** the marker in parallel (intent wins; `present` marker → `v1`; unreadable → `unknown`; marker absent **and** `SANDBOX_RUNTIME=v1` → `pending`, so a half-provisioned tenant never boots legacy), `/control` `migrate` on a fresh tenant answers `409 MIGRATION_ALREADY_V1` without addressing the DO (`503 SANDBOX_V1_RUNTIME_UNKNOWN` for an unreadable marker) and `migration-status` answers `absent` without touching the DO. |
| `f73b3fab` | **First recovery point after the boot.** `v1EnsureGateway` dispatches a detached, best-effort `runInitialSnapshot()` after a successful boot (owner run and the adopted-gateway path) so a born-v1 tenant is not loss-bounded by the first idle-cadence tick. Only for a tenant with no recovery point (`verifyRecoveryPoint` = `pointer-absent` and no `conversion.json`); single-flight + a shared `takeOwnedSnapshot()` with the cadence; a failure logs `sandbox.v1.initial_snapshot_failed` and retries on a later boot (10-minute floor) without failing readiness.                                                                                               |
| `35e37628` | **Lerma side.** `instances.sandbox_runtime` (`legacy` default \| `v1`, dev-only and staging-only, immutable after creation), `SANDBOX_RUNTIME` on both the provisioning and re-deploy binding paths (`SANDBOX_V1_CUTOVER=true`, `SandboxV1` DO binding) and the `:sandbox_runtime_marker` provisioning step that writes `sandbox-migration/<canonical-slug>/runtime.json` **create-only, immediately before `:dispatch_deploy`** (idempotent `:existing`, refuses an existing conversion intent, deletes the marker on rollback only when that run created it). The migration preflight reports `already_v1: true` / reason `sandbox_runtime_v1` with no control call.  |
| `82bc961e` | Docs-only (`ENV-VARIABLES` table reformat); deployed as the bundle tag so the promoted artifact matches the pushed tree exactly.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |

### Deployment (2026-11-08)

- Bundle + both images: `unstable-82bc961e95022541bd5abc688293c5e5219b14a9`
  (`fouria:unstable-…` @
  `sha256:371bcb013fb577f9c07ce0f7ccbbf6edacabaaf109dd86c0a104b630a3e6e400` and
  `fouria:v1-unstable-…` @
  `sha256:37e48974e2d0c6d39d370f54986dbd02461e60eb651b0a7d38de238e8cadc303`, R2
  `fouria-bundles/unstable-…/{fouria.mjs,container-manifest.json}`).
- Staging Lerma: `ghcr.io/0xcab0/lerma:v1-sandbox-82bc961e` (same layers as the
  `35e37628` build; the tag commit is docs-only and no file it changes is copied
  into the image), shipped by `docker save | ssh … docker load`, compose backup
  `/opt/lerma/docker-compose.yml.bak-pre-82bc961e`; `lerma-pre` `healthy`, `/health`
  `200`, migration `20261108120000` applied at boot.
- Dispatch (staging) **not** redeployed: it already forwards `migration-status`
  (`POST /_control/client-sandbox-v1-dev/migration-status` answered
  `200 {state: "committed", recordId: "77cadb66-…"}` before the rollout), and
  `workers/fouria-dispatch` has no develop-only divergence.
- One promotion: `Lerma accepted unstable-82bc961e…: 2 running instance(s)` →
  `client-sandbox-v1 is running unstable-82bc961e…` (both instances, same image +
  the digest-pinned `SandboxV1` sandbox image).

### Live born-v1 drill (2026-11-08)

A throwaway staging client (`select-runtime-smoke-82bc`, id 63) was provisioned
through `POST /api/clients/63/instances` with
`{"instance": {"environments": ["dev"], "sandbox_runtime": "v1", "version": "unstable-82bc961e…"}}`
(provisioning application key + `X-FourIA-Application: fouria`; the key is never
printed). `202 {run_id, sandbox_runtime: "v1"}`.

| Check              | Result                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Step order         | `provisioning_step_events`: `… fetch_bundle → telegram_ingress_queue → sandbox_runtime_marker (succeeded) → dispatch_deploy (succeeded) → …` — the marker lands **before** the script deploy                                                                                                                                                                                                                                                                                         |
| Instance row       | `sandbox_runtime: "v1"`, `environment: "dev"`, `status: running`, `deployed_version: unstable-82bc961e…`                                                                                                                                                                                                                                                                                                                                                                             |
| Marker             | `fouria-backup-select-runtime-smoke-82bc-dev-staging/sandbox-migration/select-runtime-smoke-82bc/runtime.json` = `{"mode":"fresh","provisionedAt":"2026-11-08T19:37:44Z","runtime":"v1","tenant":"select-runtime-smoke-82bc","version":1}`; the retried provisioning logged `v1 marker existing` (create-only, `provisionedAt` unchanged)                                                                                                                                            |
| Deployed bindings  | `SANDBOX_RUNTIME=v1`, `SANDBOX_V1_CUTOVER=true`, `SandboxV1` (`durable_object_namespace`, class `SandboxV1`), `WORKER_PLATFORM=staging` (Cloudflare API, dispatch-namespace script settings)                                                                                                                                                                                                                                                                                         |
| Container          | `tenant-select-runtime-smoke-82bc-dev-sandbox-v1-staging` (`scheduling_policy: durable_object`) started an instance in `cdg14` with the digest-pinned v1 image `fouria@sha256:37e48974…`                                                                                                                                                                                                                                                                                             |
| `/api/health`      | `200`, `status: running`, `startup`/`readiness`/`liveness` `success`, `components: container/durableObject/r2/aiGateway all up`, `restoreStatus {hadBackup: false, restored: false}` (fresh onboarding, no `OPENCLAW_RESTORED`)                                                                                                                                                                                                                                                      |
| `migration-status` | `POST https://dev.fouria.io/_control/select-runtime-smoke-82bc-dev/migration-status` → `200 {"state":"absent"}` (no conversion job; the read does not touch the DO)                                                                                                                                                                                                                                                                                                                  |
| `migrate`          | `POST …/migrate` → `409 {"code":"MIGRATION_ALREADY_V1","tenantError":"tenant is provisioned on SandboxV1; there is nothing to convert"}`                                                                                                                                                                                                                                                                                                                                             |
| Initial snapshot   | `sandbox-migration/select-runtime-smoke-82bc/stable.json` → `{"recordId":"c45449dc-ce29-4999-b7e2-471f3f4d8ce1","committedAt":"2026-11-08T20:09:01.264Z"}` — **4 s after readiness (20:08:57)**, no operator action; `records/c45449dc-….json` (`dir /home/openclaw`, `size 7 948 131`, `format tar+zstd/1`, `trigger "backup"`) and the encrypted payload `sandbox-v1/openclaw/c45449dc-….tar.zst.enc` (7 948 180 B) + `.meta.json` (254 B); **no plaintext archive** in the bucket |
| Operator snapshot  | `POST /_control/select-runtime-smoke-82bc-dev/backup` → `200` in 4.0 s; pointer advanced to `0038d1e6-3c86-4c9a-8024-52579fbc84ff` @ `20:10:10.196Z` (two records retained)                                                                                                                                                                                                                                                                                                          |
| Lerma preflight    | `GET /api/provisioning/migrate/select-runtime-smoke-82bc` → `200 {already_v1: true, runtime: "v1", reasons: ["sandbox_runtime_v1","no_legacy_recovery_point","already_converted"], duration_ms: 0}` — no control call                                                                                                                                                                                                                                                                |
| No conversion      | `conversion.json` is absent from the bucket; the tenant was born on v1 and never converted                                                                                                                                                                                                                                                                                                                                                                                           |

**One platform wedge, found and recovered, worth knowing.** The first provision
(19:37) produced a `SandboxV1` container application with **zero instances** even
though the DO reported `running: true`: `containerFetch` timed out on every health
probe and `/control …/backup` hung until the client gave up, so the gateway never
launched and no initial snapshot could happen. The container image itself was
fine (it runs locally and the same tag boots elsewhere). `DELETE
/api/instances/:id` + a re-provision (same client, same marker) recreated the app
and the instance started in `cdg14` within 30 s; the gateway reached readiness
~5 min later. The retry re-used the create-only marker (`v1 marker existing`),
which is also the idempotency proof. Treat a born-v1 instance whose container app
stays at zero instances as an operator action: deprovision and re-provision; the
marker survives, so the tenant stays v1.

Corroboration from an earlier run of the same code: `arua` was provisioned born
v1 at 19:05:59 (`runtime.json` `mode: "fresh"`, `migration-status` `absent`, a
running `tenant-arua-dev-sandbox-v1-staging` instance) and has been committing v1
records on its snapshot cadence (`stable.json` advanced through 19:31 / 19:46 /
20:01, all `trigger "backup"`), so the runtime decision, serving and persistence
also hold outside this drill.

### Cleanup (2026-11-08)

The throwaway client, its instance, the tenant Worker script and both container
applications were removed (`DELETE /api/instances/:id` → `204`, which deprovisions
the script and both container applications; `DELETE /api/clients/63` → `204`,
which decommissions the client — the row stays in the `archived` state, the
platform's normal soft delete, and is no longer listed as active). No
`select-runtime-smoke` script or container application remains. Its R2 bucket was emptied (11 objects deleted, including
the marker and the recovery point) but the bucket itself is still refused by R2
as "not empty" because the initial snapshot left an in-progress multipart upload;
R2 aborts those automatically, after which the empty bucket can be deleted. No
existing tenant was converted and no production instance was touched.

Verification for this bundle: `bun run format:check && bun run lint &&
bun run typecheck && bun run test` (4077 passed, 8 skipped, 199 files), `cctr
apps/fouria/test/business-rules/` (606/606), `scripts/staging-validation.test.ts`
(83/83), `bun run plan:sandbox-v1:check`, `mix check`, `mix license_audit`, the
full Lerma suite (1373 passed, 4 excluded) and `workers/fouria-dispatch
--project unit` (129 passed).

## Wake-lifecycle merge into `v1-sandbox` and the combined deployment (2026-11-08)

Staging had been serving `unstable-c8aae8cd…`, a build of the pushed feature
branch `fix/sandbox-v1-wake-lifecycle`. The born-v1 promotion
(`unstable-82bc961e…`) superseded it, so the two lines were combined on
`v1-sandbox` instead of leaving the staging tenant on the older build: the wake
lifecycle (keep-alive renewal, the provisioned instance type on wake, the
no-empty-recovery-point skip, and the durable automation wake) is part of the
migration track now, not a sibling branch.

### Merge

`git merge --no-ff origin/fix/sandbox-v1-wake-lifecycle` into `v1-sandbox`
(merge commit `cd854394`, first parent `6fac3e42`). The branch had already
merged this track at `8ce6a547` (based on `99ed1d0d`), so its content replayed
cleanly: `apps/fouria/src/sandbox-v1.ts` auto-merged and the two features
compose there — the cadence turn routes through `takeOwnedSnapshot` (born-v1)
while the gateway-owner promise schedules the initial snapshot and the restart
path re-arms the keep-alive (wake lifecycle).

Two conflicts, both in docs, resolved preserving both sides:

| File                    | Resolution                                                                                                                                                                                                          |
| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `apps/fouria/AGENTS.md` | kept the wake-lifecycle recent-changes bullet **and** the newer born-v1 bullet (the one that also documents the first recovery point taken after the first boot)                                                    |
| `docs/ENV-VARIABLES.md` | kept the reformatted table, merged the wake-corrected `SANDBOX_RUNTIME` text (keep-alive renewal, no snapshot wake, production 6h cap), the production `3h` cadence cap and added the `CONTAINER_INSTANCE_TYPE` row |

No code conflict needed a judgement call, so no resolution was guessed.

### Deployment

- Bundle + both runtime images: `unstable-cd8543948638f27e98081fb47b3312d5dad15acd`
  (`fouria:unstable-…` @ `sha256:371bcb01…`, `fouria:v1-unstable-…` @
  `sha256:37e48974…`, R2 `fouria-bundles/unstable-cd854394…/{fouria.mjs,container-manifest.json}`).
- The published bundle carries **both** halves (grep of the R2 `fouria.mjs`):
  `debug/lifecycle`, `keepalive-pending`, `wokenFor`, `CONTAINER_INSTANCE_TYPE`
  (wake lifecycle) and `runtime-marker`, `MIGRATION_ALREADY_V1` (born v1).
- Staging Lerma: `ghcr.io/0xcab0/lerma:v1-sandbox-cd854394`, built from the repo
  root and shipped by `docker save | ssh … docker load`, compose backup
  `/opt/lerma/docker-compose.yml.bak-pre-cd854394`, retargeted from
  `v1-sandbox-82bc961e`; `lerma-pre` `healthy`, in-container `/health` `200`.
- One promotion: `Lerma accepted unstable-cd854394…: 2 running instance(s)` →
  `client-sandbox-v1 is running unstable-cd854394…`.

### Live verification (2026-11-08)

| Check                       | Result                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Status gate                 | both instances `deployed_version = serving_version = unstable-cd854394…`, `status: running`, `container_health.errors: 0`, `healthy: 2` / `3`, `rollout_active: false`                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `checkpoint-park` (dev)     | `200` `parked: true`, `backupId 05aa5c01-b8dd-42f1-bbf9-4946b7f813d0`, `sessionsPatched: 0` (no live sessions)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `resume` (dev)              | `200` `resumed: true`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `backup` after resume       | `200` (`ok: true`) — proves the park guard was released (a parked tenant answers `409 TENANT_PARKED`)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| Recovery point (R2)         | `sandbox-migration/client-sandbox-v1/stable.json` → `recordId 19fa77db-87f1-47de-9771-04e1cbcad4cb`, `committedAt 2026-11-08T20:36:17Z` (12 s before the check)                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `migration-status` (Lerma)  | `GET /api/provisioning/migrate/client-sandbox-v1` → `200 {state: "committed", already: true, record_id: 19fa77db…}` (read-only; the non-dev instance still reports `sandbox_runtime: legacy`)                                                                                                                                                                                                                                                                                                                                                                                                     |
| `/api/health` (both hosts)  | `200` on `staging-client-sandbox-v1.fouria.io` and `staging-client-sandbox-v1-dev.fouria.io`; `status: running`, startup/readiness/liveness `success`, `container/durableObject/r2/aiGateway` all up                                                                                                                                                                                                                                                                                                                                                                                              |
| Drill client decommissioned | no `select-runtime-smoke` script in `fouria-tenants-staging` (5 scripts: `tenant-client`, `tenant-client-dev`, `tenant-client-sandbox-v1`, `tenant-client-sandbox-v1-dev`, `tenant-arua-dev`) and no matching container application (9 total)                                                                                                                                                                                                                                                                                                                                                     |
| Drill bucket                | `fouria-backup-select-runtime-smoke-82bc-dev-staging` still exists and is empty, but `r2 bucket delete` is refused again (`10008 not empty`, the in-progress multipart upload from the initial snapshot). The abort needs the S3 API, which this machine cannot reach — `*.r2.cloudflarestorage.com` resolves to real Cloudflare IPs but the egress path presents `core1.netops.test`, so `curl --aws-sigv4` fails TLS verification before any request. R2 auto-aborts the upload; the empty bucket is deletable afterwards. Only `tenant-arua-dev` (a separate born-v1 tenant) was left running. |

The deployed artifact is `unstable-cd854394…` (the merge commit). The docs
commits on top of it (`6852699a` and this note) change no file that is copied
into the bundle or either image, so `unstable-cd854394…` still describes exactly
what staging is serving.

No tenant was converted by this deployment: the merge and the promotion changed
the deployed image only. `client-sandbox-v1` keeps `sandbox_runtime: legacy`
(never migrated, per the reserved-path rule) and `client-sandbox-v1-dev` keeps its
existing v1 recovery point; no production instance was touched.

Verification for this bundle: `bun run format:check && bun run lint &&
bun run typecheck && bun run test` (4101 passed, 8 skipped, 201 files, no type
errors), `cctr apps/fouria/test/business-rules/` (614/614), `scripts/staging-validation.test.ts`
(83/83), `bun run plan:sandbox-v1:check` (166 features, 14 milestones), `mix
check`, `mix license_audit` (62 deps), the full Lerma suite (1373 passed, 4
excluded) and `workers/fouria-dispatch --project unit` (129 passed). The live
drill suite self-skips without `E2E_V1_LIVE=1` (3/3 skipped).

## Rollback coverage + upgrade-pipeline repair (2026-11-09)

The OCM-4a incident's rollback fix was one-sided. `Instances.upgrade_instance/3`
(the promotion lane) passed `preserve_do_namespaces: true`, but the
dashboard/API pipeline (`Instances.update_instance_version/3` →
`TenantProvisioner.upgrade_instance/3` → `:dispatch_deploy`) called
`Containers.deploy_script_and_container/1` **without** it, so a failed container
binding could still delete a live tenant's routing script exactly as in the
incident. Two commits closed it:

| Commit     | Change                                                                                                                                                                                                                                                                                                                                             |
| ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `5e3e5618` | `TenantProvisioner`'s `:dispatch_deploy` passes `preserve_do_namespaces: upgrade?(meta)`; a fresh provision keeps the destructive rollback so a retry starts clean.                                                                                                                                                                                |
| `d57bec6d` | The shared `Instances.deploy_script_and_container/6` takes the flag explicitly (fresh `create_user_worker_instance/3` = false, upgrade / AI-gateway reassignment = true), and the upgrade test asserts the full binding set a rebuilt script needs (gateway token, derived CDP secret, AI Gateway/API token, backup key, Telegram queue, runtime). |

Deployment: `unstable-d57bec6d…` published; branch Lerma
`v1-sandbox-d57bec6d` deployed (compose backup
`/opt/lerma/docker-compose.yml.bak-pre-d57bec6d`); one promotion of
`client-sandbox-v1` converged.

| Check                       | Result                                                                                                                                                             |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Both tenant scripts         | identical etag `b0a58070…`, modified 12:58Z / 13:02Z                                                                                                               |
| Bindings                    | 33 (production-env) / 34 (dev); `SANDBOX_RUNTIME=legacy`, `SANDBOX_V1_CUTOVER=false`                                                                               |
| Health                      | `HealthMonitor.check_instance_now/1` → `:ok` on both                                                                                                               |
| v1 park/resume/backup (dev) | `200` / `200` / `200`; recovery point advanced to `ed7f0db5-…` at 13:05Z                                                                                           |
| Born-v1 `tenant-arua-dev`   | upgraded to `unstable-d57bec6d…`; `deployed_version = serving_version`, 35 bindings, `SANDBOX_RUNTIME=v1`, marker intact, recovery point `8c2729a2-…`, 3/3 healthy |
| Staging Lerma               | `v1-sandbox-d57bec6d`, healthy                                                                                                                                     |

Operational follow-up: the staging provisioning key was rotated (old key
`401`, new key `200`; `SECRETS.yaml` and the GitHub
`STAGING_PROVISIONING_API_KEY` secret updated). The born-v1 `tenant-arua-dev`
was moved off the unreachable `f1590509` merge and onto this build.

## Gateway proxy attribution fix, validated live (2026-11-09)

Every dev instance on the v1 runtime served
`{"error":{"message":"Proxy client attribution is required…","type":"proxy_attribution_required"}}`
on the root HTML/WS proxy. Root cause: OpenClaw attributes proxy-shaped traffic
(`Forwarded` / `X-Forwarded-*` / `X-Real-IP`) against `gateway.trustedProxies`
_before_ Gateway auth and rejects a source it cannot trust. The legacy runtime
reaches the container from the sandbox bridge (`10.1.0.x`, healed into
`trustedProxies` by moltlazy), but the v1 transport forwards through
`ctx.container.getTcpPort(port)` from the `SandboxV1` Durable Object, whose
ingress address is platform-owned and is not a range a tenant config can list.

Fix (`eddef5e0`): `SandboxV1.fetch()` probes attribution once per gateway
incarnation (`GET /` with the rebuilt forwarded headers) and, when the Gateway
refuses it, strips every proxy-shaped header so OpenClaw attributes the request
as **direct-remote** (always accepted, same token/device auth). The verdict is
cached per DO lifetime, re-probed after `v1KillGateway` / `v1Destroy` /
`startGatewayOwner`, and guarded by a generation counter.

Publishing for the live drill reused the running container image (unchanged by
this commit): the `unstable-d57bec6d…` legacy + `v1-unstable-d57bec6d…` manifests
were copied to `unstable-eddef5e0…` / `v1-unstable-eddef5e0…` in
`registry.cloudflare.com/35afea16440634aa2350331d2a736eec/fouria` (identical
digests), and the new Worker bundle + `container-manifest.json` were published to
`fouria-bundles/unstable-eddef5e0…/`. Promotion: `POST
/api/provisioning/upgrade/arua {"version":"unstable-eddef5e0…"}`.

| Check                           | Result                                                                                                                                                                                                                                                                   |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Promotion convergence           | `status: running`, `deployed_version = serving_version = unstable-eddef5e0…`, `container_health` 3 healthy / 0 failed / 0 errors, `rollout_active: false`                                                                                                                |
| Container                       | not replaced (same image digest): instance `0899aee8…` `running` since 2026-11-08 — only the script changed                                                                                                                                                              |
| `/api/health` (service token)   | `200` `status: running`, `degraded: false`, startup/readiness/liveness `success`, gateway pid 4 / 1.39 GiB / 23 threads                                                                                                                                                  |
| Live gateway attribution (new)  | tenant log `Gateway refused proxy attribution; forwarding without forwarded client headers` — the DO probe reproduced the rejection against the real Gateway                                                                                                             |
| Proxied request through the fix | `GET /api/status/attribution-live-{1,2,3}` (an allowed dispatch health path that falls through to the catch-all proxy) → `[PROXY] Handling request` + `[HTTP] Response status: 404` (the Gateway's answer for an unknown path), **not** `403 proxy_attribution_required` |
| Post-cleanup                    | tenant observability settings restored (`persist: false`, 10 % logs), temporary Access service token + its two app policies deleted, public host back to `302` (Access)                                                                                                  |

Validation tooling note: staging tenants are ROOT-only at the dispatch gate, so
the public proxy path cannot be exercised with a service token. The reachable
`/api/status/*` prefix is allowlisted as a health path and falls through to the
gateway proxy, which made the end-to-end request observable; tenant Worker logs
were read by briefly enabling `persist` (they normally go only to Lerma's OTLP
ingest) and the setting was restored immediately afterwards.

## Preflight bundle verification (2026-11-09)

The known-gap "Preflight asymmetry" is closed. `cfd2dda8` adds
`DispatchNamespace.get_script_content/2` and makes
`Instances.at_requested_version?/2` verify that the requested bundle's main
module is embedded in the deployed script (the content endpoint returns the whole
uploaded script; the parsed main module is a verbatim substring). Both the
promotion candidate filter and the error-retry preflight use it, so a failed
script upload (image at target, bundle older) is retried; an unreadable
bundle/content probe falls back to the image so a transient failure cannot loop.
`685d686a` makes the endpoint report the number of instances actually marked
upgrading.

Deployment: branch Lerma `v1-sandbox-685d686a` (compose backup
`/opt/lerma/docker-compose.yml.bak-pre-685d686a`), healthy. No Worker promotion
was needed; staging keeps `unstable-d57bec6d…`.

| Check                        | Result                                                                                                                                                                                                                    |
| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Deployed script content      | 10,875,960 bytes; contains the parsed main module (2,855,594 bytes) and the `d57bec6d` commit marker                                                                                                                      |
| Re-promotion preflight       | `POST /api/provisioning/upgrade/client-sandbox-v1` (version `unstable-d57bec6d…`) → `202 {instance_count: 0}`; both instances stayed `running` at d57bec6d; Lerma fetched the bundle once per instance and did not deploy |
| Negative path (bundle older) | Suite: `at_requested_version?/2` image/bundle matrix + controller retry test; live failure injection remains the next step                                                                                                |

## Born-v1 first-provision self-heal (2026-11-11)

`51df6b0a` closes the zero-instance first-provision failure recorded above
(2026-11-08: the `SandboxV1` application existed with **zero instances**, the DO
reported running, `containerFetch` timed out and the gateway never booted; the
only recovery was a manual deprovision + re-provision).

- `Containers.recreate_v1_application/1` deletes and recreates the v1
  application against the same DO namespace (bounded create retry). The deployed
  script and its Durable Object state are untouched.
- `TenantProvisioner.verify_health/3` takes that action only for a **fresh**
  provision (`reconcile_v1`) with **no committed recovery point** whose v1
  application reports no instances (or is missing), then retries the probe once.
  An upgrade never opts in and a tenant with a pointer is never reconciled.

Deployment: branch Lerma `v1-sandbox-51df6b0a` (compose backup
`/opt/lerma/docker-compose.yml.bak-pre-51df6b0a`), healthy; no migrations. All
five staging instances remain running (`tenant-arua-dev` on
`unstable-eddef5e0…`, both `client-sandbox-v1` tenants on
`unstable-d57bec6d…`, `tenant-client{,-dev}` on `unstable-73e820e6…`).

| Check            | Result                                                                                                                                                                               |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Module loaded    | `function_exported?(Containers, :recreate_v1_application, 1)` → `true`; `SandboxV1Status` and `TenantProvisioner` loaded                                                             |
| Tests            | Targeted `containers_test` + `tenant_provisioner_pipeline_test`: 148 passed; full Lerma suite: 1404 passed / 4 excluded; `mix format --check-formatted` + `mix credo --strict` clean |
| Live wedge drill | Not run — it needs a disposable born-v1 tenant on this image. `tenant-arua-dev` is ineligible: it has a recovery point, and the reconcile deliberately refuses those                 |

## Born-v1 self-heal: live drill attempt and the `active` fix (2026-11-11)

Live drill attempt on a disposable born-v1 tenant (`bornv1-heal-7fb503`, client
64, `dev`, `unstable-eddef5e0…`; deprovisioned and deleted afterwards). Two
findings, one fixed and one still open.

**1. Fixed (`b0a22c02`) — a `durable_object` application reports `active`, not
`healthy`.** The live health payload for a v1 application is
`{"instances": {"active": 1, "starting": 0}}`; there is no `healthy` key.
`Containers.fetch_application_health/1` read `healthy`, so a **live** v1 tenant
looked instance-less: the promotion preflight would re-deploy it on every
promotion, and the new born-v1 reconcile would have recreated a running
application. The v1 summary now maps `active` into `healthy` (falling back to
`healthy`). Live proof: `GET /api/provisioning/upgrade/arua` reports
`container_health {runtime: v1, healthy: 1, rollout_active: false}`, and
re-promoting `arua` at its current version returns `202 {instance_count: 0}` —
before the fix the preflight retried it.

**2. The zero-instance wedge could not be reproduced by deleting the app
record.** The v1 application was created at `00:32:25` and deleted at
`00:32:29` (API `200`, absent from the account list). The first health probe
still started a container, the gateway booted, and the initial snapshot
committed at `00:35:02`. The app record is therefore not the only thing that
gates a placement (or its deletion is not synchronous), so recreating only the
application may not be sufficient for the original transient — the documented
operator workaround also re-deployed the script. The async health task was also
lost to an unrelated Lerma BEAM restart at `00:36:25` (the container did not
restart; `Startup health` re-ran), so the reconcile path was never reached. The
drill is **inconclusive**, not a failure of the path.

Cleanup: `DELETE /api/instances/2537773c-…` → `204`, `DELETE /api/clients/64` →
`204`, the tenant script and both container applications are absent, and the R2
bucket was emptied (5 objects) and deleted.

Next step: fault-inject a placement failure (or make the provisioner re-deploy
the script when the application stays empty), since deleting the app record
alone does not reproduce the wedge.

## Develop integration sync, production-surface removal and the local gate (2026-11-12)

Merged `origin/develop` (`e62f5a37`, Agent Studio #572) into `v1-sandbox` at
`305d489e`. Only the two business-rules corpora conflicted; both were resolved
additively (the activity/alarm/boot sections and the Agent Studio sections, each
with its own count trailer). Code and docs auto-merged (`agentStudioRoutes` +
`activityRoutes`, the Studio API client, and the Agent Studio sections in
`WORKER-DEVELOPMENT.md` / `TESTING.md`).

**Decision.** ADR-SV1-010 records that the track integrates into `develop` as one
reviewed merge commit and that integration is not a production cutover. The
production `wrangler.jsonc` environment no longer declares the
`fouria-sandbox-v1` application or the `SandboxV1` binding (`019a22f1`); the
class and the append-only `v7` migration remain. `validate-runtime-image-pins`
now requires the v1 surface in local/staging and fails if production declares it
(`803831fa`).

**Local gate.** On the merged tree (Elixir files identical to `305d489e`):

| Check                                       | Result                                             |
| ------------------------------------------- | -------------------------------------------------- |
| `bun run format:check`, `lint`, `typecheck` | pass                                               |
| `bun run test`                              | 4254 passed, 8 skipped (209 files), no type errors |
| `mix check`, `mix license_audit`            | pass (3630 mods/funs, 62 deps)                     |
| `mix workspace.run -t test`                 | 1417 passed, 4 excluded                            |
| `bun run plan:sandbox-v1:check`             | pass (166 features / 14 milestones)                |

`mix precommit` remains blocked by 20 **pre-existing** compiler warnings under
`--warnings-as-errors` across Lerma/Fouria; none are introduced by this merge,
and the mandatory gates above are green.

**CI.** On `803831fa` the PR checks (`unit`, `test`, `quality`, `build-and-push`,
`changes`, `validate-secrets`, `Validation lane tests`) passed and PR #565 is
`MERGEABLE`/`CLEAN`. The manual `Test` workflow (`workflow_dispatch`,
`run_e2e=true`, `full_tests=true`, `production`) was started as run
`43199464508`; its observation is appended when it completes.

## Staging validation after the `cec359fa` promotion (2026-11-12)

- **Promotion:** `client-sandbox-v1` converged on `unstable-cec359fa…` at
  `10:55:30Z` (2 instances); both tenant scripts share etag `b7893a73…`
  (`tenant-client-sandbox-v1` modified 10:52:03, `…-dev` 10:55:19). The legacy
  production application serves `fouria:unstable-cec359fa…`; the dev instance
  serves through the v1 application (v1 `active: 1`).
- **v1 snapshot (PER-2):** `POST /_control/client-sandbox-v1-dev/backup` → `200`;
  the stable pointer advanced `7beb1684…` (`10:55:08Z`) → `3b9030fa…`
  (`10:56:40Z`).
- **SEC-3 park/resume:** `checkpoint-park` → `{parked: true, backupId:
  7e4b11c1…}`; a `backup` while parked → `409 TENANT_PARKED`; `resume` →
  `{resumed: true}`; a post-resume `backup` → `200` and the pointer advanced to
  `7b6f95d5…` (`10:57:40Z`). The cold boot served from the committed record.
- **Tenant isolation:** `migration-status` for `client-sandbox-v1-dev` →
  `200 {state: "committed", recordId: "77cadb66…"}`; for the
  production-environment `client-sandbox-v1` → `200 {state: "absent"}` (it is
  still legacy; its bucket contains only `backups/*.sqsh`). No Access token was
  used: the checks went through the dispatch control plane.
- **BIL-3 finding (cost attribution):** the AE `fourai_cost_metrics` rows from
  the v1 snapshot/park are written with `blob2 = "default"`, not
  `client-sandbox-v1`. The tenant script carries
  `WORKER_URL=https://staging-client-sandbox-v1-dev.fouria.io` but Lerma binds
  no `CUSTOMER_SLUG`/`WORKER_NAME`. `tenantSlugFromEnv` already falls back to
  `WORKER_URL`; `deriveCustomerSlug` (used by `getCostContext`) did not, so it
  returned `default`. **Fixed in this PR:** Lerma now binds the canonical base
  slug as `CUSTOMER_SLUG` on both fresh provision and upgrade; the cost resolver
  sanitizes the explicit binding with the shared slug allowlist, fails closed for
  a non-empty invalid value (never adopting `WORKER_NAME`/`WORKER_URL`), and
  never uses a URL as the identity; `isValidSummaryCustomer` accepts exactly the
  emitted domain so rows are never write-only. The next promotion re-deploys the
  scripts with the binding; the corrected AE rows are the live confirmation.
- **BIL-3 adversarial review (independent session):** the fix was attacked
  before merge, and it found a real cross-tenant hole in the _pre-existing_
  guard: `otel-ingest.ts` skipped the tenant check whenever the expected
  identity was the `default` sentinel (`expected !== 'default' && …`), while the
  billable row's customer came from the container-supplied `customer.slug`. An
  unidentified Worker could therefore mint billable rows for another tenant and
  choose which `billing-freeze/<slug>` gate applied. The guard now falls back to
  the routing resolver (`WORKER_NAME`/`WORKER_URL`) and answers
  `503 TENANT_UNRESOLVED` on a billable instance with no identity. The same
  review closed the consistency gaps: Lerma reserves `default` and the
  `tenant-` dispatch prefix as client slugs, `canonical_tenant_slug/1` strips
  exactly one affix (so `tenant-tenant-foo` no longer collapses to `foo`), the
  shared sanitizer rejects padded values instead of trimming them, the cost
  summary and emitter share one slug domain, out-of-range `from`/`to` values are
  clamped instead of throwing, and a loopback `WORKER_URL` is a local
  (non-billable) instance. Covered by `sandbox-otel-bridge.test.ts`,
  `metrics.test.ts`, `cost-summary.test.ts`, `dev-mode.test.ts`,
  `sandbox-tenant.test.ts`, and the Lerma client/naming tests.
- **BIL-3 live confirmation (2026-11-16).** Staging Lerma was redeployed from the
  branch image `ghcr.io/0xcab0/lerma:v1-sandbox-69e2d9cc` (compose backup
  `/opt/lerma/docker-compose.yml.bak-pre-69e2d9cc`) and both
  `client-sandbox-v1` scripts were re-uploaded at their existing bundle
  (`unstable-cec359fa…`) via `Instances.upgrade_instance/3`. Both scripts now
  carry `CUSTOMER_SLUG` as a `plain_text` binding with the value
  `client-sandbox-v1` (35 bindings on the dev script, 34 on the production-env
  script; previously absent). The dev instance is `running` and the legacy
  production-env instance is `sleeping`; `last_deployed_at` advanced to
  `14:12:14Z` / `14:14:41Z`. Analytics Engine
  (`fourai_cost_metrics_staging`) rows emitted after the flip are attributed to
  `blob2 = "client-sandbox-v1"` for `health-check`, `gateway-memory-sample` and
  `gateway-resources`; the same row types were `blob2 = "default"` before the
  redeploy. The remaining `default` rows belong to the other staging tenants
  (`client`, `arua`), which have not been upgraded yet and pick up the binding
  on their next upgrade.
- **Cloud E2E harness repaired (2026-11-13).** A fresh `wrangler deploy` failed
  for hours with "Durable Object-managed Container application settings were not
  fully applied" after all five retries. Root cause: the top-level (local/E2E)
  `wrangler.jsonc` still declares the `SandboxV1` application as the shared
  `fouria-sandbox-v1`, which belongs to the production Worker, so every fresh E2E
  run tried to adopt it. The per-run rename is restored
  (`<worker>-sandbox-v1`), and teardown now validates the teardown token (a stale
  `E2E_CLOUDFLARE_API_TOKEN` answered `9109` and made every container-app delete
  silently no-op; three leaked apps were still billing and were deleted
  manually). With the per-group transcripts finally uploaded, the four legacy
  failures were triaged and fixed: `h_cli_knowledge` (the roster read raced the
  plugin reload → bounded retry in `get-agent-id`), `y_production_profile` (no
  scheduled snapshot because `BACKUP_CHANGE_WATCH` is off in E2E → on-demand
  `POST /debug/backup/sync`), `zz_gateway_hard_restart` (`/status/polling`'s RPC
  fast path omits `processId` → read the authoritative `/api/health` snapshot,
  plus a post-restore boot wait; the snapshot now exposes the `/proc` pid when
  the tracker misses a restore-relaunched gateway), and
  `zzz_software_provisioning` (missing `CUSTOMER_SLUG` → fixture binding, now
  9/9). Re-runs of every previously failing suite are green.
- **Not run (Access-free limitation):** a v1 `freeze` would leave the dev
  tenant stopped (`unfreeze` is a no-op and there is no Access-free wake); the
  snapshot-before-destroy path is corpus-covered, and the safe SEC-3
  park/resume path above proves the cold boot from the committed record.
  Tenant-HTTP drills (channels/RPC, browser) remain Cloudflare-Access-blocked.
