F FourIA staging 2026.10.29 GitHub ↗

Sandbox v1 — staging observation record

On this page

Append-only record of the step 5 observation for each milestone on the v1-sandbox track. The target is Lerma client client-sandbox-v1 (id 20) and its two instances (tenant-client-sandbox-v1-dev, tenant-client-sandbox-v1) in the Paso4 staging account, dispatch namespace fouria-tenants-staging.

Nothing here is a production observation: no production tenant has been converted, and the migration authority (WORKER_PLATFORM === "staging") is not present in production.

How to reproduce

# Instance state + health (staging Lerma)
#   docker exec -i lerma-pre sh -s <<'EOF'
#   ... Application.get_env(:lerma, :provisioning)[:dispatch_namespace] ...
/app/bin/lerma rpc 'Lerma.Instances.list_all_for_client(20)'

# Deployed script identity, bindings and bundle content (Cloudflare API)
#   GET /accounts/<acct>/workers/dispatch/namespaces/fouria-tenants-staging/scripts
#   GET .../scripts/<name>/settings
#   GET .../scripts/<name>/content        # grep for SANDBOX_V1_* markers
#   GET /accounts/<acct>/containers/applications

# Worker telemetry (Lerma ingest of the Observability destinations)
#   from(e in "observability_events", where: e.script_name in ^names, ...) |> Repo.all()

S1 — Authority, identity and platform gate (unstable-a90015f5...)

  • Both instances report the version; Lerma health :ok.
  • Health events for production: down (503, 7.5 s) at 20:00:52 → up at 20:03:37 during the rollout window.
  • No migration opportunity was exercised (there is no conversion yet).

S2 — Conversion intent, lease and DO conversion (unstable-486b2938...)

  • Both instances report the version, status sleeping, health :ok.
  • Dockerfile.v1 includes squashfs-tools.
  • No intent exists, so both tenants remain legacy. No conversion was triggered.

S3 — Serving parity: proxy, health and admin RPC (unstable-41be4409...)

Verified

CheckResult
Tenant scripts modified at the promotion times✅ tenant-client-sandbox-v1 08:20:47Z, …-dev 08:22:41Z
Bundle contains the S3 code✅ SANDBOX_V1_RUNTIME_UNKNOWN, SANDBOX_V1_CONTROL_UNSUPPORTED, SANDBOX_V1_SLEEPING, sandbox.runtime.selected, __fouriaSandboxV1, v1EnsureGateway
Same bundle on both instances✅ identical script etag
Container image✅ fouria:unstable-41be4409… on both legacy tenant apps
Container health✅ 3 healthy / 0 active per app (idle-slept)
Lerma instance state✅ status sleeping, deployed_version = unstable-41be4409…, HealthMonitor.check_instance_now/1 → :ok
Worker telemetry after the deploy✅ spans 200 (80 / 67), no error outcome, no exception class after 2026-11-04T20:00Z
503 spansall 13 + 13 from 2026-11-04 20:02–20:03Z (first cold boot), none after the S3 deploy
Latest activity✅ 2026-11-05T08:57Z
Hosted /healthz✅ 302 (Cloudflare Access), no 5xx

Blocking finding: the v1 path is deployed but cannot be selected

The live tenant script bindings (read from the dispatch-namespace settings) are:

ACTIVITY, ANALYTICS, BACKUP_BUCKET, BACKUP_BUCKET_NAME, BACKUP_CHANGE_DETECTION, BACKUP_CHANGE_WATCH, BACKUP_ENCRYPTION_KEY, BACKUP_ENCRYPTION_REQUIRED, BROWSER, CDP_SECRET, CF_ACCESS_*, CF_AI_GATEWAY_*, CLOUDFLARE_*, DEBUG_ROUTES, LEADS_BUCKET, MOLTBOT_GATEWAY_TOKEN, OPENCODE_API_KEY, PASO4_TELEGRAM_GROUP_ID, R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY, Sandbox, TELEGRAM_INGRESS, TELEGRAM_INGRESS_HOST, TELEGRAM_INGRESS_QUEUE, WORKER_URL.

Missing for the v1 track:

  1. WORKER_PLATFORM (and SANDBOX_V1_CUTOVER). resolveRuntimeIntent requires the exact trusted value staging; without the binding it always returns legacy. The S3 runtime selection therefore never activates.
  2. SandboxV1 Durable Object binding (and its class migration). The bundle exports SandboxV1, but with no namespace binding acquireSandboxV1() would throw SandboxV1 binding is not configured if selection ever reached it.
  3. A v1 container application (scheduling_policy: "durable_object", v1 image). The account has only the per-tenant legacy applications (tenant-<slug>-sandbox-staging, image fouria:unstable-…). The fouria-sandbox-v1-staging application declared in wrangler.jsonc is created by a standalone wrangler deploy, not by Lerma’s dispatch-namespace upload, so it does not exist for tenants.

Root cause

Tenant scripts are uploaded by Lerma into fouria-tenants-staging, not by wrangler deploy with this repo’s wrangler.jsonc environment. The two Lerma paths differ:

  • Lerma.Provisioning.TenantProvisioner.required_var_bindings/2 adds WORKER_PLATFORM / SANDBOX_V1_CUTOVER (S1 change).
  • Lerma.Instances.instance_bindings/2, used by the upgrade/promotion redeploy, does not add them (and neither path adds SandboxV1). The promotion therefore re-uploads the tenant with the legacy binding set.
  • The deployed staging Lerma runs develop, which does not contain the S1 provisioning change at all, so instances created before it also lack the bindings.

Verdict

  • S3 is deployed (bundle identity proven) and inert (runtime selection cannot choose v1; the v1 runtime is not provisioned). This is a truthful “deployed, not activatable” state, not a runtime failure: legacy serving is unchanged, with no post-deploy errors.
  • Activation belongs to OCM-4 (provisioning, rollout, promotion) with the conversion orchestration in S5 / PER-3:
    1. add WORKER_PLATFORM, SANDBOX_V1_CUTOVER and the SandboxV1 DO binding (plus its class migration) to Instances.instance_bindings/2;
    2. provision/declare a per-tenant v1 container application with the v1 image and scheduling_policy: "durable_object" (naming + rollout policy);
    3. redeploy the staging Lerma with the v1-branch changes so an upgrade carries the bindings;
    4. then run the conversion job and repeat this observation against the v1 runtime (HTTP/WS proxy, passive health, single owner).

Next observation

Repeat S3 once the binding/application gap above is closed: the same API reads must show SandboxV1 in the tenant bindings and a per-tenant v1 container application, and the observability stream must show v1-path traffic (sandbox.runtime.selected: v1) with no proxy/WS errors.

OCM-4a — staging provisioning of the v1 surface (code landed, not yet deployed)

Closing the gap above is provisioning work, implemented on v1-sandbox:

  • Lerma.Cloudflare.SandboxV1 is the single selection point: enabled?/0 is true only for cf_environment == "staging"; image_enabled?/1 additionally requires an unstable-* validation image (the publisher pushes the matching v1-unstable-* tag); v1_image/1, binding/0, container_config/3 and drop_binding/1 keep the binding, the class migration and the container application keyed on the same predicate.
  • Provisioning and upgrade bindings both add WORKER_PLATFORM and SANDBOX_V1_CUTOVER; the staging platform also gets the SandboxV1 DO binding. The upgrade path (Instances.instance_bindings/2) previously lacked the platform bindings entirely — that was the root cause above.
  • DispatchNamespace derives the new_sqlite_classes set from the declared container entries, so SandboxV1 is migrated only when it is also bound (production stays legacy-only). namespace_id_for_class/3 resolves the per-class namespace.
  • Containers.deploy_script_and_container/1 declares both container entries on staging/unstable images, prepares the v1 image artifact, and creates the per-tenant …-sandbox-v1-<platform> application with scheduling_policy: "durable_object". It drops the SandboxV1 binding when the image has no v1 tag, so an upload can never bind an unmigrated class.
  • Deprovisioning removes both applications.

Verified locally (unit + full Lerma suite); the staging effect is not yet observed because the running staging Lerma predates this change. The next step is to deploy the branch Lerma to staging (:latest-pre lane), promote a tenant, and repeat the checks above. Note the manual fallback: a promotion from develop Lerma would re-upload the legacy binding set, so the branch Lerma must be the one driving the upgrade.

OCM-4a live validation on staging (2026-11-05)

The branch Lerma was temporarily deployed to the staging host under a unique local image tag (transferred with docker save/load, compose repointed, then restored to :latest-pre afterwards), a fresh validation build was published (unstable-8ce99552…), and both tenants were upgraded through that Lerma.

Verified live

CheckResult
Worker bundleboth tenant scripts at unstable-8ce99552…, identical script etag
Platform bindingsWORKER_PLATFORM = staging and SANDBOX_V1_CUTOVER = false on both
v1 DO bindingSandboxV1 bound (prod namespace c1b83745…, dev f50564f8…)
v1 container appstenant-client-sandbox-v1{,-dev}-sandbox-v1-staging, scheduling_policy: "durable_object", empty application config (image comes from ctx.container.images)
Named imagescript metadata carries images.sandbox as a digest-pinned Cloudflare Registry reference (…/fouria@sha256:35b931…)
Legacy appsboth on fouria:unstable-8ce99552…, 3/3 healthy, no errors
Instance stateboth running, HealthMonitor.check_instance_now/1 → :ok; /healthz → 302 (Access)
Telemetry after recoveryno non-ok spans or exception classes; latest activity current
Runtime selectionstill legacy — no conversion intent exists, so the v1 surface is inert as designed

Incident during the first validation attempt (both scripts deleted)

The first two upgrade attempts used an earlier branch image whose v1 container-application body still contained configuration.image / instance_type (a durable_object application rejects them). The API rejected the app creation after the script upload, and the atomic-deploy rollback deleted the just-uploaded script — for a live tenant that also removed its routing, leaving both instances in error (the containers themselves stayed healthy). Recovery was a re-run of upgrade_instance once the code was fixed.

Two code changes came out of it:

  1. preserve_do_namespaces now covers the script. An upgrade rollback must never delete a live tenant’s script (it is what routes the tenant); the newly uploaded script and its DO namespaces are kept and the failure is reported for a retry. Fresh provisions still clean up.
  2. The upgrade binding set is self-sufficient. Instances.instance_bindings/2 previously relied on keep_bindings for the per-instance/platform bindings, so a rebuild after a deleted script left the tenant without the gateway token, CDP secret, AI Gateway credential, Access AUD, agent roster, Telegram queue, and other vars. It now re-asserts the full set: MOLTBOT_GATEWAY_TOKEN (stored), a deterministically derived CDP_SECRET, AI Gateway/API token, OPENCODE_API_KEY (dev), the per-instance Access AUD (resolved by the tenant’s own hostname), agent/scope flags, account/team vars, Telegram ingress queue/host, change-detection vars and DEBUG_ROUTES. Verified after recovery: 32 bindings on dev, 31 on production (only DEBUG_ROUTES is dev-only).

Operational notes

  • The staging Lerma is back on :latest-pre; the branch image was a temporary validation deployment. A later promotion from develop Lerma would again drop the v1 bindings, so the branch Lerma must drive the upgrade until this lands.
  • The v1 container applications are idle (no instances) and start only when a converted tenant’s DO calls container.start().
  • Next: S5 / PER-3 conversion orchestration (intent + convertToV1 through the control plane) so a tenant can be converted and served on SandboxV1.

S5 / PER-3 — legacy → v1 conversion, live (2026-11-05)

Bundle unstable-e7d193fd… promoted to client-sandbox-v1 (id 20) from the branch Lerma (ghcr.io/0xcab0/lerma:v1-sandbox-02327309, deployed temporarily).

CheckResult
POST /_control/:slug/migrate200 { state: "committed", recordId: "77cadb66-cc11-4561-a6b5-edc1f1a52248" } on client-sandbox-v1-dev
Conversion recordsandbox-migration/client-sandbox-v1/conversion.json — v1 DirectoryBackup, dir=/home/openclaw, 22.5 MB tar+zstd/1, source f19e007a…
C8 barriersandbox-migration/client-sandbox-v1/stable.json points at the record (committed 12:04:26)
Runtime gatebackup on the converted instance → 503 SANDBOX_V1_CONTROL_UNSUPPORTED (v1)
Gate is per-tenantbackup on client-sandbox-v1 (still legacy) → 200 (no conversion intent)
v1 containertenant-client-sandbox-v1-dev-sandbox-v1-staging / instance openclaw running, image digest sha256:bd925b80…
Served end-to-endGET /api/health (health-monitor service token) → 200, status: "running", startup.ok: true, readiness.ok: true, aiGateway/container/durableObject/r2 all up
Instance recordsboth instances running on unstable-e7d193fd…

Defects found and fixed while making the conversion work

  1. A queued conversion stayed queued — at the time this read as “the DO alarm never fires in the tenant runtime” (no alarm span, no error), so the control plane briefly awaited v1ConvertNow() inline (the legacy checkpoint’s long-request pattern). Corrected: alarms do fire on this lane — the idle-snapshot cadence alarm later committed record 4437c278-… with no operator action — so the inline path was removed. migrate is now queue-only (202 + poll) and the conversion runs on the DO alarm, with the durable marker re-arm (ensureConversionArmed) covering a lost alarm. See WORKER-DEVELOPMENT.md § Sandbox v1 conversion orchestration (S5).
  2. exec stdin silently truncated the upload. Passing a ReadableStream as exec stdin is accepted but did not deliver every byte (unsquashfs read a partial file and exited 2, while the R2 object was a valid SQUASHFS 4.0 lz4 image, verified locally and in the sandbox base image). Uploads now use stdin: 'pipe', close the pipe before awaiting the exit code, and surface the extractor’s stderr.
  3. The archive is not a copy of /home/openclaw. The legacy bridge stages the roots under .fouria-legacy-stage/{openclaw,clawd} and leaves /home/openclaw/.openclaw + /home/openclaw/clawd as symlinks into /root. The v1 image has real directories (a DirectoryBackup does not follow symlinks), so extraction now materializes the staged roots as real v1 directories and relocates the absolute legacy paths (/root/openclaw → /home/openclaw, length preserving and applied in place to the state SQLite with a node:sqlite quick_check; /root/clawd → /home/openclaw/clawd in text files only). A snapshot without a complete legacy stage fails closed.
  4. Serving before the barrier committed. A present intent used to resolve as v1 immediately, so a failed conversion could wake a fresh empty v1 container and onboard over the legacy recovery point. The gate now requires the record + pointer (C8) and answers 503 SANDBOX_V1_CONVERSION_PENDING until then.
  5. A gateway launch failure retracted a committed conversion. convertToV1 no longer launches: the commit is the data artifact and a launch failure is logged (sandbox.v1.conversion.launch_failed) and retried by the next request’s v1 boot, instead of reporting failed and re-running the destructive conversion on retry.
  6. Lerma marked a healthy rollout as error. await_container_healthy required active_rollout_id to clear, but the platform keeps reporting it after every instance serves the new image (both tenants: 3/3 healthy for the whole 10-minute budget). Completion is now “no active rollout or the application image carries the target version”.

Pre-flight proof of the materialization

materializeLegacyArchive was driven against the real stable archive inside the published v1 image (a docker exec-backed fake container): the extracted state boots start-openclaw.sh start with OPENCLAW_RESTORED=true, /health and the Control UI answer 200, the agent workspace is present, and the state database reports quick_check: ok with zero /root/openclaw references.

Known gaps (next slices)

  • The Control UI (GET /) sits behind the tenant host Access application, which rejects the health-monitor service token (service_token_status: false in the login meta) and needs an interactive identity, so the browser surface could not be exercised headlessly. The health path is a separate, token- accepting Access app and is what proved readiness (status: running, startup/readiness ok, all components up).
  • v1 control actions other than the idempotent migrate still answer 503 SANDBOX_V1_CONTROL_UNSUPPORTED (checkpoint/backup/resume on v1).
  • Only the dev-env instance was converted; the production-env staging instance remains legacy until the S6 fleet decision.

Deployment state left behind

  • Staging Lerma (root@46.225.138.152, container lerma-pre) runs the branch image ghcr.io/0xcab0/lerma:v1-sandbox-02327309; the original compose file is kept at /opt/lerma/docker-compose.yml.bak-v1. Roll back with /tmp/rollback-lerma.sh (or restore that backup and docker compose up -d --no-deps --force-recreate app).
  • The staging Dispatch (dev.fouria.io) is the branch deploy, so it still forwards migrate and echoes the conversion state. A develop deploy replaces it and drops that forwarding.
  • Both are deliberately left in place while the v1 lane continues (the converted tenant needs the v1-aware upgrade path); they are staging-only and must be reverted before any production promotion.

PER-2 — v1 snapshots, recovery point and restore drill (2026-11-05)

Bundles unstable-6795827b… (catalog + snapshots + freeze) and unstable-473c46b0… (pre-catalog adoption) promoted to client-sandbox-v1 (id 20); the drills ran on client-sandbox-v1-dev.

CheckResult
POST /_control/:slug/backup200; sandbox.v1.snapshot.committed (record 79165e52…, 22.5 MB) — served by the DO, no .sqsh written
Catalog + pointersandbox-migration/client-sandbox-v1/records/<id>.json; stable.json advanced to the new record
Encryption at restsandbox-v1/openclaw/<id>.tar.zst.enc (22.5 MB) + <id>.meta.json (encrypted: true, plaintext size/sha256) — no plaintext object
POST /_control/:slug/freeze200; snapshot committed then the container was destroyed (status: stopped)
Cold boot restoreone non-passive GET /api/health (health-monitor token) → new container instance started 17:18:17 → 200, running, startup.ok/readiness.ok within ~15 s
Retention4 records → 3; the oldest catalogued record’s entry and 22.5 MB ciphertext removed, the new record protected
Pre-catalog adoptiona conversion record written before the catalog existed was adopted on the next snapshot (fix 473c46b0)

Notes:

  • A durable_object container application is an instance application: the rollout API rejects it (Instance applications do not support rollouts), so a container replacement is either the platform’s own instance churn or the DO’s freeze → next-request boot. freeze therefore doubles as the restore drill lever.
  • A non-passive /api/health boot is detached: the first probe answers 503 stopped in ~0.5 s while the registered boot runs, and the next probe is 200 running. The loading page’s startup WebSocket drives the same boot.
  • Retention only manages catalogued records; a record committed before the catalog existed needs adoption, which is why a snapshot now adopts conversion.json before listing (found live: the pruned record’s payload was deleted correctly, the un-catalogued conversion archive was not).

OCM-2 — durable lanes, boot guards and repair (2026-11-05)

Bundle unstable-889ec3a2… promoted to client-sandbox-v1 (id 20); drills on client-sandbox-v1-dev.

CheckResult
Scheduler runtime resolutionthe alarm snapshot after the promotion wrote v1 record 2c97395d… (catalog + pointer), not a legacy archive
Legacy lane untouchedlegacy app instance stays inactive; backups/f19e007a… and backup-handle.json still date from 2026-11-04
Freeze on the new bundle200, backupId: 8910f736…; pointer advanced to it
Cold boot after the freezeinstance started 17:50:25; 503 stopped ×3 (~36 s, detached boot) → 200 running with startup.ok/readiness.ok
Launch guards (unit)non-zero early exit fails in seconds (not after the 300 s budget); a corruption signature queues doctor-repair (gateway-launch-state-database)

Notes:

  • A v1 upgrade redeploy runs checkpoint (v1 snapshot), not checkpoint-park: park is still legacy-shaped and belongs to SEC-3, so the rollout is not write-blocked on v1 yet.
  • The scale-to-zero path is the platform inactivity stop (no DO idle callback), so “stop without re-archiving on idle expiry” remains open (PER-2 note).
  • The corrupt-state-DB repair is covered by unit tests only: forcing real SQLite corruption on the staging tenant would risk the tenant for no extra signal.

SEC-3 — v1 rollout park and resume (2026-11-06)

Bundle unstable-548cb408… promoted to client-sandbox-v1 (id 20); the drill ran on client-sandbox-v1-dev.

CheckResult
checkpoint-park200 { parked: true, backupId: 2455d8c8…, sessionsPatched: 0, sessionsDrained: false } (no live sessions)
Recovery point committedthe pointer advanced to the park record before the write lock armed
Write while parkedPOST /_control/:slug/backup → 409 TENANT_PARKED (echoed by the dispatch as code)
resume200 { resumed: true }; the park record was cleared
Readiness after resumehealth probe 200 running on the first attempt (park never destroys the container, so no restore is needed)
Unparked resumeno-op that never wakes the tenant (unit-tested)

Notes:

  • The Lerma promotion of this bundle used checkpoint (the direct-upgrade path), not checkpoint-park, and logged Pre-replacement checkpoint unconfirmed plus a skipped legacy job: the v1 payload carried recordId but the control contract names the committed archive backupId, so the dispatch did not echo the evidence. fdb1f639 returns both. Follow-up (closed): right after the promotion the runtime still answered a plain checkpoint without backupId (the checkpoint-park branch from 548cb408 was already serving), so a bundle-blackend question was recorded. A re-deploy of the same bundle settled it: POST …/checkpoint now returns { ok, action: "checkpoint", backupId: "dd0b1be3-7116-4a7d-b970-21f54b08cc5d" } and the deployed content carries the alias. The transferable finding is the preflight asymmetry: upgrade_all_instances_async skips an instance when the live container image already matches the requested version, so a script upload that did not take effect cannot be repaired by re-promoting. Recorded as a follow-up (verify the deployed bundle, not only the image) rather than fixed in this slice.
  • Park is intentionally a write-block only: the container is replaced by the rollout, not by the park, so resume needs no restore. killGateway stays process-scoped.
  • The session drain found nothing to patch on this tenant; the read-only drain and the durable mode map are exercised by unit tests (session-drain.test.ts, session-mode-restore.test.ts).

SEC-3 (Lerma side) — the direct bundle upgrade parks and resumes (2026-11-06)

Instances.upgrade_instance/3 (the path the staging validation promotion API uses) re-uploaded the bundle and rolled the container without write-blocking the tenant. It now mirrors @upgrade_pipeline: parked checkpoint first (fail-closed on a failed or non-parked response), released after the rollout settles and on every error path, sleeping instances skipped without a resume request.

Verified live against the branch Lerma image v1-sandbox-e5dd5278 on client-sandbox-v1-dev:

CheckResult
Containers.checkpoint/2{:ok, %{backup_id: "545f7844-…", parked: true, checkpointed: true}} (direct probe)
Containers.resume/1{:ok, %{resumed: true}}
Upgrade parksa full upgrade_instance moved the v1 recovery point to record d1c9f143-1aba-4bc1-b8e0-31522497eb06 (committed inside the upgrade)
Upgrade releasesPOST …/backup → 200 (not 409 TENANT_PARKED) and /api/health 200 running right after
Missing park (not running)the first drill ran while the container was idle: skipped: not-running was accepted, no resume was sent, and the pointer did not move

Note: logs emitted inside lerna rpc probes are not visible in docker logs (the RPC process does not own the console), so this drill is evidenced by the durable v1 pointer + the post-upgrade write acceptance rather than by Lerma log lines. A promotion-driven run (Task in the app) is the way to read those lines.

Rollout park through the real promotion lane (2026-11-06)

Bundle unstable-fd05f0c3… (the v1 outbound-bridge fix) was promoted through Lerma’s promotion API, i.e. the same Task-driven upgrade_instance/3 path a release uses. This is the first end-to-end park → replacement → resume run:

PhaseEvidence
Park16:06:41 Pre-replacement checkpoint parked for tenant-client-sandbox-v1-dev: backup_id="d1c02116-…" parked=true
Recovery pointsandbox-migration/client-sandbox-v1/stable.json → d1c02116-5363-4ab8-bb3e-388a776ce03d (committed before the upload)
Script + rolloutscript PUT …/scripts/tenant-client-sandbox-v1-dev, then the container application rolled to unstable-fd05f0c3…
SettleWaiting for … container health … healthy 3, rollout_active: false
Resume16:09:21 Resume after container replacement for tenant-client-sandbox-v1-dev: resumed=true
AfterPOST …/backup → 200 (park cleared); /api/health → 200 running, readiness.ok: true; instance running on fd05f0c3…

Defects found by the drill

  1. Premature convergence. The promotion gate read the DB only: upgrade_all_instances_async sets deployed_version to the requested version before deploying, and HealthMonitor.mark_healthy/3 flipped the in-flight upgrading instance back to running (the container being replaced still answers 200). The gate therefore declared success ~30 s into a rollout that took ~2m40s. Fixed: the status endpoint reports serving_version (the live container-application image tag), promote-validation.sh requires it to match, and the health monitor no longer transitions upgrading instances.
  2. Stuck rollout carried across promotions. tenant-client-sandbox-v1 (prod-env) still had rollout_active: true on the previous image (unstable-548cb408…) from the earlier promotion, so the fd05 promotion timed out settling it and marked the instance error for hours; because the gate ignores error instances, the promotion still reported success. Re-running upgrade_instance superseded the stuck rollout and the instance is now running/healthy on fd05f0c3…. Follow-up worth doing: for a converted tenant, gate on the SandboxV1 application (or skip the legacy app), since the legacy app is unused by the v1 runtime and can wedge independently.
  3. upgrade_instance reliability verified for the Lerma-side park: sleeping containers report skipped: not-running (no park, no resume), a failed or plain (parked: false) checkpoint aborts before the upload, and the park is released on the error path too (all covered by test/lerma/instances_upgrade_test.exs).

Still open for this bundle

The v1 outbound virtual hosts fix (.github-published src/sandbox-outbound.ts

  • SandboxV1 intercept registration) is unit-verified and live on both instances (fd05f0c3…), but its runtime effect (the container reaching otel.internal) has no operator-visible signal in the v1 lane yet: the tenant was idle, so no model spans were exported and no [otel] Received … log line was produced. Confirm on the next agent turn (or when an operator checks the tenant’s diagnostics-otel health).

PER-2 — restore by record id on v1 (2026-11-06)

POST /api/admin/storage/restore/:id on a converted tenant used to fall into the legacy .sqsh path (unsupported on v1), so an operator could not apply an older v1 recovery point. The route now branches on the runtime:

StepBehaviour
Maintenance windowopened (restore) before the DO touches the container; the v1 ensureGateway branch honors it, so no request can reboot under the swap
Stop writersDO kills the single gateway owner, then waits (≤5 s) for /health to fail so the SDK swap does not hit EBUSY
Apply the recordDirectoryBackup.restore(record, { dir: HOME_DIR }) — the SDK swaps the directory atomically
Recovery pointby default the stable pointer advances to the restored record (?setCurrent=0 applies it without promoting)
Fail-closedunknown/unreadable record → 404, container untouched; a restore already in progress is refused by an in-DO flag
BootWorker boots the restored state and reports gatewayReady; a boot failure never undoes the restore (next request retries)

v1 restores are synchronous (there is no legacy job queue for the record format).

Live drill status: the code path is unit- and corpus-verified (sandbox-v1.test.ts, storage.test.ts, business-rules/sandbox-v1-persistence.txt), but the live drill is blocked by Access: /api/admin/* on a tenant host is behind the host Cloudflare Access app, and the only service token available to this lane (the health monitor) is accepted by the health app, not the host app. An operator session (or a browser login) is required to exercise the route on client-sandbox-v1-dev; the bundle unstable-9952203a… ships the code.

Promotion gate fix verified live (2026-11-06)

unstable-9952203a… (v1 restore-by-record) was promoted through the validation lane after redeploying the branch Lerma that carries the new gate:

  • the branch Lerma build v1-sandbox-9952203a exposes serving_version in GET /api/provisioning/upgrade/:slug;
  • the promotion waited for the real rollout: Lerma accepted … at the start, then convergence only after both instances reported serving_version == unstable-9952203a… (rollout_active: false, healthy 3); the pre-fix run had declared success ~30 s in while the rollout was mid-flight;
  • park/resume evidence on the dev instance: recovery point advanced to 26c78dac-ffcb-4574-a4b8-5b82c4be792b (committed at 21:57:22, before the image rollout), and POST /_control/…/backup afterwards returns 200 (the park was released by resume);
  • final state: both instances deployed_version = serving_version = unstable-9952203a…, container health 3/3, tenant-client-sandbox-v1-dev running, tenant-client-sandbox-v1 sleeping (idle container).

Operational note: the promotion gate requires serving_version, so the Lerma image running on staging must include that change; an older Lerma image makes a promotion hang until its timeout (it never sees the field). The staging Lerma was redeployed from v1-sandbox-9952203a (compose backup: /opt/lerma/docker-compose.yml.bak-latest-pre, which still points at latest-pre).

Combined promotion: promotion-gate repair + v1 idle snapshot cadence (2026-11-07)

Two slices landed together and shipped as one validation bundle, so the staging lane picked up the cadence and the promotion-gate repair in a single rollout:

CommitSlice
4389a6e4Lerma promotion repair + truthful gate. POST /api/provisioning/upgrade/:slug also retries error instances whose deployed_version is not the requested version (an instance already declaring the target is excluded, since re-driving a broken container at the target only loops). promote-validation.sh is_converged now requires zero error instances, so a promotion can no longer report success while a tenant stays down at the old image.
855810b9v1 idle snapshot cadence (PER-2). A converted tenant is owned by SandboxV1, and raw ctx.container has no eviction hook, so the platform’s inactivity stop silently dropped writes since the last committed record. The DO now arms a durable marker + ctx.storage.setAlarm on startContainer() and, on alarm(), commits a backup snapshot then re-arms. It never wakes a stopped container, skips while a restore is in flight, and skips when stable.json is newer than the interval (deduping with the ActivityScheduler). resolveSnapshotIntervalMs resolves SANDBOX_V1_SNAPSHOT_INTERVAL_MS → BACKUP_INTERVAL_MINUTES → 1 h, capped at half the inactivity timeout (staging 7.5 m, dev 1 h). selectRecordsToPrune breaks a same-millisecond createdAt tie by record id, so retention is deterministic.
795f1dabTest hygiene (unrelated to the track). The ensure_container_application/5 keyless dev-SSH test now pins its own :provisioning[:dev_instance_ssh_public_key]; config/runtime.exs loads the developer-local apps/lerma/.env (which sets DEV_INSTANCE_SSH_PUBLIC_KEY), so the test only passed on a machine without that entry.

Autogenerated-types covenant (checked, no fix needed)

855810b9 adds SANDBOX_V1_SNAPSHOT_INTERVAL_MS?: string to apps/fouria/src/types.ts. apps/fouria/AGENTS.md calls that file autogenerated, but bun run types (wrangler types) writes apps/fouria/worker-configuration.d.ts only — src/types.ts is the hand-maintained OpenClawEnv, and the call site already narrows it with its own SnapshotCadenceEnv ({ SANDBOX_V1_SNAPSHOT_INTERVAL_MS?, BACKUP_INTERVAL_MINUTES? }). The addition therefore stands, no call-site cast was needed, and the regenerated worker-configuration.d.ts (unrelated workerd/binding churn) was reverted rather than committed.

Deployment

  • Bundle + both runtime images: unstable-795f1dab0d6b2d4bce7b6a98c9b5afb3be089e1f (fouria:unstable-… + fouria:v1-unstable-…, R2 fouria-bundles/unstable-…/fouria.mjs and container-manifest.json).
  • Staging Lerma image: ghcr.io/0xcab0/lerma:v1-sandbox-795f1dab, built from the repo root and shipped by docker save | ssh … docker load (no ghcr push), compose backup /opt/lerma/docker-compose.yml.bak-pre-795f1dab, container lerma-pre healthy and /health 200.
  • One promotion for everything: Lerma accepted unstable-795f1dab…: 2 running instance(s) → client-sandbox-v1 is running unstable-795f1dab…. The direct bundle-only path (Instances.upgrade_instance/3) completes park → script upload → container rollout → resume inside the POST, so the gate converged on its first poll (park/resume visible in the Lerma log below).

Live drills (2026-11-07, unstable-795f1dab…)

DrillResult
Status gateboth instances deployed_version = serving_version = unstable-795f1dab…, status running / sleeping, container_health.errors: 0, healthy: 3 / 2 (rollout_active: false)
checkpoint-park (dev)200 parked: true, backupId: 63f79239-9d43-4824-96b0-83f37e34b71e, sessionsPatched: 0 (no live sessions)
resume (dev)200 resumed: true
backup after resume200 (ok: true) — proves the park guard was released (a parked tenant answers 409 TENANT_PARKED); the dispatch projection only echoes backupId for park/freeze
Recovery point (R2)sandbox-migration/client-sandbox-v1/stable.json → recordId 1a169e07-cb50-444c-8ae5-5976bfe72a20, committedAt 2026-11-07T08:50:12Z (written by the drills)
Lerma park/resume evidencePre-replacement checkpoint parked for tenant-client-sandbox-v1-dev … parked=true (08:46:35Z) and Resume after container replacement … resumed=true (08:48:39Z)
Idle cadenceunit-verified, live observation pending the next hourly tick (dev cadence is capped at 1 h); it is not waited on

Verification for this bundle: bun run format:check && bun run lint && bun run typecheck && bun run test (3974 passed, 8 skipped), cctr apps/fouria/test/business-rules/ (582/582), scripts/staging-validation.test.ts (83/83), bun run plan:sandbox-v1:check, mix check, mix license_audit, and the full Lerma suite (1242 passed, 4 excluded).

CDP bridge merge fix + HEAD redeploy (2026-11-07)

The develop merge (806a2580) brought the CDP bridge (#563) in, but the merge resolution left cdp.internal out of the shared outbound host map: the v1-side refactor had moved TenantSandbox.outboundByHost into src/sandbox-outbound.ts, so handleCdpRequest existed while no route referenced it. Two apps/fouria/src/index.test.ts node-pool suites failed on the merged tree (4020 passed, 8 skipped, 2 failed) — format:check, lint and typecheck stayed green because the map is generic, so this was a merge-resolution regression rather than a type error, and because both runtimes consume the same map v1 was missing the intercept too.

CommitSlice
e56ac9c8cdp.internal joins the shared host map. fix(fouria): register the cdp.internal bridge on the shared outbound host map adds [CDP_INTERNAL_HOST]: handleCdpRequest to SANDBOX_OUTBOUND_ROUTES (6 hosts: releases/docs/leads/software.r2.internal, otel.internal, cdp.internal), drops the bridge imports left dead in src/index.ts, and pins the host in sandbox-outbound.test.ts / sandbox-v1.test.ts; the two node-pool suites now stub @cloudflare/sandbox-legacy, which the CDP bridge pulls in transitively.

Deployment

  • Bundle + both runtime images: unstable-e56ac9c8d79f062ee6b66bfed1f221e1e82f102f (fouria:unstable-… + fouria:v1-unstable-… on registry.cloudflare.com, R2 fouria-bundles/unstable-…/{fouria.mjs,container-manifest.json}).
  • Staging Lerma image: ghcr.io/0xcab0/lerma:v1-sandbox-e56ac9c8, built from the repo root and shipped by docker save | ssh … docker load (no ghcr push), compose backup /opt/lerma/docker-compose.yml.bak-pre-e56ac9c8, container lerma-pre healthy and /health 200.
  • One promotion for everything: Lerma accepted unstable-e56ac9c8…: 2 running instance(s) → client-sandbox-v1 is running unstable-e56ac9c8…. The bundle-only path parked both instances (v1-dev 20:02:50Z, v1 20:07:39Z), replaced the container and resumed each (v1-dev 20:07:16Z, v1 20:11:01Z).

Live drills (2026-11-07, unstable-e56ac9c8…)

DrillResult
Status gateboth instances deployed_version = serving_version = unstable-e56ac9c8…, status running, container_health.errors: 0, healthy: 2 / 3, rollout_active: false
checkpoint-park (dev)200 parked: true, backupId 8f4e88bc-368a-48c5-888e-30317895f6f8, sessionsPatched: 0 (no live sessions)
resume (dev)200 resumed: true
backup after resume200 (ok: true) — the park guard is released (a parked tenant answers 409 TENANT_PARKED)
Recovery point (R2)sandbox-migration/client-sandbox-v1/stable.json → recordId 0e426f0e-577b-40a1-a4a6-c0abcafaa20e, committedAt 2026-11-07T20:12:55Z, trigger: "backup" (the park’s checkpoint record is 8f4e88bc-…)
cdp.internal (deployed)both tenant scripts in fouria-tenants-staging (10,767,930 bytes, same content) contain cdp.internal, and SANDBOX_OUTBOUND_ROUTES lists all 6 hosts including [CDP_INTERNAL_HOST]: handleCdpRequest
Lerma park/resume evidencePre-replacement checkpoint parked for tenant-client-sandbox-v1-dev … parked=true (20:02:50Z) / Resume … resumed=true (20:07:16Z); same pair for tenant-client-sandbox-v1 (20:07:39Z / 20:11:01Z)
Idle cadencestill [suite] only — not waited on for this deploy (dev cadence is capped at 1 h)

What is still unmeasured. The cdp.internal registration is now observed in the deployed bundle on both runtimes; the end-to-end browser round-trip (a converted tenant driving Browser Rendering through the container’s cdp-proxy.mjs) was not exercised — test/e2e/r_browser_remote_profile.txt is cloud-profile only and was deliberately not run.

Verification for this bundle: bun run format:check && bun run lint && bun run typecheck && bun run test (4022 passed, 8 skipped, 199 files), cctr apps/fouria/test/business-rules/ (593/593), scripts/staging-validation.test.ts (83/83), bun run plan:sandbox-v1:check, mix check, mix license_audit, and the full Lerma suite (1259 passed, 4 excluded).

Operator surface: Lerma migration + ROOT debug tools, and the already_converted signal (2026-11-07)

Two slices that make a conversion reachable by an operator instead of by a hand-rolled control call, plus the correctness fix that came out of validating them live. Nothing here converts a tenant.

CommitSlice
c01d00b1ROOT debug tools work on Sandbox v1. Every /debug/* route was audited against the client a converted tenant actually serves with (V1SandboxClient): a shared readFinishedProcessLogs() (legacy logs() → v1 output()) fixes /processes?logs=true and /logs?id= for a finished process; /gateway-logs returns path + runtime; /restart-openclaw gained a v1 branch (no legacy checkpoint, no restore-needed); /reprovision checkpoints with v1CreateSnapshot and validates restorability with the C8 verifyRecoveryPoint; /backup/:mode snapshots through the DO; /destroy-container clears the startup cache and the isolate start dedupe; /env reports sandbox_runtime_actual; /debug/sleep answers 503 SANDBOX_V1_SLEEP_UNSUPPORTED (SDK 1.0 has no stop primitive). Suite-verified only (see the drill table).
804c512eLerma can migrate a legacy tenant. Containers.migrate/2 posts the migrate control action with a 30-minute budget and normalises {state, record_id, duration_ms}; SandboxV1Migration computes eligibility (staging platform, never the reserved client slug, a client’s dev environment only, active instance, committed legacy recovery point) and the dry-run report; POST /api/provisioning/migrate/:slug serves ?dry_run=1 plus real runs (200/403/409/502); mix lerma.sandbox_v1.migrate prints the same table and requires --yes off a TTY.
6602f1fcThe dry-run stops calling an already-converted tenant eligible. A conversion leaves the legacy archive in R2 (it is the conversion’s source), so the legacy recovery point alone cannot tell: Containers.v1_recovery_point/1 reads the v1 stable pointer (sandbox-migration/<canonical tenant>/stable.json, in the same per-instance bucket the legacy recovery point resolves), R2.object_body/2 supplies the record id, and SandboxV1Migration emits already_converted (true/false/nil when unreadable) + v1_record_id per instance and at the top level, as both the blocking reason and a warning. A real run on a converted tenant answers 200 {state: "committed", already: true, record_id} without a control call, so a no-op never occupies the 30-minute request path.

Deployment (2026-11-07)

  • Bundle + both runtime images: unstable-6602f1fc98fc41692678ea05718adde8505f6b56 (fouria:unstable-… + fouria:v1-unstable-… on registry.cloudflare.com, R2 fouria-bundles/unstable-…/{fouria.mjs,container-manifest.json}).
  • Staging Lerma image: ghcr.io/0xcab0/lerma:v1-sandbox-6602f1fc, built from the repo root and shipped by docker save | ssh … docker load (no ghcr push); compose backup /opt/lerma/docker-compose.yml.bak-pre-6602f1fc; lerma-pre healthy and /health 200.
  • One promotion for everything: Lerma accepted unstable-6602f1fc…: 2 running instance(s) → client-sandbox-v1 is running unstable-6602f1fc…. The park checkpoints were 556d5773-… (dev, parked 21:34:57Z → resumed 21:40:02Z) and 5eb28f82-… (production-env, parked 21:40:31Z → resumed 21:44:10Z).
  • Status gate afterwards: both instances status: running, deployed_version = serving_version = unstable-6602f1fc…, container_health.healthy 3 (dev) / 2, zero errors.

Idle cadence — first live tick (observed, not injected)

sandbox-migration/client-sandbox-v1/stable.json advanced to 4437c278-baf2-4ec2-af7f-fa9598f5eef7 at 2026-11-07T21:18:55.381Z with records/4437c278-….json recording trigger: "backup" — a scheduled v1 snapshot committed with no operator action and no promotion in flight (the trigger: "checkpoint" records are the promotion parks). This is the live tick the earlier entries listed as pending; the ~1 h dev interval matches resolveSnapshotIntervalMs.

Live drills (2026-11-07, unstable-6602f1fc… + v1-sandbox-6602f1fc)

All requests used the provisioning application key with X-FourIA-Application: fouria; the key is never printed.

DrillResult
Dry-run, converted tenant (client-sandbox-v1)200, eligible: false, reasons: ["already_converted"], warnings: ["already_converted"], top-level already_converted: true, v1_record_id 556d5773-5ea2-4fff-b777-3221234dcaf1. Per instance: tenant-client-sandbox-v1-dev already_converted: true; tenant-client-sandbox-v1 already_converted: false + production_environment
Dry-run, production-environment instance?instance=tenant-client-sandbox-v1 → eligible: false, reasons: ["production_environment"], already_converted: false (its own bucket has no pointer — it is still legacy), warning checkpoint_recommended_before_conversion
Dry-run, reserved slug (client)409, code: MIGRATION_NOT_ELIGIBLE, reasons: ["reserved_client"], no control call
Dry-run, non-existent client (client-sandbox)404 — the slug briefed for the negative gate does not exist on staging; the production-environment refusal was recorded on client-sandbox-v1’s own production instance instead
Idempotency drill (real run, no dry_run)200 in 1.16 s: {state: "committed", already: true, record_id: 556d5773-…, duration_ms: 0} — no control call, so no 30-minute no-op path
Recovery point unchanged by that runstable.json 556d5773-… @ 21:34:56.087Z before and after; conversion.json still 77cadb66-cc11-4561-a6b5-edc1f1a52248 @ 2026-11-05T12:04:26Z (no new conversion record)
Tenant serving after the promotionGET /api/health on staging-client-sandbox-v1.fouria.io and …-dev.fouria.io (health-monitor service token): 200, status: running, startup.ok + readiness.ok true, container/durableObject/r2/aiGateway up
ROOT debug toolsNot verified live: GET /debug/version and /debug/gateway-logs on the tenant host answer 302 to paso4.cloudflareaccess.com (service_token_status: false), so a headless call cannot reach them. Unchanged from the earlier entries — suite-verified only

What this bundle did not do. No tenant was converted. The production-env instance of client-sandbox-v1 is still legacy (already_converted: false), and policy keeps it that way: only a client’s dev environment tenant is eligible (contract C2/S7). No migrate control action was sent during these drills.

Verification for this bundle: bun run format:check && bun run lint && bun run typecheck && bun run test (4037 passed, 8 skipped, 199 files), cctr apps/fouria/test/business-rules/ (596/596), scripts/staging-validation.test.ts (83/83), bun run plan:sandbox-v1:check, mix check, mix license_audit, the full Lerma suite (1305 passed, 4 excluded) and workers/fouria-dispatch --project unit (127 passed).

Durable migration job + read-only status, live-validated (2026-11-08)

The migrate control action stopped being a synchronous conversion. It is now a queue-only enqueue of a durable job on the tenant’s SandboxV1 Durable Object, and polling moved to a separate read-only action so that asking for the state can never convert a tenant. Nothing here converts a tenant: the converted instance in this drill was converted on 2026-11-05.

CommitSlice
be255fddThe conversion runs as a durable Durable Object job. migrate calls v1RequestConversion() and answers immediately (202 queued/running, 200 committed + recordId, 502 failed, 503 SANDBOX_V1_UNAVAILABLE when the binding/DO is unreachable — nothing recorded, safe to retry). The conversion itself streams the legacy squashfs, extracts it in the v1 container and uploads a DirectoryBackup on the DO alarm() (conversion first, cadence second, each wrapped), is durable (migration:convert-request marker + migration:convert-status + 1-based attempt) and idempotent, alarm() orders running → committed before dropping the marker and before the launch (a launch failure never retracts the commit), a failure records failed with a bounded message instead of throwing, ensureConversionArmed() re-arms a lost alarm from conversionState()/requestConversion()/status()/ensureGateway() without waking the container, and #conversionRun is the single owner so overlapping alarm turns cannot start a second destructive conversion. v1ConvertNow was removed from the DO, the RPC surface and V1SandboxClient.
92a21080migration-status is a read-only action. X-FourIA-Control: migration-status calls only v1ConversionState() — never v1RequestConversion() — so it cannot enqueue, start work or wake the container; a legacy tenant answers 200 {state: "absent"} without addressing a v1 DO. It never answers 202. The Dispatch Worker forwards it (CONTROL_ACTIONS, control echo of state/recordId, no snapshot CPU budget). Lerma’s Containers.migration_status/2 normalises absent/queued/running/committed/failed. a3b3e162 is the Lerma half (enqueue + poll: GET /api/provisioning/migrate/:slug reads status, the Mix task enqueues and then polls through migration-status, and the old edge/524 caveat no longer applies to the trigger). Its commit message reads feat(fouria): … because two sessions raced; the content is the Lerma change.

The alarm-driven contract is the reason the trigger no longer needs a long request budget: POST returns in ~1 s, and the operator polls the read-only action with a budget they own (:migrate_wait_timeout_ms, 30 min).

Deployment (2026-11-08, one combined rollout)

  • Bundle + both runtime images: unstable-92a21080233dca606cc4e6f91a99eb22397968cf (fouria:unstable-… @ sha256:7d984068b1269b4a2ed32c2cdd6dd7afdb5b48ec68fd9f4f3ff48b0ed2dd7801 and fouria:v1-unstable-… on registry.cloudflare.com, R2 fouria-bundles/unstable-…/{fouria.mjs,container-manifest.json}).
  • Dispatch Worker (staging) redeployed — required by the contract and easy to miss: without it the tenant-facing action exists but /_control/:slug/migration-status answers 400 Unknown control action (observed before the deploy), so a Lerma GET that reaches the control plane can only fail. Deployed from this branch (bun run --cwd workers/fouria-dispatch deploy:staging, version 8da2700f-f886-4d68-a923-8022c947953d); workers/fouria-dispatch has no develop-only commits, so the deploy is an additive superset.
  • Staging Lerma image: ghcr.io/0xcab0/lerma:v1-sandbox-92a21080, built from the repo root and shipped by docker save | ssh … docker load (no ghcr push); compose backup /opt/lerma/docker-compose.yml.bak-pre-92a21080; lerma-pre healthy, /health 200.
  • One promotion: Lerma accepted unstable-92a21080…: 2 running instance(s) → client-sandbox-v1 is running unstable-92a21080… (both instances rolled with the same image + the digest-pinned SandboxV1 sandbox image).

Live validation of the async contract (2026-11-08)

Provisioning application key + X-FourIA-Application: fouria; the key is never printed. Read-only calls only — no POST on a legacy tenant.

DrillResult
GET converted client (client-sandbox-v1)200 in 1.59 s: {ok: true, state: "committed", already: true, record_id: "19753f1a-2554-4f39-b021-73602719af06", reasons: ["already_converted"], v1_record_id: "19753f1a-…", duration_ms: 0} — the already_converted short-circuit, so no control call at all (Lerma logs show only the two pointer GETs)
POST converted client (no dry_run)200 in 1.58 s: identical body (state: committed, already: true, record_id: 19753f1a-…, duration_ms: 0) — no enqueue and no control call
GET reserved slug (client)409, code: MIGRATION_NOT_ELIGIBLE, reasons: ["reserved_client"], no control call
GET client with no instances (tarjetero)409, code: MIGRATION_NOT_ELIGIBLE, reasons: ["no_active_instance"] — no control call
Read-only action, converted tenant (deployed Lerma → dispatch → tenant Worker → v1ConversionState)Containers.migration_status(%{cf_worker_name: "tenant-client-sandbox-v1-dev"}) → {:ok, %{state: :committed, record_id: "77cadb66-cc11-4561-a6b5-edc1f1a52248", duration_ms: 355}} (the conversion record, not the pointer)
Read-only action, legacy tenant on the new bundle (tenant-client-sandbox-v1){:ok, %{state: :absent, record_id: nil, duration_ms: 282}} — the legacy branch answers absent without addressing a v1 DO, and nothing was enqueued
Same action through the dispatch edge (POST /_control/client-sandbox-v1-dev/migration-status)200 {ok: true, status: 200, state: "committed", recordId: "77cadb66-…"}; …/client-sandbox-v1/migration-status → 200 {state: "absent"}
No conversion started by any of the abovesandbox-migration/client-sandbox-v1/stable.json identical before and after ({"recordId":"19753f1a-…","committedAt":"2026-11-08T12:20:51.098Z"}); conversion.json identical (77cadb66-…, createdAt 2026-11-05T12:04:26.260Z); no new records/<id>.json; Lerma logs show no migration queued/committed/failed line for these calls
Park → resume → backup (client-sandbox-v1-dev, v1)200 / 200 / 200: checkpoint-park {backupId: 6b1f7d89-9e5b-459a-87b1-a1068deadc6f, parked: true, sessionsPatched: 0, sessionsDrained: false} (5.5 s), resume {resumed: true} (1.6 s), backup (4.5 s)
Recovery point after the backupstable.json advanced to 54eac3a0-d489-441e-ac99-8f65e96af4d9 @ 2026-11-08T12:33:53.317Z; retention pruned to 3 records (19753f1a-…, 6b1f7d89-…, 54eac3a0-…)
/api/health on the converted dev tenant200, status: running, startup.ok + readiness.ok true, container/durableObject/r2/aiGateway up, 1.35 s
/api/health on the production-env legacy tenant503, status: stopped, components.container: sleeping — the idle-slept legacy container is not woken by a headless probe (by design); POST /api/warmup with the health-monitor token answers 302 (host Access app), so a wake needs an operator browser session. Same state the earlier entries recorded as sleeping
Status gate (GET /api/provisioning/upgrade/client-sandbox-v1)both instances status: running, deployed_version = serving_version = unstable-92a21080…, container_health.errors: 0, healthy 2 (production-env) / 3 (dev), rollout_active: false

What could not be exercised live. The Worker’s 202 {state: "queued"} enqueue path (and the migration-status reads queued/running) require an eligible legacy tenant: on staging the only legacy client is the reserved client slug (409 by contract C2) and tarjetero has no instances, so a real enqueue would have meant converting a tenant — explicitly out of scope. That path is covered by src/index.test.ts (202 queued/running, 200 committed, 502 failed, 503 unreachable), src/sandbox-v1.test.ts (queue/arm, alarm commit, cadence-after-conversion, overlapping alarms, lost-alarm re-arm) and workers/fouria-dispatch/test/unit/control.test.ts (status forwarding + echo).

Two follow-ups worth knowing: (1) a tenant still on a pre-change bundle answers 400 Unknown control action for migration-status (observed on tenant-client-dev) — the status action only exists on instances promoted with this bundle, which is also the bundle that makes migrate durable; (2) the instance image_tag column still shows the image recorded at provisioning (unstable-e7c70d3e…) while deployed_version/serving_version and the container’s actual image are unstable-92a21080… — a pre-existing field that no path in this change writes.

Verification for this bundle: bun run format:check && bun run lint && bun run typecheck && bun run test (4050 passed, 8 skipped, 199 files), cctr apps/fouria/test/business-rules/ (600/600), scripts/staging-validation.test.ts (83/83), bun run plan:sandbox-v1:check, mix check, mix license_audit, the full Lerma suite (1327 passed, 4 excluded) and workers/fouria-dispatch --project unit (129 passed). The dispatch integration project fails 11/27 with asyncLocalStorage.disable() is not implemented in this environment (Node 22.23.2 / workerd); it is environmental and predates the change.

Provisioning a tenant directly on Sandbox v1 (“born v1”), live-validated (2026-11-08)

A tenant no longer has to be converted: a fresh instance can be provisioned straight onto the SandboxV1 runtime. The only durable evidence of that choice is the runtime marker Lerma writes before the script deploy, which the Worker reads (tri-state, fail-closed) instead of the conversion intent.

CommitSlice
99ed1d0dWorker contract. src/sandbox-migration/runtime-marker.ts (strict runtime.json shape + tri-state read, the marker’s tenant must match the scope it was read from), resolveRuntimeIntent reads the conversion intent and the marker in parallel (intent wins; present marker → v1; unreadable → unknown; marker absent and SANDBOX_RUNTIME=v1 → pending, so a half-provisioned tenant never boots legacy), /control migrate on a fresh tenant answers 409 MIGRATION_ALREADY_V1 without addressing the DO (503 SANDBOX_V1_RUNTIME_UNKNOWN for an unreadable marker) and migration-status answers absent without touching the DO.
f73b3fabFirst recovery point after the boot. v1EnsureGateway dispatches a detached, best-effort runInitialSnapshot() after a successful boot (owner run and the adopted-gateway path) so a born-v1 tenant is not loss-bounded by the first idle-cadence tick. Only for a tenant with no recovery point (verifyRecoveryPoint = pointer-absent and no conversion.json); single-flight + a shared takeOwnedSnapshot() with the cadence; a failure logs sandbox.v1.initial_snapshot_failed and retries on a later boot (10-minute floor) without failing readiness.
35e37628Lerma side. instances.sandbox_runtime (legacy default | v1, dev-only and staging-only, immutable after creation), SANDBOX_RUNTIME on both the provisioning and re-deploy binding paths (SANDBOX_V1_CUTOVER=true, SandboxV1 DO binding) and the :sandbox_runtime_marker provisioning step that writes sandbox-migration/<canonical-slug>/runtime.json create-only, immediately before :dispatch_deploy (idempotent :existing, refuses an existing conversion intent, deletes the marker on rollback only when that run created it). The migration preflight reports already_v1: true / reason sandbox_runtime_v1 with no control call.
82bc961eDocs-only (ENV-VARIABLES table reformat); deployed as the bundle tag so the promoted artifact matches the pushed tree exactly.

Deployment (2026-11-08)

  • Bundle + both images: unstable-82bc961e95022541bd5abc688293c5e5219b14a9 (fouria:unstable-… @ sha256:371bcb013fb577f9c07ce0f7ccbbf6edacabaaf109dd86c0a104b630a3e6e400 and fouria:v1-unstable-… @ sha256:37e48974e2d0c6d39d370f54986dbd02461e60eb651b0a7d38de238e8cadc303, R2 fouria-bundles/unstable-…/{fouria.mjs,container-manifest.json}).
  • Staging Lerma: ghcr.io/0xcab0/lerma:v1-sandbox-82bc961e (same layers as the 35e37628 build; the tag commit is docs-only and no file it changes is copied into the image), shipped by docker save | ssh … docker load, compose backup /opt/lerma/docker-compose.yml.bak-pre-82bc961e; lerma-pre healthy, /health 200, migration 20261108120000 applied at boot.
  • Dispatch (staging) not redeployed: it already forwards migration-status (POST /_control/client-sandbox-v1-dev/migration-status answered 200 {state: "committed", recordId: "77cadb66-…"} before the rollout), and workers/fouria-dispatch has no develop-only divergence.
  • One promotion: Lerma accepted unstable-82bc961e…: 2 running instance(s) → client-sandbox-v1 is running unstable-82bc961e… (both instances, same image + the digest-pinned SandboxV1 sandbox image).

Live born-v1 drill (2026-11-08)

A throwaway staging client (select-runtime-smoke-82bc, id 63) was provisioned through POST /api/clients/63/instances with {"instance": {"environments": ["dev"], "sandbox_runtime": "v1", "version": "unstable-82bc961e…"}} (provisioning application key + X-FourIA-Application: fouria; the key is never printed). 202 {run_id, sandbox_runtime: "v1"}.

CheckResult
Step orderprovisioning_step_events: … fetch_bundle → telegram_ingress_queue → sandbox_runtime_marker (succeeded) → dispatch_deploy (succeeded) → … — the marker lands before the script deploy
Instance rowsandbox_runtime: "v1", environment: "dev", status: running, deployed_version: unstable-82bc961e…
Markerfouria-backup-select-runtime-smoke-82bc-dev-staging/sandbox-migration/select-runtime-smoke-82bc/runtime.json = {"mode":"fresh","provisionedAt":"2026-11-08T19:37:44Z","runtime":"v1","tenant":"select-runtime-smoke-82bc","version":1}; the retried provisioning logged v1 marker existing (create-only, provisionedAt unchanged)
Deployed bindingsSANDBOX_RUNTIME=v1, SANDBOX_V1_CUTOVER=true, SandboxV1 (durable_object_namespace, class SandboxV1), WORKER_PLATFORM=staging (Cloudflare API, dispatch-namespace script settings)
Containertenant-select-runtime-smoke-82bc-dev-sandbox-v1-staging (scheduling_policy: durable_object) started an instance in cdg14 with the digest-pinned v1 image fouria@sha256:37e48974…
/api/health200, status: running, startup/readiness/liveness success, components: container/durableObject/r2/aiGateway all up, restoreStatus {hadBackup: false, restored: false} (fresh onboarding, no OPENCLAW_RESTORED)
migration-statusPOST https://dev.fouria.io/_control/select-runtime-smoke-82bc-dev/migration-status → 200 {"state":"absent"} (no conversion job; the read does not touch the DO)
migratePOST …/migrate → 409 {"code":"MIGRATION_ALREADY_V1","tenantError":"tenant is provisioned on SandboxV1; there is nothing to convert"}
Initial snapshotsandbox-migration/select-runtime-smoke-82bc/stable.json → {"recordId":"c45449dc-ce29-4999-b7e2-471f3f4d8ce1","committedAt":"2026-11-08T20:09:01.264Z"} — 4 s after readiness (20:08:57), no operator action; records/c45449dc-….json (dir /home/openclaw, size 7 948 131, format tar+zstd/1, trigger "backup") and the encrypted payload sandbox-v1/openclaw/c45449dc-….tar.zst.enc (7 948 180 B) + .meta.json (254 B); no plaintext archive in the bucket
Operator snapshotPOST /_control/select-runtime-smoke-82bc-dev/backup → 200 in 4.0 s; pointer advanced to 0038d1e6-3c86-4c9a-8024-52579fbc84ff @ 20:10:10.196Z (two records retained)
Lerma preflightGET /api/provisioning/migrate/select-runtime-smoke-82bc → 200 {already_v1: true, runtime: "v1", reasons: ["sandbox_runtime_v1","no_legacy_recovery_point","already_converted"], duration_ms: 0} — no control call
No conversionconversion.json is absent from the bucket; the tenant was born on v1 and never converted

One platform wedge, found and recovered, worth knowing. The first provision (19:37) produced a SandboxV1 container application with zero instances even though the DO reported running: true: containerFetch timed out on every health probe and /control …/backup hung until the client gave up, so the gateway never launched and no initial snapshot could happen. The container image itself was fine (it runs locally and the same tag boots elsewhere). DELETE /api/instances/:id + a re-provision (same client, same marker) recreated the app and the instance started in cdg14 within 30 s; the gateway reached readiness ~5 min later. The retry re-used the create-only marker (v1 marker existing), which is also the idempotency proof. Treat a born-v1 instance whose container app stays at zero instances as an operator action: deprovision and re-provision; the marker survives, so the tenant stays v1.

Corroboration from an earlier run of the same code: arua was provisioned born v1 at 19:05:59 (runtime.json mode: "fresh", migration-status absent, a running tenant-arua-dev-sandbox-v1-staging instance) and has been committing v1 records on its snapshot cadence (stable.json advanced through 19:31 / 19:46 / 20:01, all trigger "backup"), so the runtime decision, serving and persistence also hold outside this drill.

Cleanup (2026-11-08)

The throwaway client, its instance, the tenant Worker script and both container applications were removed (DELETE /api/instances/:id → 204, which deprovisions the script and both container applications; DELETE /api/clients/63 → 204, which decommissions the client — the row stays in the archived state, the platform’s normal soft delete, and is no longer listed as active). No select-runtime-smoke script or container application remains. Its R2 bucket was emptied (11 objects deleted, including the marker and the recovery point) but the bucket itself is still refused by R2 as “not empty” because the initial snapshot left an in-progress multipart upload; R2 aborts those automatically, after which the empty bucket can be deleted. No existing tenant was converted and no production instance was touched.

Verification for this bundle: bun run format:check && bun run lint && bun run typecheck && bun run test (4077 passed, 8 skipped, 199 files), cctr apps/fouria/test/business-rules/ (606/606), scripts/staging-validation.test.ts (83/83), bun run plan:sandbox-v1:check, mix check, mix license_audit, the full Lerma suite (1373 passed, 4 excluded) and workers/fouria-dispatch --project unit (129 passed).

Wake-lifecycle merge into v1-sandbox and the combined deployment (2026-11-08)

Staging had been serving unstable-c8aae8cd…, a build of the pushed feature branch fix/sandbox-v1-wake-lifecycle. The born-v1 promotion (unstable-82bc961e…) superseded it, so the two lines were combined on v1-sandbox instead of leaving the staging tenant on the older build: the wake lifecycle (keep-alive renewal, the provisioned instance type on wake, the no-empty-recovery-point skip, and the durable automation wake) is part of the migration track now, not a sibling branch.

Merge

git merge --no-ff origin/fix/sandbox-v1-wake-lifecycle into v1-sandbox (merge commit cd854394, first parent 6fac3e42). The branch had already merged this track at 8ce6a547 (based on 99ed1d0d), so its content replayed cleanly: apps/fouria/src/sandbox-v1.ts auto-merged and the two features compose there — the cadence turn routes through takeOwnedSnapshot (born-v1) while the gateway-owner promise schedules the initial snapshot and the restart path re-arms the keep-alive (wake lifecycle).

Two conflicts, both in docs, resolved preserving both sides:

FileResolution
apps/fouria/AGENTS.mdkept the wake-lifecycle recent-changes bullet and the newer born-v1 bullet (the one that also documents the first recovery point taken after the first boot)
docs/ENV-VARIABLES.mdkept the reformatted table, merged the wake-corrected SANDBOX_RUNTIME text (keep-alive renewal, no snapshot wake, production 6h cap), the production 3h cadence cap and added the CONTAINER_INSTANCE_TYPE row

No code conflict needed a judgement call, so no resolution was guessed.

Deployment

  • Bundle + both runtime images: unstable-cd8543948638f27e98081fb47b3312d5dad15acd (fouria:unstable-… @ sha256:371bcb01…, fouria:v1-unstable-… @ sha256:37e48974…, R2 fouria-bundles/unstable-cd854394…/{fouria.mjs,container-manifest.json}).
  • The published bundle carries both halves (grep of the R2 fouria.mjs): debug/lifecycle, keepalive-pending, wokenFor, CONTAINER_INSTANCE_TYPE (wake lifecycle) and runtime-marker, MIGRATION_ALREADY_V1 (born v1).
  • Staging Lerma: ghcr.io/0xcab0/lerma:v1-sandbox-cd854394, built from the repo root and shipped by docker save | ssh … docker load, compose backup /opt/lerma/docker-compose.yml.bak-pre-cd854394, retargeted from v1-sandbox-82bc961e; lerma-pre healthy, in-container /health 200.
  • One promotion: Lerma accepted unstable-cd854394…: 2 running instance(s) → client-sandbox-v1 is running unstable-cd854394….

Live verification (2026-11-08)

CheckResult
Status gateboth instances deployed_version = serving_version = unstable-cd854394…, status: running, container_health.errors: 0, healthy: 2 / 3, rollout_active: false
checkpoint-park (dev)200 parked: true, backupId 05aa5c01-b8dd-42f1-bbf9-4946b7f813d0, sessionsPatched: 0 (no live sessions)
resume (dev)200 resumed: true
backup after resume200 (ok: true) — proves the park guard was released (a parked tenant answers 409 TENANT_PARKED)
Recovery point (R2)sandbox-migration/client-sandbox-v1/stable.json → recordId 19fa77db-87f1-47de-9771-04e1cbcad4cb, committedAt 2026-11-08T20:36:17Z (12 s before the check)
migration-status (Lerma)GET /api/provisioning/migrate/client-sandbox-v1 → 200 {state: "committed", already: true, record_id: 19fa77db…} (read-only; the non-dev instance still reports sandbox_runtime: legacy)
/api/health (both hosts)200 on staging-client-sandbox-v1.fouria.io and staging-client-sandbox-v1-dev.fouria.io; status: running, startup/readiness/liveness success, container/durableObject/r2/aiGateway all up
Drill client decommissionedno select-runtime-smoke script in fouria-tenants-staging (5 scripts: tenant-client, tenant-client-dev, tenant-client-sandbox-v1, tenant-client-sandbox-v1-dev, tenant-arua-dev) and no matching container application (9 total)
Drill bucketfouria-backup-select-runtime-smoke-82bc-dev-staging still exists and is empty, but r2 bucket delete is refused again (10008 not empty, the in-progress multipart upload from the initial snapshot). The abort needs the S3 API, which this machine cannot reach — *.r2.cloudflarestorage.com resolves to real Cloudflare IPs but the egress path presents core1.netops.test, so curl --aws-sigv4 fails TLS verification before any request. R2 auto-aborts the upload; the empty bucket is deletable afterwards. Only tenant-arua-dev (a separate born-v1 tenant) was left running.

The deployed artifact is unstable-cd854394… (the merge commit). The docs commits on top of it (6852699a and this note) change no file that is copied into the bundle or either image, so unstable-cd854394… still describes exactly what staging is serving.

No tenant was converted by this deployment: the merge and the promotion changed the deployed image only. client-sandbox-v1 keeps sandbox_runtime: legacy (never migrated, per the reserved-path rule) and client-sandbox-v1-dev keeps its existing v1 recovery point; no production instance was touched.

Verification for this bundle: bun run format:check && bun run lint && bun run typecheck && bun run test (4101 passed, 8 skipped, 201 files, no type errors), cctr apps/fouria/test/business-rules/ (614/614), scripts/staging-validation.test.ts (83/83), bun run plan:sandbox-v1:check (166 features, 14 milestones), mix check, mix license_audit (62 deps), the full Lerma suite (1373 passed, 4 excluded) and workers/fouria-dispatch --project unit (129 passed). The live drill suite self-skips without E2E_V1_LIVE=1 (3/3 skipped).

Rollback coverage + upgrade-pipeline repair (2026-11-09)

The OCM-4a incident’s rollback fix was one-sided. Instances.upgrade_instance/3 (the promotion lane) passed preserve_do_namespaces: true, but the dashboard/API pipeline (Instances.update_instance_version/3 → TenantProvisioner.upgrade_instance/3 → :dispatch_deploy) called Containers.deploy_script_and_container/1 without it, so a failed container binding could still delete a live tenant’s routing script exactly as in the incident. Two commits closed it:

CommitChange
5e3e5618TenantProvisioner’s :dispatch_deploy passes preserve_do_namespaces: upgrade?(meta); a fresh provision keeps the destructive rollback so a retry starts clean.
d57bec6dThe shared Instances.deploy_script_and_container/6 takes the flag explicitly (fresh create_user_worker_instance/3 = false, upgrade / AI-gateway reassignment = true), and the upgrade test asserts the full binding set a rebuilt script needs (gateway token, derived CDP secret, AI Gateway/API token, backup key, Telegram queue, runtime).

Deployment: unstable-d57bec6d… published; branch Lerma v1-sandbox-d57bec6d deployed (compose backup /opt/lerma/docker-compose.yml.bak-pre-d57bec6d); one promotion of client-sandbox-v1 converged.

CheckResult
Both tenant scriptsidentical etag b0a58070…, modified 12:58Z / 13:02Z
Bindings33 (production-env) / 34 (dev); SANDBOX_RUNTIME=legacy, SANDBOX_V1_CUTOVER=false
HealthHealthMonitor.check_instance_now/1 → :ok on both
v1 park/resume/backup (dev)200 / 200 / 200; recovery point advanced to ed7f0db5-… at 13:05Z
Born-v1 tenant-arua-devupgraded to unstable-d57bec6d…; deployed_version = serving_version, 35 bindings, SANDBOX_RUNTIME=v1, marker intact, recovery point 8c2729a2-…, 3/3 healthy
Staging Lermav1-sandbox-d57bec6d, healthy

Operational follow-up: the staging provisioning key was rotated (old key 401, new key 200; SECRETS.yaml and the GitHub STAGING_PROVISIONING_API_KEY secret updated). The born-v1 tenant-arua-dev was moved off the unreachable f1590509 merge and onto this build.

Gateway proxy attribution fix, validated live (2026-11-09)

Every dev instance on the v1 runtime served {"error":{"message":"Proxy client attribution is required…","type":"proxy_attribution_required"}} on the root HTML/WS proxy. Root cause: OpenClaw attributes proxy-shaped traffic (Forwarded / X-Forwarded-* / X-Real-IP) against gateway.trustedProxies before Gateway auth and rejects a source it cannot trust. The legacy runtime reaches the container from the sandbox bridge (10.1.0.x, healed into trustedProxies by moltlazy), but the v1 transport forwards through ctx.container.getTcpPort(port) from the SandboxV1 Durable Object, whose ingress address is platform-owned and is not a range a tenant config can list.

Fix (eddef5e0): SandboxV1.fetch() probes attribution once per gateway incarnation (GET / with the rebuilt forwarded headers) and, when the Gateway refuses it, strips every proxy-shaped header so OpenClaw attributes the request as direct-remote (always accepted, same token/device auth). The verdict is cached per DO lifetime, re-probed after v1KillGateway / v1Destroy / startGatewayOwner, and guarded by a generation counter.

Publishing for the live drill reused the running container image (unchanged by this commit): the unstable-d57bec6d… legacy + v1-unstable-d57bec6d… manifests were copied to unstable-eddef5e0… / v1-unstable-eddef5e0… in registry.cloudflare.com/35afea16440634aa2350331d2a736eec/fouria (identical digests), and the new Worker bundle + container-manifest.json were published to fouria-bundles/unstable-eddef5e0…/. Promotion: POST /api/provisioning/upgrade/arua {"version":"unstable-eddef5e0…"}.

CheckResult
Promotion convergencestatus: running, deployed_version = serving_version = unstable-eddef5e0…, container_health 3 healthy / 0 failed / 0 errors, rollout_active: false
Containernot replaced (same image digest): instance 0899aee8… running since 2026-11-08 — only the script changed
/api/health (service token)200 status: running, degraded: false, startup/readiness/liveness success, gateway pid 4 / 1.39 GiB / 23 threads
Live gateway attribution (new)tenant log Gateway refused proxy attribution; forwarding without forwarded client headers — the DO probe reproduced the rejection against the real Gateway
Proxied request through the fixGET /api/status/attribution-live-{1,2,3} (an allowed dispatch health path that falls through to the catch-all proxy) → [PROXY] Handling request + [HTTP] Response status: 404 (the Gateway’s answer for an unknown path), not 403 proxy_attribution_required
Post-cleanuptenant observability settings restored (persist: false, 10 % logs), temporary Access service token + its two app policies deleted, public host back to 302 (Access)

Validation tooling note: staging tenants are ROOT-only at the dispatch gate, so the public proxy path cannot be exercised with a service token. The reachable /api/status/* prefix is allowlisted as a health path and falls through to the gateway proxy, which made the end-to-end request observable; tenant Worker logs were read by briefly enabling persist (they normally go only to Lerma’s OTLP ingest) and the setting was restored immediately afterwards.

Preflight bundle verification (2026-11-09)

The known-gap “Preflight asymmetry” is closed. cfd2dda8 adds DispatchNamespace.get_script_content/2 and makes Instances.at_requested_version?/2 verify that the requested bundle’s main module is embedded in the deployed script (the content endpoint returns the whole uploaded script; the parsed main module is a verbatim substring). Both the promotion candidate filter and the error-retry preflight use it, so a failed script upload (image at target, bundle older) is retried; an unreadable bundle/content probe falls back to the image so a transient failure cannot loop. 685d686a makes the endpoint report the number of instances actually marked upgrading.

Deployment: branch Lerma v1-sandbox-685d686a (compose backup /opt/lerma/docker-compose.yml.bak-pre-685d686a), healthy. No Worker promotion was needed; staging keeps unstable-d57bec6d….

CheckResult
Deployed script content10,875,960 bytes; contains the parsed main module (2,855,594 bytes) and the d57bec6d commit marker
Re-promotion preflightPOST /api/provisioning/upgrade/client-sandbox-v1 (version unstable-d57bec6d…) → 202 {instance_count: 0}; both instances stayed running at d57bec6d; Lerma fetched the bundle once per instance and did not deploy
Negative path (bundle older)Suite: at_requested_version?/2 image/bundle matrix + controller retry test; live failure injection remains the next step

Born-v1 first-provision self-heal (2026-11-11)

51df6b0a closes the zero-instance first-provision failure recorded above (2026-11-08: the SandboxV1 application existed with zero instances, the DO reported running, containerFetch timed out and the gateway never booted; the only recovery was a manual deprovision + re-provision).

  • Containers.recreate_v1_application/1 deletes and recreates the v1 application against the same DO namespace (bounded create retry). The deployed script and its Durable Object state are untouched.
  • TenantProvisioner.verify_health/3 takes that action only for a fresh provision (reconcile_v1) with no committed recovery point whose v1 application reports no instances (or is missing), then retries the probe once. An upgrade never opts in and a tenant with a pointer is never reconciled.

Deployment: branch Lerma v1-sandbox-51df6b0a (compose backup /opt/lerma/docker-compose.yml.bak-pre-51df6b0a), healthy; no migrations. All five staging instances remain running (tenant-arua-dev on unstable-eddef5e0…, both client-sandbox-v1 tenants on unstable-d57bec6d…, tenant-client{,-dev} on unstable-73e820e6…).

CheckResult
Module loadedfunction_exported?(Containers, :recreate_v1_application, 1) → true; SandboxV1Status and TenantProvisioner loaded
TestsTargeted containers_test + tenant_provisioner_pipeline_test: 148 passed; full Lerma suite: 1404 passed / 4 excluded; mix format --check-formatted + mix credo --strict clean
Live wedge drillNot run — it needs a disposable born-v1 tenant on this image. tenant-arua-dev is ineligible: it has a recovery point, and the reconcile deliberately refuses those

Born-v1 self-heal: live drill attempt and the active fix (2026-11-11)

Live drill attempt on a disposable born-v1 tenant (bornv1-heal-7fb503, client 64, dev, unstable-eddef5e0…; deprovisioned and deleted afterwards). Two findings, one fixed and one still open.

1. Fixed (b0a22c02) — a durable_object application reports active, not healthy. The live health payload for a v1 application is {"instances": {"active": 1, "starting": 0}}; there is no healthy key. Containers.fetch_application_health/1 read healthy, so a live v1 tenant looked instance-less: the promotion preflight would re-deploy it on every promotion, and the new born-v1 reconcile would have recreated a running application. The v1 summary now maps active into healthy (falling back to healthy). Live proof: GET /api/provisioning/upgrade/arua reports container_health {runtime: v1, healthy: 1, rollout_active: false}, and re-promoting arua at its current version returns 202 {instance_count: 0} — before the fix the preflight retried it.

2. The zero-instance wedge could not be reproduced by deleting the app record. The v1 application was created at 00:32:25 and deleted at 00:32:29 (API 200, absent from the account list). The first health probe still started a container, the gateway booted, and the initial snapshot committed at 00:35:02. The app record is therefore not the only thing that gates a placement (or its deletion is not synchronous), so recreating only the application may not be sufficient for the original transient — the documented operator workaround also re-deployed the script. The async health task was also lost to an unrelated Lerma BEAM restart at 00:36:25 (the container did not restart; Startup health re-ran), so the reconcile path was never reached. The drill is inconclusive, not a failure of the path.

Cleanup: DELETE /api/instances/2537773c-… → 204, DELETE /api/clients/64 → 204, the tenant script and both container applications are absent, and the R2 bucket was emptied (5 objects) and deleted.

Next step: fault-inject a placement failure (or make the provisioner re-deploy the script when the application stays empty), since deleting the app record alone does not reproduce the wedge.

Develop integration sync, production-surface removal and the local gate (2026-11-12)

Merged origin/develop (e62f5a37, Agent Studio #572) into v1-sandbox at 305d489e. Only the two business-rules corpora conflicted; both were resolved additively (the activity/alarm/boot sections and the Agent Studio sections, each with its own count trailer). Code and docs auto-merged (agentStudioRoutes + activityRoutes, the Studio API client, and the Agent Studio sections in WORKER-DEVELOPMENT.md / TESTING.md).

Decision. ADR-SV1-010 records that the track integrates into develop as one reviewed merge commit and that integration is not a production cutover. The production wrangler.jsonc environment no longer declares the fouria-sandbox-v1 application or the SandboxV1 binding (019a22f1); the class and the append-only v7 migration remain. validate-runtime-image-pins now requires the v1 surface in local/staging and fails if production declares it (803831fa).

Local gate. On the merged tree (Elixir files identical to 305d489e):

CheckResult
bun run format:check, lint, typecheckpass
bun run test4254 passed, 8 skipped (209 files), no type errors
mix check, mix license_auditpass (3630 mods/funs, 62 deps)
mix workspace.run -t test1417 passed, 4 excluded
bun run plan:sandbox-v1:checkpass (166 features / 14 milestones)

mix precommit remains blocked by 20 pre-existing compiler warnings under --warnings-as-errors across Lerma/Fouria; none are introduced by this merge, and the mandatory gates above are green.

CI. On 803831fa the PR checks (unit, test, quality, build-and-push, changes, validate-secrets, Validation lane tests) passed and PR #565 is MERGEABLE/CLEAN. The manual Test workflow (workflow_dispatch, run_e2e=true, full_tests=true, production) was started as run 43199464508; its observation is appended when it completes.

Staging validation after the cec359fa promotion (2026-11-12)

  • Promotion: client-sandbox-v1 converged on unstable-cec359fa… at 10:55:30Z (2 instances); both tenant scripts share etag b7893a73… (tenant-client-sandbox-v1 modified 10:52:03, …-dev 10:55:19). The legacy production application serves fouria:unstable-cec359fa…; the dev instance serves through the v1 application (v1 active: 1).
  • v1 snapshot (PER-2): POST /_control/client-sandbox-v1-dev/backup → 200; the stable pointer advanced 7beb1684… (10:55:08Z) → 3b9030fa… (10:56:40Z).
  • SEC-3 park/resume: checkpoint-park → {parked: true, backupId: 7e4b11c1…}; a backup while parked → 409 TENANT_PARKED; resume → {resumed: true}; a post-resume backup → 200 and the pointer advanced to 7b6f95d5… (10:57:40Z). The cold boot served from the committed record.
  • Tenant isolation: migration-status for client-sandbox-v1-dev → 200 {state: "committed", recordId: "77cadb66…"}; for the production-environment client-sandbox-v1 → 200 {state: "absent"} (it is still legacy; its bucket contains only backups/*.sqsh). No Access token was used: the checks went through the dispatch control plane.
  • BIL-3 finding (cost attribution): the AE fourai_cost_metrics rows from the v1 snapshot/park are written with blob2 = "default", not client-sandbox-v1. The tenant script carries WORKER_URL=https://staging-client-sandbox-v1-dev.fouria.io but Lerma binds no CUSTOMER_SLUG/WORKER_NAME. tenantSlugFromEnv already falls back to WORKER_URL; deriveCustomerSlug (used by getCostContext) did not, so it returned default. Fixed in this PR: Lerma now binds the canonical base slug as CUSTOMER_SLUG on both fresh provision and upgrade; the cost resolver sanitizes the explicit binding with the shared slug allowlist, fails closed for a non-empty invalid value (never adopting WORKER_NAME/WORKER_URL), and never uses a URL as the identity; isValidSummaryCustomer accepts exactly the emitted domain so rows are never write-only. The next promotion re-deploys the scripts with the binding; the corrected AE rows are the live confirmation.
  • BIL-3 adversarial review (independent session): the fix was attacked before merge, and it found a real cross-tenant hole in the pre-existing guard: otel-ingest.ts skipped the tenant check whenever the expected identity was the default sentinel (expected !== 'default' && …), while the billable row’s customer came from the container-supplied customer.slug. An unidentified Worker could therefore mint billable rows for another tenant and choose which billing-freeze/<slug> gate applied. The guard now falls back to the routing resolver (WORKER_NAME/WORKER_URL) and answers 503 TENANT_UNRESOLVED on a billable instance with no identity. The same review closed the consistency gaps: Lerma reserves default and the tenant- dispatch prefix as client slugs, canonical_tenant_slug/1 strips exactly one affix (so tenant-tenant-foo no longer collapses to foo), the shared sanitizer rejects padded values instead of trimming them, the cost summary and emitter share one slug domain, out-of-range from/to values are clamped instead of throwing, and a loopback WORKER_URL is a local (non-billable) instance. Covered by sandbox-otel-bridge.test.ts, metrics.test.ts, cost-summary.test.ts, dev-mode.test.ts, sandbox-tenant.test.ts, and the Lerma client/naming tests.
  • BIL-3 live confirmation (2026-11-16). Staging Lerma was redeployed from the branch image ghcr.io/0xcab0/lerma:v1-sandbox-69e2d9cc (compose backup /opt/lerma/docker-compose.yml.bak-pre-69e2d9cc) and both client-sandbox-v1 scripts were re-uploaded at their existing bundle (unstable-cec359fa…) via Instances.upgrade_instance/3. Both scripts now carry CUSTOMER_SLUG as a plain_text binding with the value client-sandbox-v1 (35 bindings on the dev script, 34 on the production-env script; previously absent). The dev instance is running and the legacy production-env instance is sleeping; last_deployed_at advanced to 14:12:14Z / 14:14:41Z. Analytics Engine (fourai_cost_metrics_staging) rows emitted after the flip are attributed to blob2 = "client-sandbox-v1" for health-check, gateway-memory-sample and gateway-resources; the same row types were blob2 = "default" before the redeploy. The remaining default rows belong to the other staging tenants (client, arua), which have not been upgraded yet and pick up the binding on their next upgrade.
  • Cloud E2E harness repaired (2026-11-13). A fresh wrangler deploy failed for hours with “Durable Object-managed Container application settings were not fully applied” after all five retries. Root cause: the top-level (local/E2E) wrangler.jsonc still declares the SandboxV1 application as the shared fouria-sandbox-v1, which belongs to the production Worker, so every fresh E2E run tried to adopt it. The per-run rename is restored (<worker>-sandbox-v1), and teardown now validates the teardown token (a stale E2E_CLOUDFLARE_API_TOKEN answered 9109 and made every container-app delete silently no-op; three leaked apps were still billing and were deleted manually). With the per-group transcripts finally uploaded, the four legacy failures were triaged and fixed: h_cli_knowledge (the roster read raced the plugin reload → bounded retry in get-agent-id), y_production_profile (no scheduled snapshot because BACKUP_CHANGE_WATCH is off in E2E → on-demand POST /debug/backup/sync), zz_gateway_hard_restart (/status/polling’s RPC fast path omits processId → read the authoritative /api/health snapshot, plus a post-restore boot wait; the snapshot now exposes the /proc pid when the tracker misses a restore-relaunched gateway), and zzz_software_provisioning (missing CUSTOMER_SLUG → fixture binding, now 9/9). Re-runs of every previously failing suite are green.
  • Not run (Access-free limitation): a v1 freeze would leave the dev tenant stopped (unfreeze is a no-op and there is no Access-free wake); the snapshot-before-destroy path is corpus-covered, and the safe SEC-3 park/resume path above proves the cold boot from the committed record. Tenant-HTTP drills (channels/RPC, browser) remain Cloudflare-Access-blocked.