Sandbox v1 migration report and QA guide
On this page
What this is. The PR-visible record of what the
v1-sandboxtrack changed, what serving a tenant on the Sandbox SDK 1.0 (SandboxV1) runtime is expected to improve over the legacySandboxruntime, and how to QA it. Written for reviewers of PR #565 and for the operator running the next staging drill.What this is not. Not a normative contract, not a production cutover plan, not a durability SLA. The invariants are C1–C10 in
sandbox-v1-cutover-contracts.md; the per-capability plan issandbox-v1-evaluation-plan.json.
Status
| Property | Value |
|---|---|
| Track / branch | v1-sandbox — integration source for develop (ADR-SV1-010); merges as one reviewed PR. The old sandbox-v1-validation lane stays frozen |
| Review surface | PR #565, base develop |
| Contract / decisions | sandbox-v1-cutover-contracts.md (C1–C10), sandbox-v1-decisions.md (ADR-SV1-001…009) |
| Evaluation plan | sandbox-v1-evaluation-plan.json + .md — 166 features / 14 milestones |
| Evidence record | sandbox-v1-staging-observations.md (append-only) |
| Staging target | Lerma client client-sandbox-v1 (id 20): tenant-client-sandbox-v1-dev, tenant-client-sandbox-v1 |
| Staging hosts | staging-client-sandbox-v1-dev.fouria.io, staging-client-sandbox-v1.fouria.io |
| Last published bundle | unstable-82bc961e95022541bd5abc688293c5e5219b14a9 (fouria:unstable-… + fouria:v1-unstable-…) |
| Staging Lerma image | ghcr.io/0xcab0/lerma:v1-sandbox-82bc961e |
| HEAD at writing | 82bc961e (docs: ENV-VARIABLES reformat on top of 35e37628) — deployed and promoted to staging |
| Production cutover | None. No production tenant is converted; WORKER_PLATFORM is absent in production, so the gate refuses there |
Reading rules. Every claim is labelled: [live] observed on staging and recorded in the observations doc; [suite] proven by unit/integration/corpus tests only; [expect] expected/directional and not measured — a hypothesis, not a result.
Executive summary
- A converted staging tenant serves on
SandboxV1: HTTP/WS proxy through the DO, real/healthreadiness, non-waking passive probes, one gateway owner,503 SANDBOX_V1_SLEEPINGinstead of a wake.[live] - Conversion is a durable one-way data migration: write-ahead intent → per-tenant R2 lease → legacy recovery point → squashfs extract → v1
DirectoryBackup+ catalog +stable.json, with in-place path relocation (/root/openclaw→/home/openclaw).[live] - v1 persistence is its own: records under
sandbox-migration/<tenant>/records/, F4E1 ciphertext at rest, pointer-based cold boot, retention (default 3), restore-by-id. Restore-by-id is[suite]; its live drill is blocked by Access. - Zero-loss rollouts exist on v1:
checkpoint-park→ replacement →resume, with409 TENANT_PARKEDwrite blocking, session drain to read-only, and durable mode restore. A promotion-driven park → rollout → resume was observed end to end.[live] - An idle snapshot cadence bounds the loss window that SDK 1.0’s missing eviction hook otherwise leaves open.
[live]— atrigger: "backup"record was committed with no operator action (4437c278-…,2026-11-07T21:18:55Z). - Provisioning is real on staging: Lerma creates the
SandboxV1DO binding, a…-sandbox-v1-<platform>application withscheduling_policy: "durable_object", and a digest-pinned named v1 image.[live] - A tenant can be provisioned directly on v1 (“born v1”): Lerma writes the create-only runtime marker
sandbox-migration/<slug>/runtime.jsonbefore the script deploy, deploysSANDBOX_RUNTIME=v1+SANDBOX_V1_CUTOVER=true, and the Worker resolves the runtime from the marker (no conversion record).migrateanswers409 MIGRATION_ALREADY_V1, the fresh container onboards in v1, and the first recovery point is committed seconds after readiness.[live] - The promotion gate no longer lies: it converges on
serving_version, refuses success while any instance iserror, and retries error instances.[live] - An operator can run the conversion from Lerma with a preflight that tells the truth:
POST /api/provisioning/migrate/:slug?dry_run=1/mix lerma.sandbox_v1.migrate, includingalready_converted+ the v1 record id for a tenant that has already been converted.[live] - The outbound virtual hosts (
otel.internal,*.r2.internal,cdp.internal) are registered on the v1 container, so billing/observability, the R2 bridges and the agent browser survive the switch. Registration[live]; runtime effect unobserved. - Not verified live: restore-by-id via
/api/admin/*, the cadence tick, corrupt-state-DB repair, the OCM-3 verify sweep. No production cutover, no performance or cost claim. Integration intodevelopis not a cutover (ADR-SV1-010):WORKER_PLATFORM === "staging"remains the only migration authority, the reservedclientnever migrates, and Lerma keeps v1 provisioning staging/dev-only.
Scope and architecture
Two runtimes side by side
| Concern | Legacy (Sandbox) | v1 (SandboxV1) |
|---|---|---|
| DO class / package | Sandbox; @cloudflare/sandbox-legacy@0.13.0-next.776.1 aliased to @cloudflare/sandbox | SandboxV1; @cloudflare/sandbox@1.0.3 |
| Container application | tenant-<slug>-sandbox-<platform>, scheduling_policy: "default", image at application level | tenant-<slug>-sandbox-v1-<platform>, scheduling_policy: "durable_object", digest-pinned named image (ctx.container.images.sandbox) |
| Image | apps/fouria/Dockerfile | apps/fouria/Dockerfile.v1 (node:24-trixie-slim + sandbox-shim from cloudflare/sandbox:1.0.3) |
| Snapshot primitive | sandbox.createBackup() → squashfs .sqsh | DirectoryBackup of /home/openclaw (tar+zstd/1) |
| Catalog / pointer | backups/<id>/meta.json, backup-handle.json, stable-backup.json | sandbox-migration/<tenant>/{records/<id>.json,conversion.json,stable.json} |
| Encryption at rest | F4E1, backups/<id>/data.sqsh.enc | F4E1, sandbox-v1/<disc>/<id>.tar.zst.enc + <id>.meta.json |
| Restore | restoreBackup() + legacy root materialization | DirectoryBackup.restore(record, { dir }) — SDK atomic swap |
| Idle stop | onActivityExpired hook → archive → stop | no eviction hook: DO alarm cadence + platform inactivity stop |
| Control actions | full set (backup, checkpoint, checkpoint-park, freeze, unfreeze, resume, migrate) | backup/checkpoint/checkpoint-park/freeze/resume/migrate; unfreeze no-op; others 503 SANDBOX_V1_CONTROL_UNSUPPORTED |
| Outbound hosts | class statics (TenantSandbox.outboundByHost) | ctx.container.interceptOutboundHttp(...) per instance, re-registered on every start |
| Runtime selection | default | durable intent in sandbox-migration/<tenant>/intent.json |
Both runtimes share one host map (src/sandbox-outbound.ts) and one acquisition boundary (acquireSandbox, src/sandbox-contract.ts), so serving cannot drift.
Gate, intent and runtime resolution
- Authority (C1).
canMigrate(env)is true only for the trustedWORKER_PLATFORM === "staging"binding. Absent, empty, whitespace,production,dev,local,STAGING,Host: staging-<slug>andX-FourIA-Role: rootall refuse — in/controlmiddleware, before the sandbox stub is acquired, so no R2 write and no DO/container addressing happens. - Identity (C10).
migrationRefusal(env)resolves the canonical tenant viatenantSlugFromEnv(neverctx.containerId, never the context fallback) and refusesnull/defaultand the reservedclientslug. - Intent (C5, C6). The intent is written before the first converting step. Reads are tri-state:
absent(explicit 404) permits legacy;presentandunknownmean v1, permanently. No automatic legacy fallback on routing, serving, gateway start, freeze, backup, checkpoint or abort. - Reservation (C2).
migrateonclientis refused; every other control action keeps its pre-contract behaviour there.
Request and boot flow
Dispatch (auth, tenant, X-FourIA-*) → tenant Worker: resolveRuntimeIntent() # tri-state
→ acquireSandbox(env, id, options, runtime)
legacy → getSandbox() stub
v1 → V1SandboxClient over the SandboxV1 DO
→ v1 HTTP/WS: stub.fetch(..., X-FourIA-Sandbox-Port) → DO rewrites to
http://localhost:<port> → ctx.container.getTcpPort(port).fetch(...)
→ v1 exec: v1Exec / v1ProcessOutput / v1ProcessWait / v1ProcessSignal /
v1ProcessState / v1ListProcesses # ExecProcess handles cannot cross RPC
→ boot: ensureGateway → v1EnsureGateway → adopt healthy | restore pointer record
(OPENCLAW_RESTORED=true) | start-openclaw.sh start (single owner)
The tenant Worker never drives the v1 container directly: it calls the DO entrypoint (the entrypoint, not an RPC method, is what carries a WebSocket upgrade back), and the DO owns ctx.container.
v1 object layout in R2
| Path | Purpose |
|---|---|
sandbox-migration/<tenant>/intent.json | Write-ahead intent — the point of no return |
sandbox-migration/<tenant>/conversion.json | Conversion record (fallback catalog entry) |
sandbox-migration/<tenant>/records/<id>.json | Catalogued record (size, sha256, format, trigger) |
sandbox-migration/<tenant>/stable.json | Recovery point pointer |
sandbox-v1/<disc>/<id>.tar.zst.enc + .meta.json | Ciphertext + sidecar (encrypted: true, plaintext size/sha) |
Changes performed
Counts and dispositions are from the evaluation plan (features[].disposition); commits are those on origin/develop..HEAD at 806a2580.
M0 — control plane, lane and tooling
| Aspect | Content |
|---|---|
| Goal | One written definition, one machine-readable plan, one staging lane. |
| Changed | docs/plans/sandbox-v1-{cutover-contracts,decisions,scaffold,staging-observations,evaluation-plan}.{md,json} + schema; scripts/generate-sandbox-v1-evaluation-plan.mjs, scripts/validate-sandbox-v1-plan.mjs, scripts/sandbox-v1-plan.test.ts, scripts/sync-v1-sandbox.sh; apps/fouria/Dockerfile.v1; wrangler.jsonc (SandboxV1 binding + fouria-sandbox-v1{,-staging} apps, migration v7); P5b scripts/{release-manifest,release-registry-digest,validate-runtime-image-pins}.mjs + tests. |
| Behaviour | bun run plan:sandbox-v1[:check|:sync]; the restored v1 deploy surface is a deliberate reversal of audit finding F9 for this track only (the v1 image must be publishable), with no route selecting it. |
| Commits | 8436b4fe, 0f563b20, a90015f5. |
S1 / SEC-1 — authority, identity and platform gate (19: 16 preserve, 3 verify)
| Aspect | Content |
|---|---|
| Goal | Make an irreversible migration impossible to trigger from an untrusted signal. |
| Changed | src/sandbox-migration/platform.ts (canMigrate, migrationRefusal), src/sandbox-tenant.ts (default rejected in every branch), src/index.ts (/control middleware gate), staging vars; Lerma TenantProvisioner.required_var_bindings/2 + Instances.instance_bindings/2 add WORKER_PLATFORM, SANDBOX_V1_CUTOVER=false. |
| Behaviour | 403 MIGRATION_PLATFORM_FORBIDDEN, 409 MIGRATION_SLUG_UNRESOLVED, 409 MIGRATION_RESERVED_CLIENT; an allowed staging tenant stopped at 501 MIGRATION_NOT_IMPLEMENTED until S2. Every refusal precedes any write or addressing. Gate is migrate-only. |
| Tests / docs | sandbox-migration/platform.test.ts, sandbox-tenant.test.ts, src/index.test.ts, corpus business-rules/sandbox-v1-cutover.txt, tenant_provisioner_pipeline_test.exs; ENV-VARIABLES.md, OPERATOR-GUIDE.md §2.6. |
| Commits | 99a41a6f, provisioning in 8ce99552. |
S2 / PER-1 — conversion primitives and DO-owned conversion (12: 11 rewrite, 1 verify)
| Aspect | Content |
|---|---|
| Goal | Move the tenant’s data into a v1 record, once, with no replayable double-run. |
| Changed | New src/sandbox-migration/: state.ts (tri-state), intent.ts (write-ahead), lease.ts (create-only acquire, etag-CAS takeover, owner-scoped renew/release, fail-closed), conversion.ts (extract/materialize/rewrite), intent-gate.ts; src/sandbox-v1.ts (convertToV1, verifyConversionRecord); Dockerfile.v1 installs squashfs-tools. |
| Behaviour | Lease TTL is a liveness bound, never an RPO claim. A snapshot without a complete stage-manifest.json fails closed. The legacy archive is not a copy of /home/openclaw (staged roots + symlinks), so conversion extracts real directories, copies home-root workspaces, and rewrites /root/openclaw → /home/openclaw in place (length-preserving, node:sqlite quick_check, zero references) and /root/clawd → /home/openclaw/clawd in text files. Upload uses stdin: 'pipe' and closes the pipe before awaiting the exit code (a ReadableStream stdin silently truncated the archive). A post-commit launch failure logs sandbox.v1.conversion.launch_failed and is retried by the next v1 boot — it never retracts the commit. 503 SANDBOX_V1_{CONVERSION,RUNTIME}_PENDING until the C8 barrier completes. |
| Tests / docs | sandbox-migration/{state,intent,lease,conversion,intent-gate}.test.ts, src/index.test.ts, gateway/process.test.ts, corpus sandbox-v1-persistence.txt; sandbox-v1-scaffold.md. |
| Commits | 486b2938, 559cc0a0, e949d961, 15d8e894, e7d193fd. |
S3 / OCM-1 — serving parity (13: 4 rewrite, 9 verify)
| Aspect | Content |
|---|---|
| Goal | Serve a converted tenant through the DO with legacy HTTP, WS, health and CLI semantics. |
| Changed | src/sandbox-v1-client.ts (V1SandboxClient, process RPC shim, killGateway, destroy, unsupported-op errors), src/sandbox-contract.ts (selection, fail-closed unknown), src/index.ts (v1 acquisition/proxy), src/gateway/process.ts (ensureV1Gateway), src/routes/public.ts (health parity). |
| Behaviour | stub.fetch carries HTTP and WebSocket upgrades; a stopped container answers 503 SANDBOX_V1_SLEEPING (a probe cannot wake it); v1Status() is the non-waking ctx.container.running read, readiness is a real /health, 503 until ready, /healthz stays sub-second; unsupported legacy ops throw SANDBOX_V1_UNSUPPORTED_OPERATION; ensureGateway delegates, so the DO is the single owner across isolates. |
| Tests / docs | sandbox-v1-client.test.ts, sandbox-v1.test.ts, health-probes.test.ts, process.test.ts, sandbox-contract.test.ts, index.test.ts, corpus sandbox-v1-serving.txt; WORKER-DEVELOPMENT.md §S3. |
| Commits | 317012c4, 41be4409. |
S5 / PER-3 — conversion orchestration and durable lanes (3: 3 rewrite)
| Aspect | Content |
|---|---|
| Goal | Make migrate a real, idempotent, durable-external control action. |
| Changed | src/index.ts (/control migrate, queue-only), src/sandbox-v1.ts (v1RequestConversion, v1ConversionState, alarm() → runConversionAlarm()/runSnapshotCadence(), ensureConversionArmed, #conversionRun single owner), src/sandbox-v1-client.ts (queue RPC surface; no synchronous conversion), workers/fouria-dispatch/src/index.ts (CONTROL_ACTIONS + forwarding + CPU budget), src/activity/scheduler.ts (runtime resolution before acquisition). |
| Behaviour | migrate is queue-only: POST /control enqueues the durable DO job and returns 202 {state: queued|running} in milliseconds (200 {state: committed, recordId}, 502 {state: failed, code: SANDBOX_V1_CONVERSION_FAILED}, 503 SANDBOX_V1_UNAVAILABLE). The conversion runs on the DO alarm — conversion first, cadence second, each wrapped so neither starves the other — because the idle-snapshot cadence alarm proved alarms fire on this lane (record 4437c278-…). Repeats are idempotent (committed via DO state or the R2 record, in-flight queued/running, new work only from absent/failed, attempt increments). A lost alarm self-heals on the read paths (ensureConversionArmed), arming never postpones an earlier alarm, #conversionRun folds overlapping turns, and committed is durable before the marker drop and the launch, so a launch failure never retracts the commit. The DO rebuilds the container env from its own bindings — no secrets in DO storage. |
| Tests / docs | sandbox-v1.test.ts, index.test.ts, workers/fouria-dispatch/test/unit/control.test.ts, corpus sandbox-v1-cutover.txt; WORKER-DEVELOPMENT.md §S5. |
| Commits | e910b967, 0a47e362, fdb1f639, be255fdd (durable DO job + v1ConvertNow removal), 92a21080 (migration-status read-only action). |
S4 / PER-2 — owner ops, records, recovery point, retention (6: 4 rewrite, 1 verify, 1 preserve)
| Aspect | Content |
|---|---|
| Goal | Give v1 the legacy persistence guarantees on the record format, without legacy archive orchestration. |
| Changed | src/sandbox-migration/records.ts (catalog, pointer, verifyRecoveryPoint), src/sandbox-v1.ts (v1CreateSnapshot, v1RestoreRecord, v1Destroy, cadence), src/index.ts (/control routing), src/routes/api/storage.ts (restore-by-id branch). |
| Behaviour | Snapshot → post-encrypt in place (fail closed on BACKUP_ENCRYPTION_REQUIRED with no key) → catalog immediately → advance stable.json → prune oldest-first beyond BACKUP_MAX_VERSIONS (default 3), newest protected. freeze snapshots then destroys; a failed snapshot keeps the tenant running (SANDBOX_V1_FREEZE_SNAPSHOT_FAILED); unfreeze is a no-op. Cold boot restores what the pointer names (falling back to conversion.json) and refuses a conversion record with no pointer. verifyRecoveryPoint is the single C8 read: pointer-absent/record-absent/record-unknown/pointer-record-mismatch/complete. Restore-by-id opens the maintenance window first, stops the single owner, waits ≤5 s for /health to fail (avoids EBUSY), applies the record, promotes by default (?setCurrent=0 applies without promoting), 404 on unknown without touching the container. Cadence: startContainer() arms a durable marker + setAlarm; alarm() runs conversion then cadence; never wakes a stopped container, skips while #restoring, skips when the pointer is newer than the interval; resolveSnapshotIntervalMs = SANDBOX_V1_SNAPSHOT_INTERVAL_MS → BACKUP_INTERVAL_MINUTES → 1 h, capped at half the inactivity timeout (staging 7.5 m, dev 1 h); pruning ties on createdAt break by record id. |
| Tests / docs | records.test.ts, sandbox-v1.test.ts, routes/api/storage.test.ts, gateway/process.test.ts, corpus sandbox-v1-persistence.txt; WORKER-DEVELOPMENT.md §cadence/§snapshots/§restore-by-id, ENV-VARIABLES.md. |
| Commits | 6cd23099, 6795827b, 473c46b0, 6de3b34d, 9952203a, 855810b9. |
OCM-1 / BIL-1 — outbound virtual hosts on v1
| Aspect | Content |
|---|---|
| Goal | Restore the Worker-owned services the container reaches by name, which SDK 1.0 no longer provides as class statics. |
| Changed | src/sandbox-outbound.ts (SANDBOX_OUTBOUND_ROUTES, one map for both runtimes); src/sandbox-v1.ts startContainer() registers ctx.container.interceptOutboundHttp(host, ctx.exports.SandboxOutboundGateway({ props: { host } })) per host. cdp.internal (Browser Rendering: the container’s local CDP proxy → the tenant Sandbox DO, credential never in the container) is part of the same map; the develop merge that dropped it was repaired in the follow-up fix below. |
| Behaviour | Intercepts belong to the container instance, so they are re-registered on every start (including a restarted DO that finds its container running). A bundle without the export warns and keeps direct egress instead of failing the boot. An unknown host answers a structured 404; an escaping exception would surface as ECONNRESET. |
| Tests / docs | sandbox-outbound.test.ts (host set incl. cdp.internal), sandbox-v1.test.ts (one intercept per host incl. cdp.internal, re-registration, missing export warns), corpus sandbox-v1-serving.txt; WORKER-DEVELOPMENT.md §outbound. |
| Commit | fd05f0c3; cdp.internal re-registered by the follow-up fix fix(fouria): register the cdp.internal bridge on the shared outbound host map (PR #565). |
OCM-2 — durable lanes, boot guards and repair (7: 5 rewrite, 2 verify)
| Aspect | Content |
|---|---|
| Goal | Paths outside a request resolve the right runtime; a failed launch fails fast and heals. |
| Changed | src/activity/scheduler.ts (async sandbox() → resolveRuntimeIntent → acquireSandbox(..., runtime); v1 snapshot jobs; v1 restore/deprovision reported skipped), src/sandbox-v1.ts (startGatewayOwner races launch exit against the /health poll). |
| Behaviour | pending/unknown tenants are skipped (activity.sandbox.skipped), never guessed — before this, an alarm wake or scheduled snapshot on a converted tenant booted the legacy container and advanced the legacy pointer. A non-zero exit before the port opens is terminal in seconds (a zero exit is the lost-lock race); the launch log is tailed and a SQLite corruption signature queues doctor-repair (gateway-launch-state-database), which re-queues the boot after a successful repair. |
| Tests / docs | activity/scheduler.test.ts, sandbox-v1.test.ts, corpus sandbox-v1-cutover.txt; WORKER-DEVELOPMENT.md §boot/owner/repair. |
| Commit | 889ec3a2. |
SEC-3 — rollout park and resume (6: 3 rewrite, 3 verify)
| Aspect | Content |
|---|---|
| Goal | A container replacement must not lose writes made after the last archive. |
| Changed | src/index.ts (checkpoint-park, resume), src/tenant-park.ts (guards), gateway/session-drain.ts, gateway/session-mode-restore.ts, src/sandbox-v1.ts (fail-closed park snapshot, v1KillGateway), dispatch forwarding/echo fields; Lerma Cloudflare.Containers.checkpoint / resume, @upgrade_pipeline, upgrade_instance/3. |
| Behaviour | Order is deliberate: drain sessions to read-only (bounded; an outliving turn reports sessionsDrained: false and the park proceeds) → commit the recovery point fail-closed (502 SANDBOX_V1_PARK_SNAPSHOT_FAILED, so no rollout starts without a restorable copy) → only then open the park and arm the exec lock. Writes are blocked by tenantParkGuard (409 TENANT_PARKED), by /control refusing every action except resume/checkpoint-park, and by assertTenantExecAllowed before any launch RPC; reads stay available. resume clears the park, boots through the DO, and applies recorded session modes after a real /health (or reports sessionModes: "pending"); resuming an unparked tenant never wakes it. Telegram users get the parked notice from the ingress and the update is acked. killGateway stays process-scoped; nothing destroys the container. |
| Tests / docs | tenant-park.test.ts, maintenance.test.ts, gateway/sandbox-exec.test.ts, session-drain.test.ts, session-mode-restore.test.ts, telegram-ingress/*.test.ts, index.test.ts, control.test.ts, Lerma containers_test.exs, instances_upgrade_test.exs; WORKER-DEVELOPMENT.md §park, OPERATOR-GUIDE.md §2.5, apps/lerma/README.md. |
| Commits | 548cb408, fdb1f639, e5dd5278, 1bf82623, 4389a6e4. |
OCM-4 — provisioning, rollout, promotion and release (27: 3 rewrite, 13 verify, 11 preserve)
| Aspect | Content |
|---|---|
| Goal | Let Lerma provision and promote v1, and stop a promotion claiming success it cannot prove. |
| Changed | apps/lerma/lib/lerma/cloudflare/sandbox_v1.ex (single selection point: enabled?/0 staging-only, image_enabled?/1 = unstable-* with a matching v1-unstable-*, v1_image/1, binding/0, container_config/3, drop_binding/1), Containers.deploy_script_and_container/1, DispatchNamespace (class migration from declared entries; preserve_do_namespaces also covers the script), Instances.upgrade_instance/3, json_views.ex (serving_version), .github/scripts/promote-validation.sh. |
| Behaviour | A durable_object application takes no application-level image; the image is a digest-pinned named image read by the DO (images.sandbox). An app body that still carried configuration.image was rejected after the script upload and the atomic-deploy rollback deleted the live tenant’s script — hence rollback now preserves the script and DO namespaces. Upgrade bindings are self-sufficient (gateway token, derived CDP secret, AI Gateway/API token, Access AUD, agent/scope flags, Telegram queue, change-detection vars, DEBUG_ROUTES). The gate reads serving_version (live image) not deployed_version (set before the rollout), refuses convergence while any instance is error, and retries error instances at an older version. |
| Tests / docs | cloudflare/{sandbox_v1,dispatch_namespace,containers}_test.exs, instances_upgrade_test.exs, tenant_provisioner_pipeline_test.exs, provisioning_controller_test.exs, scripts/staging-validation.test.ts; apps/lerma/README.md, .github/workflows/README-staging-validation.md. |
| Commits | 8ce99552, a160d39c, d0f8e50e, 19ca61d4, 67edc22e, 02327309, 8fa9e733, 1bf82623, 4389a6e4. |
ROOT debug tools on v1 (audit: 12 ✅, 1 ⚠️ partial, 1 ❌ unsupported)
| Aspect | Content |
|---|---|
| Goal | The operator escape hatch must survive a conversion. /debug/* (DEBUG_ROUTES=true, ROOT) and the dashboard’s “Debug Tools” run against whatever client the request acquired — after conversion that is V1SandboxClient, not the legacy Sandbox, so every route had to be checked against the v1 surface instead of assuming createBackup/logs()/stop(). |
| Changed | apps/fouria/src/routes/debug.ts: a shared readFinishedProcessLogs() (legacy logs() → v1 output() fallback) for /processes?logs=true and /logs?id=; the gateway log routes tail /tmp/gateway-stdout.log and return path + runtime; a v1 branch in /restart-openclaw (maintenance window → killGateway → cache/start-dedupe reset, no legacy checkpoint or restore-needed); /reprovision checkpoints with v1CreateSnapshot and validates restorability with verifyRecoveryPoint (C8) instead of backups/<id>/meta.json; /backup/:mode snapshots through the DO (no gateway boot); /destroy-container also clears the startup cache + isolate start dedupe; /sleep answers 503 SANDBOX_V1_SLEEP_UNSUPPORTED; /env reports sandbox_runtime_actual. |
| Behaviour | One refusal and one caveat, everything else works. POST /debug/sleep: v1 has no stop primitive (sleeping is the platform inactivity timer), so it returns a structured 503 rather than a 202 for work that never runs. POST /debug/rescue-container: diagnosis/destroy work, but v1 containerFetch never wakes a stopped container, so a sleeping tenant reads as unreachable. Process ids are per DO incarnation (empty list after an eviction; use /debug/gateway-logs), and a live process still has no readable log stream (file tail only). |
| Tests / docs | apps/fouria/src/routes/debug.test.ts (Sandbox v1 (converted tenant) debug routes: 15 cases driving a __fouriaSandboxV1-branded double); corpus test/business-rules/sandbox-v1-serving.txt; src/client/pages/__tests__/DiagnosticDashboard.test.tsx (the restart card renders only the route’s real fields, including runtime); compatibility matrix in apps/fouria/docs/WORKER-DEVELOPMENT.md § “Debug tools on Sandbox v1 (ROOT diagnostics)”; apps/fouria/AGENTS.md. |
| Commits | this PR. |
Operator migration path (Lerma)
| Aspect | Content |
|---|---|
| Goal | Make a conversion something an operator runs, with a preflight that tells the truth about this tenant, instead of a hand-rolled curl against /control. |
| Changed | Lerma.Cloudflare.Containers.migrate/2 (enqueues the migrate control action with a short 60 s receive budget — :migrate_enqueue_timeout_ms — normalises {state, record_id, duration_ms}, maps the Worker’s gate codes and logs state + duration without the control bearer) and Containers.migration_status/2 (a read-only call of the Worker’s migration-status action — it can never enqueue — normalising absent / queued / running / committed / failed; Containers.migrate_wait_timeout_ms/0is the operator poll budget, 30 min by default);Lerma.Cloudflare.SandboxV1Migration.preflight/2(eligibility + warnings from the instance rows plus two metadata-only R2 reads);Containers.legacy_recovery_point?/1andContainers.v1_recovery_point/1(+R2.object_exists?/2, R2.object_body/2); POST /api/provisioning/migrate/:slug; mix lerma.sandbox_v1.migrate. |
| Entry points | POST /api/provisioning/migrate/:slug (same provisioning auth as the upgrade API: Authorization: Bearer <key> + X-FourIA-Application: fouria) — ?dry_run=1 (or a body dry_run) answers the eligibility report; a real run enqueues. GET /api/provisioning/migrate/:slug reads state through the read-only migration-status action (same auth/scope, same ?instance=<cf_worker_name> selector) — only the POST can start a conversion. mix lerma.sandbox_v1.migrate --client <slug> [--instance <name>] [--dry-run] [--yes] [--wait SECONDS] [--poll-interval SECONDS] prints the same table, requires --yes off a TTY, then polls (default 1800 s / 10 s, --wait 0 = enqueue only) and exits 0 (eligible/committed), 1 (refused/declined/enqueue-only/timeout) or 2 (failed). The Mix task needs the Lerma source checkout (mix is not in the release image), so the REST endpoints are the operator lever against the deployed instance. |
| Eligibility | Blocking reasons: platform_not_staging (403 outside the staging platform — the same WORKER_PLATFORM === "staging" authority the Worker enforces), reserved_client (409, the reserved client slug never migrates, contract C2), production_environment (409; only a client’s non-production (dev) tenant is eligible — the rehearsal rule, no existing customer is the first conversion), instance_not_active, no_active_instance, instance_not_found, no_legacy_recovery_point (nothing to restore) and already_converted. Warnings never block: legacy_recovery_point_unknown, checkpoint_recommended_before_conversion, already_converted. |
| Already converted | A conversion leaves the legacy archive in R2 (it is the source), so the legacy recovery point reports true forever: the decisive signal is the v1 stable pointer sandbox-migration/<tenant>/stable.json in that instance’s own backup bucket. The dry-run therefore reports already_converted (true / false / nil = unreadable) and v1_record_id per instance and at the top level, refuses with reason already_converted, and a real run answers 200 {state: "committed", already: true, record_id} without a control call. When the pointer is unreadable (nil) the idempotent control action is still called — the Worker is the authority, and a conversion is a no-op when already committed. |
| States | 202 queued/running (the durable job was enqueued / is in flight — poll the GET), 200 committed (record_id, duration_ms; already: true for the no-op), 200 absent on the GET when no job is recorded (a truthful read, not an error), 409 ineligible, 403 outside the staging platform, 502 failed (or state: "unknown" for a control-plane error), 503 unavailable when the tenant Worker has no SandboxV1 binding. The gate is migrate-only: freeze/unfreeze/backup/checkpoint/checkpoint-park/resume keep their behaviour, including on client. |
| One-way boundary | Nothing here offers a rollback. The intent (sandbox-migration/<tenant>/intent.json) is the authority, so SANDBOX_V1_CUTOVER=false does not move a converted tenant back; the levers are restoring a catalogued v1 record or an explicit rebuild, both audited. Because the trigger only enqueues, the old 524/idempotency caveat no longer applies to POST: the job is durable in the tenant’s Durable Object, so poll until terminal — the read-only migration-status action can never start work, and reports absent for a tenant with no job — or re-run the preflight (which reports already_converted + the record id once committed), and never retry with a different target. |
| Tests / docs | apps/lerma/test/lerma/cloudflare/{containers_test.exs,sandbox_v1_migration_test.exs}, apps/lerma/test/lerma_web/controllers/api/provisioning_controller_test.exs, workers/fouria-dispatch/test/unit/control.test.ts; apps/lerma/README.md (“Migrating a legacy (v0) tenant to Sandbox v1”), workers/fouria-dispatch/README.md. |
| Commits | 804c512e (endpoint + Mix task + eligibility), 6602f1fc (already_converted / v1_recovery_point), a3b3e162 (enqueue + poll; its commit message reads feat(fouria): … although the change is Lerma’s), 92a21080 (read-only migration-status). |
Provisioning directly on Sandbox v1 (born v1)
| Aspect | Content |
|---|---|
| Goal | Let a new tenant start on the Sandbox SDK 1.0 runtime instead of converting it after the fact. Conversion is a one-way data migration with a long downtime; a fresh tenant has no legacy state, so the cheapest correct path is to choose the runtime at provisioning and never create legacy state at all. |
| Runtime choice | instances.sandbox_runtime (legacy default | v1) is chosen once, per environment, when the instance is created (sandbox_runtimes[env] in the client page, POST /api/clients/:id/instances, or the provisioning API). It is immutable after creation: changing it is a migration, not an edit. v1 is refused for a production environment and off the staging platform (422 sandbox_runtime_v1_production / sandbox_runtime_v1_not_staging), mirroring the worker-side gate — no production cutover exists. |
| Durable marker | Lerma writes sandbox-migration/<canonical-slug>/runtime.json ({version: 1, tenant, runtime: "v1", mode: "fresh", provisionedAt}) as a hard provisioning step immediately before :dispatch_deploy. The write is create-only: a matching marker is idempotent (:existing), an existing conversion intent refuses the run (:already_converted), a marker naming another runtime is an error, and a rollback deletes it only when that run created it. The bucket keeps the environment axis (fouria-backup-<slug>-dev-<platform>) while the key uses the canonical tenant slug, exactly as the Worker derives it. |
| Worker resolution | resolveRuntimeIntent reads the conversion intent and the marker in parallel: present marker → v1 (serving, no conversion record to verify); marker unknown → unknown (fail closed); marker absent with SANDBOX_RUNTIME=v1 → pending (the instance asked for v1 but was never provisioned for it — never boot legacy and create legacy state); otherwise legacy. The conversion intent still wins, so a converted tenant is unchanged. The read is staging-platform only; other platforms never read migration state. |
| Serving + boot | A born-v1 tenant onboards in the v1 container (no OPENCLAW_RESTORED, since it has no recovery point) and v1EnsureGateway launches start-openclaw.sh start as the single owner. migrate on such a tenant answers 409 MIGRATION_ALREADY_V1 without addressing the DO (an unreadable marker answers 503 SANDBOX_V1_RUNTIME_UNKNOWN), and migration-status answers 200 {state: "absent", mode: "fresh"} without touching the DO (mode: "converted" for a converted tenant). The Lerma preflight reports already_v1: true / reason sandbox_runtime_v1 with duration_ms: 0 (no control call). |
| Initial snapshot | After a successful boot (owner run and the adopted-gateway path) the DO dispatches a detached, best-effort runInitialSnapshot() so the tenant has a recovery point within seconds instead of waiting for the first idle-cadence tick. It runs only when verifyRecoveryPoint is pointer-absent and there is no conversion.json; a single-flight promise and a shared takeOwnedSnapshot() keep it from racing the cadence. A failure is logged (sandbox.v1.initial_snapshot_failed) and retried on a later boot (10-minute floor) without ever failing readiness. |
| Tests / docs | apps/fouria/src/sandbox-migration/{runtime-marker,intent-gate}.test.ts, apps/fouria/src/index.test.ts (fresh serving, 409, absent+fresh, pending without a marker, unreadable marker), apps/fouria/src/sandbox-v1.test.ts (fresh onboarding, first recovery point, retry, skip for converted/parked), Lerma sandbox_v1_runtime_test.exs, tenant_provisioner_pipeline_test.exs, tests: corpus test/business-rules/sandbox-v1-cutover.txt; apps/fouria/docs/WORKER-DEVELOPMENT.md § “Provisioning an instance directly on Sandbox v1”; observations record § “Provisioning a tenant directly on Sandbox v1”. |
| Commits | 99ed1d0d (Worker contract), f73b3fab (first recovery point), 35e37628 (Lerma runtime choice + marker step), 82bc961e (docs-only bundle tag). |
Tooling and release surface
publish-validation.sh builds the bundle and both images, pushes fouria:unstable-<sha> + fouria:v1-unstable-<sha>, resolves true manifest digests and publishes fouria-bundles/unstable-<sha>/{fouria.mjs,container-manifest.json}. It never tags latest, never targets client or production, and never echoes credentials. validate-runtime-image-pins.mjs asserts the pairing (3 v1 apps, 3 legacy apps); release-manifest.mjs / release-registry-digest.mjs are the P5b helpers, with tests.
Expected improvements vs the legacy runtime
Verified live
| Area | Improvement | Evidence |
|---|---|---|
| Durability | A converted tenant has a catalogued recovery point with a pointer, not one .sqsh handle | records/79165e52…, pointer advanced [live] |
| Encryption at rest | F4E1 applies to v1 records: ciphertext + sidecar, no plaintext object | sandbox-v1/openclaw/<id>.tar.zst.enc, encrypted: true [live] |
| Retention | Bounded versions, oldest pruned with payload, newest protected | 4 → 3 records [live] |
| Recovery | freeze snapshots then destroys; cold boot restores the pointer and serves in ~15 s | instance started 17:18:17, 200 running [live] |
| Rollout safety | Fail-closed park before replacement, write-block while parked, resume restores modes | 409 TENANT_PARKED, resume: true, d1c02116… [live] |
| Promotion truthfulness | Convergence on serving_version + zero error instances (pre-fix: success declared ~30 s into a ~2m40s rollout) | unstable-9952203a… [live] |
| Operability | Single owner, real readiness, non-waking passive probes, 503 until ready | health drill [live] |
| Isolation / scoping | Converted tenants never enter legacy .sqsh orchestration; sibling tenants stay legacy and serve unchanged | 503 SANDBOX_V1_CONTROL_UNSUPPORTED vs 200 [live] |
| Operator preflight | Dry-run reports already_converted + the v1 record id instead of calling a converted tenant eligible | client-sandbox-v1 dry-run [live] |
| Idempotent re-run | A real run on a converted tenant is a 200 already: true no-op with no control call and an unchanged pointer | 1.16 s, stable.json/conversion.json unchanged [live] |
Verified by suite only
| Area | Improvement | Tests |
|---|---|---|
| Restore by record id | Apply an older record, with or without promoting; fail closed on unknown | sandbox-v1.test.ts, storage.test.ts, sandbox-v1-persistence.txt |
| v1 upgrade gate | A v1 tenant’s health/rollout read comes from the …-sandbox-v1-… application; a wedged legacy rollout cannot fail a promotion | containers_test.exs (wedged legacy app + healthy v1 app), instances_upgrade_test.exs |
| Recovery-point surface | Pointer, record validation and readiness reported as distinct fields (C8) | sandbox_v1_status_test.exs, instance_live_test.exs, provisioning_controller_test.exs |
| OCM-3 surface sweep | Agent/channel/knowledge suites + the v1 process shim, DO entrypoint and single owner | sandbox-v1-agents.txt |
| Billing/cost surface | OTLP bridge emit, idempotent billable rows, freeze-on-v1, cost summary | sandbox-v1-billing.txt, sandbox-v1-cost.txt, sandbox-v1-outbound.test.ts |
| Idle cadence | Bounded loss window with no traffic, without waking a stopped container | sandbox-v1.test.ts (arm/disable/fire/dedupe/never-wake/destroy) |
| Corrupt-state-DB repair | Malformed openclaw.sqlite queues doctor-repair; boot re-queued after the fix | sandbox-v1.test.ts, scheduler.test.ts |
| Outbound routing | OTLP + R2 + CDP bridges registered on v1, tenant-scoped, re-registered on start | sandbox-outbound.test.ts, sandbox-v1.test.ts, sandbox-otel-bridge.test.ts |
| Session drain/restore | Read-only drain, bounded turns, modes survive the replacement | session-drain.test.ts, session-mode-restore.test.ts |
Expected / directional (not measured)
- Smaller idle loss window
[expect]— bounded to one interval instead of “since the last scheduled snapshot”. No bytes-or-minutes measurement exists; the first live tick is pending. - Fewer restore-side failure modes
[expect]— an SDK atomic directory swap replaces the legacy stage/materialize/symlink-repair sequence (and itscp-over-symlink and multi-minute copy failures). Argument from code shape, not a measured reduction. - No legacy archive orchestration on v1
[expect]— removes the 10-minute stage bound and its lease interactions from the converted path. CPU/request-budget effect unmeasured. - Platform fit
[expect]—scheduling_policy: "durable_object"is the only policy where platform filesystem/workspace snapshots are supported, so v1 is the runtime where that capability is even possible (the flag stays off). - Performance / cost
[expect]— no win claimed. No latency, RSS, CPU or R2-byte comparison between runtimes exists in the repo; do not quote one. - Isolation posture
[expect]— R2 credentials stay out of the container (Worker-side bridges) and no public route is added; supported by routing tests, not a red-team result.
Known gaps and out of scope
| Item | State | Next step |
|---|---|---|
| Restore-by-id live drill | [suite] — /api/admin/* is behind the host Access app; the lane’s only service token is accepted by the health app | Operator/browser session on client-sandbox-v1-dev |
| Idle cadence live tick | [live] — trigger: "backup" record 4437c278-… @ 2026-11-07T21:18:55Z with no operator action | Keep watching the next ticks; a gap means nobody sampled |
| Outbound-host runtime effect | Registration [live]; effect unobserved (idle tenant, no model spans) | Confirm on the next agent turn / diagnostics-otel health |
| Corrupt-state-DB repair | [suite] — forcing real corruption on staging was judged too risky for the signal | Fault-inject on a disposable tenant only |
| OCM-3 verify sweep (29 features) | Partly addressed by the sandbox-v1-agents corpus: agent/channel/knowledge route suites + the v1 process shim, DO entrypoint, single gateway owner and the automation/Telegram runtime resolution. The live 04_channels_and_rpc_ws drill is collected but not yet run on a tenant | Run the drill against a dev tenant |
| BIL-1 live verification | No per-tenant billable row from a v1 span observed live; the bridge/emit/idempotency path is corpus- and unit-verified (sandbox-v1-billing.txt) | One billable agent action on the converted dev instance |
| Converted-tenant upgrade gate | Fixed (suite) — Containers.v1_runtime?/1 (born v1 or a present v1 pointer) makes fetch_application_health/1 read the …-sandbox-v1-… application for a v1 tenant: rollout_active: false, live counts, and serving_version falls back to deployed_version (the v1 app declares no image tag). containers_test.exs drives a wedged legacy application next to a healthy v1 one | Live failure-injection: wedge the legacy app during a promotion |
| Born-v1 first provision | Fixed (suite) — a fresh v1 provision with no committed recovery point recreates a zero-instance SandboxV1 container application (Containers.recreate_v1_application/1, bounded create retry) and retries the health probe once (TenantProvisioner.verify_health/3). Also fixed: a durable_object app reports active, not healthy, so a live v1 tenant was read as instance-less (promotion retry + reconcile) — the v1 summary now maps active. Deleting only the app record did not reproduce the wedge live (the container still booted) | Fault-inject a placement failure; consider re-deploying the script when the app stays empty |
| Preflight asymmetry | Fixed (suite) — Instances.at_requested_version?/2 verifies the deployed script against the requested main module (not only the live image), so a failed script upload is retried by re-promoting | Live failure-injection drill on a disposable tenant |
S6 operator surface | Fixed (suite) — the migrate preflight/poll response carries a recovery_point per instance with pointer (id + committedAt), record (catalog validation + issues), the C8 verdict and readiness (the HealthMonitor’s last real running observation) as distinct fields; the instance page renders the same four values. sandbox_v1_status_test.exs covers pointer/record/readiness combinations; instance_live_test.exs covers the card | Run the drill on a dev tenant and read the card |
S7 rehearsal | Not run — the first conversion was the existing staging dev instance | A disposable non-production tenant |
| Full-corpus E2E snapshot latency | [live] — every functional suite passes. y_production_profile’s pre-restore snapshot is load-sensitive: it completes in ~55s in an isolated run but exceeded 10 minutes (sync) / 7 minutes (async) when the whole corpus had just driven the gateway, so the case times out. Phases A–C, D-startup and the other phase-D suites are green; the snapshot/restore path itself is corpus-covered and passed in isolation. a_rbac_api_matrix’s /api/admin/channels/accounts also hit 380s once and passed on the next full run | Investigate gateway RSS/event-loop latency under sustained CLI/RPC load (the observed sample was ~1.5 GiB), or run y_production_profile alone in CI |
Plan totals and boundary. 166 implemented features: 52 preserve, 73 verify, 41 rewrite; 15 deprecations, 35 out-of-scope rows. The boundary is permanent for this track: no production cutover, no production tenant converted, the reserved client slug never migrates, and unselected tenants stay on legacy byte-identically.
One-way door. A converted tenant cannot be silently returned to legacy. SANDBOX_V1_CUTOVER=false does not undo a conversion, because the intent is the authority and unknown is treated as present. There is no automated downgrade; the levers are restoring a catalogued v1 record or an explicit operator-driven rebuild, both audited and neither automated here. Decide before converting.
QA guidance
Environments and access
| Item | Value / how to read it |
|---|---|
| Staging Lerma | https://lerma-dev.paso4.io — fixed constant in promote-validation.sh, not an input |
| Staging dispatch | https://dev.fouria.io → POST /_control/:slug/<action> |
| Tenant (dev) / (prod-env) | staging-client-sandbox-v1-dev.fouria.io / staging-client-sandbox-v1.fouria.io |
| Account / namespace | Paso4 staging, dispatch namespace fouria-tenants-staging |
| Backup bucket | BACKUP_BUCKET / BACKUP_BUCKET_NAME from the tenant script settings (fouria-backup-<slug> by convention) |
| Promotion lane | .github/scripts/publish-validation.sh then promote-validation.sh (or staging-validation.yml) |
/api/healthsits behind the health Access app: it accepts the health-monitor service token (CF-Access-Client-Id/CF-Access-Client-Secret). Use it for readiness drills./api/admin/*sits behind the host Access app and needs an interactive identity; a service token that works for/api/healthwill get a302there. Use a browser/operator session.GET /healthzis on the tenant host (host app), non-waking and sub-second.- The dispatch control bearer is the static internal key guarded in
workers/fouria-dispatch/src/index.ts(/_control,/_invalidate). Keep it inINTERNAL_CONTROL_KEY; do not inline it in a committed script.
Secrets rules. Never echo a token, cookie, JWT, Access assertion or account id into a shell, log, PR comment or this document. Read from the environment inside the script, pass as headers, and print only status plus selected JSON fields.
: "${STAGING_PROVISIONING_API_KEY:?read it from your secret store first}"
curl -sS -o /tmp/status.json -w '%{http_code}\n' \
-H "Authorization: Bearer $STAGING_PROVISIONING_API_KEY" \
"https://lerma-dev.paso4.io/api/provisioning/upgrade/client-sandbox-v1"
jq '{slug, instances: [.instances[] | {name, status, deployed_version, serving_version, container_health}]}' /tmp/status.jsonAutomated gates
| Gate | Command | Expected |
|---|---|---|
| Format / lint / types / unit | bun run format:check && bun run lint && bun run typecheck && bun run test | all pass — last full run 3974 passed, 8 skipped |
| Business-rules corpus | cctr apps/fouria/test/business-rules/ | all pass — last run 582/582 |
| Elixir workspace | mix check && mix license_audit | format + strict Credo + license allowlist pass |
| Lerma suite | cd apps/lerma && mix test (or mix precommit from the root) | all pass — last run 1242 passed, 4 excluded |
| Promotion-lane tests | bunx vitest run --root . scripts/staging-validation.test.ts | all pass — last run 83/83 |
| Plan freshness | bun run plan:sandbox-v1:check | exit 0 (bun run plan:sandbox-v1 regenerates) |
| Dispatch suite | bun run --cwd workers/fouria-dispatch test | all pass |
| v1 unit slice | bun run --cwd apps/fouria test -- sandbox-v1 sandbox-migration sandbox-outbound | all pass |
| Backup E2E (local) | bun run --cwd apps/fouria test:e2e:backup | all pass (needs a local container runtime) |
| CLI E2E (cloud) | bun run --cwd apps/fouria test:e2e:cli | all pass (needs a deployed staging worker) |
| CLI E2E (local) | bun run --cwd apps/fouria test:e2e:local:cli | all pass; local-only scenarios self-skip |
| Software restore smoke | bun run test:software:smoke | passes against the image without workerd/Miniflare |
| Track sync | bun run plan:sandbox-v1:sync | merges origin/develop into v1-sandbox; never pushes |
apps/fouria/test/e2e-v1/** is a live corpus (00–07 + _setup/_teardown)
against a deployed tenant; every suite self-skips unless E2E_V1_LIVE=1, so it
collects in CI but its real result is a staging observation. 05/06 are
destructive and need E2E_V1_DESTRUCTIVE=1. See
apps/fouria/test/e2e-v1/README.md.
Manual drill playbook
Use the dev instance unless a drill needs the production-env one.
1. Health and serving parity
| Step | Command / action | Expected | Verify / triage |
|---|---|---|---|
| 1.1 | GET /api/health on the converted host (health-monitor token) | 200, status: "running", startup.ok/readiness.ok true, aiGateway/container/durableObject/r2 up | script settings show the SandboxV1 binding |
| 1.2 | GET /api/health?passive=1 after idle | 200, status: "sleeping"; container started_at unchanged | proves the probe never wakes |
| 1.3 | GET /healthz | 200 { ok, status: "active"|"idle" } sub-second | a slow response means it hit the proxy catch-all |
| 1.4 | Control UI in a browser | served through the v1 DO, WS connects | Access 302 means wrong session |
503 SANDBOX_V1_SLEEPING on a non-passive probe = the DO refused to wake a stopped container: boot it with a non-passive request, do not retry passive. 503 SANDBOX_V1_RUNTIME_UNKNOWN = the intent read is unknown: stop and inspect R2, do not force a boot.
2. Migration gate refusals and idempotence
| Step | Action | Expected |
|---|---|---|
| 2.1 | migrate on a non-staging deployment | 403 MIGRATION_PLATFORM_FORBIDDEN, no R2 write, no DO call |
| 2.2 | migrate on client | 409 MIGRATION_RESERVED_CLIENT |
| 2.3 | migrate with an unresolvable slug | 409 MIGRATION_SLUG_UNRESOLVED |
| 2.4 | migrate twice on client-sandbox-v1-dev | first 200 { state: "committed", recordId }; repeat idempotent (no second conversion) |
curl -sS -X POST -H "Authorization: Bearer $INTERNAL_CONTROL_KEY" \
"https://dev.fouria.io/_control/client-sandbox-v1-dev/migrate"
Verify durably: conversion.json + stable.json exist and agree (verifyRecoveryPoint: complete). Triage: a 502 carries the conversion error in tenantError; read sandbox.v1.conversion.failed before retrying. Retrying a failed state is legitimate; retrying committed must be a no-op.
3. Snapshot, catalog, pointer, retention, encryption
curl -sS -X POST -H "Authorization: Bearer $INTERNAL_CONTROL_KEY" \
"https://dev.fouria.io/_control/client-sandbox-v1-dev/checkpoint"
# → {"ok":true,"action":"checkpoint","backupId":"<uuid>"}| Verify | Expected |
|---|---|
| Catalog | records/<id>.json exists and stable.json names that id |
| Encryption | <id>.tar.zst.enc + <id>.meta.json (encrypted: true); no plaintext .tar.zst |
| Retention | count ≤ BACKUP_MAX_VERSIONS (default 3), newest still present |
| No legacy artefacts | no new backups/<id>/data.sqsh for that tenant |
SNAPSHOT_FAILED / SANDBOX_V1_SNAPSHOT_FAILED leaves the container running and the previous pointer intact — safe to retry. A catalog entry without a payload (or vice versa) is a prune bug: capture both listings before retrying.
4. Freeze and cold-boot restore
| Step | Action | Expected |
|---|---|---|
| 4.1 | POST /_control/:slug/freeze | 200 after the snapshot commits, then the container is destroyed (status: stopped) |
| 4.2 | one non-passive GET /api/health | first probe may be 503 stopped in ~0.5 s (detached boot); next 200 running with both flags ok (~15 s in the PER-2 drill, ~36 s in the OCM-2 drill) |
| 4.3 | compare instance started_at and stable.json | started_at advanced, pointer unchanged (it is the input) |
Triage: booting into an empty tenant means the pointer/record barrier broke — stop, do not let it onboard; inspect verifyRecoveryPoint (pointer-absent vs record-absent).
5. Restore by record id
Needs an operator/browser session (/api/admin/*, host Access app).
| Step | Action | Expected |
|---|---|---|
| 5.1 | POST /api/admin/storage/restore/<recordId> | 200 with gatewayReady; stable.json advances to <recordId> |
| 5.2 | same with ?setCurrent=0 | 200; pointer does not advance |
| 5.3 | unknown/unreadable id | 404, container untouched |
| 5.4 | concurrent restore | refused (in-DO flag) |
Verify durably: pointer value, sandbox.v1.restore.applied log, and a file written before the snapshot. Triage: a boot failure after a successful apply is not a restore failure — the next request retries the boot; never re-run the destructive step.
6. Park and resume
POST /_control/client-sandbox-v1-dev/checkpoint-park # 200 { parked: true, backupId, sessionsPatched, sessionsDrained }
POST /_control/client-sandbox-v1-dev/backup # 409 { code: "TENANT_PARKED" }
POST /_control/client-sandbox-v1-dev/resume # 200 { resumed: true }| Check | Expected |
|---|---|
| Pointer before the park arms | advanced to the park record |
| Reads while parked | still served |
After resume | mutating request 200; /api/health 200 running; no restore needed (park does not destroy) |
| Telegram during park | ingress sends the parked notice, 200, no AI turn |
Verify durably: maintenance/replacement.json { parked: true } only while parked; maintenance/session-modes.json cleared once all modes are restored. 502 SANDBOX_V1_PARK_SNAPSHOT_FAILED = the park never opened (still serving); an incomplete park un-arms the lock and closes the window; the 15-minute TTL is the crash backstop.
7. Promotion lane (park → rollout → resume)
VALIDATION_BRANCH=v1-sandbox .github/scripts/publish-validation.sh --publish --sha "$(git rev-parse HEAD)"
TARGET_SLUG=client-sandbox-v1 FOURIA_VERSION=unstable-<sha> CONFIRM_TARGET=client-sandbox-v1 \
STAGING_PROVISIONING_API_KEY="$STAGING_PROVISIONING_API_KEY" .github/scripts/promote-validation.sh| Check | Expected |
|---|---|
| Gate | converges only when serving_version == unstable-<sha> on both instances and rollout_active: false |
| Park evidence | Pre-replacement checkpoint parked ... parked=true with a backup_id |
| Recovery point | stable.json advanced before the image rollout |
| Resume | Resume after container replacement ... resumed=true; a following backup returns 200 |
| Error instances | none; the gate fails if any instance is error, whatever version it declares |
Triage: a hang until timeout usually means the running Lerma predates serving_version — redeploy the branch Lerma first. A “success” in seconds while the rollout is mid-flight is the known pre-fix behaviour; check serving_version yourself. A stuck rollout_active on the legacy app of a converted tenant is the open follow-up (gate on the v1 app).
8. Idle snapshot cadence
| Step | Action | Expected |
|---|---|---|
| 8.1 | Read stable.json → committedAt and the container started_at | baseline |
| 8.2 | Leave the dev tenant running (interval capped at 1 h) | no traffic needed |
| 8.3 | Poll stable.json up to ~70 min | committedAt advances once; started_at unchanged |
| 8.4 | Check Workers Logs | one sandbox.v1.snapshot.cadence event with recordId + took |
Never trigger this with a request: a request renews activity and masks the cadence. cadence_arm_failed = the DO could not arm (boot still succeeds); cadence_failed keeps the previous pointer and re-arms. If the container idle-stopped first, no tick is expected — that is the documented never-wake behaviour.
9. Outbound bridges
| Check | How | Expected |
|---|---|---|
| OTLP | one billable agent turn on the converted dev instance | otel.internal accepted; per-tenant cost rows in the AE dataset; no 302 |
| Bridge sanity | from the container, list leads/docs/software.r2.internal | own <slug>/ prefix only; structured JSON on a bad binding, never ECONNRESET |
| Re-registration | restart the DO and repeat | intercepts present again |
ECONNRESET means a thrown exception escaped the handler (it should be a structured 500/503). A missing intercept means the bundle lacks the SandboxOutboundGateway export — the boot warns and the container falls back to direct egress.
10. ROOT debug tools
Prerequisites: DEBUG_ROUTES=true on the tenant, and a ROOT browser session
on https://<tenant>/_admin/ (the per-tenant Access app answers 302 to any
headless /debug/* call, so curl from a laptop or the container is not a valid
probe). Run the cards under Debug Tools and cross-check /debug/* directly.
| Tool (route) | Expected on a converted tenant |
|---|---|
Version (GET /debug/version) | 200 with openclaw_version, node_version, fouria_version/fouria_commit |
Process Inspector (GET /processes) | 200; status: running for the start-openclaw.sh start owner; empty after a DO eviction is normal |
Process logs (?logs=true) | a finished process returns stdout from output() (no logs_error); a running process returns logs_error: 'process running …' |
Gateway Logs (GET /gateway-logs) | 200, path: /tmp/gateway-stdout.log, runtime: 'v1', stdout = the file tail |
Gateway API (GET /gateway-api) | 200 with the probed status/body; a sleeping container reports 503 SANDBOX_V1_SLEEPING inside the body |
Environment (GET /env) | sandbox_runtime_actual: 'v1' (env-level sandbox_runtime may still say legacy) |
Container Config (/container-config) | 200 with the parsed openclaw.json, or raw when it is not JSON |
SQLite Integrity (POST /sqlite-integrity) | 202 + job id (durable) or a 200 report; the durable repair job runs on v1 too |
Stop Gateway (POST /stop-gateway) | 200; the gateway process disappears, the container stays up |
Restart OpenClaw (POST /restart-openclaw) | 200, runtime: 'v1', message “…will restart on next request”; the next request relaunches, no .sqsh checkpoint |
Backup (POST /backup/sync) | 200, backedUp: true + recordId; stable.json advances; the gateway is not booted by the request |
Destroy Container (/destroy-container) | 200; the next request cold-boots and restores the recovery point |
Reprovision (POST /reprovision) | 200 with recordId; a v1 snapshot precedes the destroy; without a restorable C8 point it fails closed 409 |
Rescue Container (/rescue-container) | diagnoses and can destroy an orphaned/Error container; a sleeping tenant reads as unreachable (v1 containerFetch never wakes) |
Sleep (POST /debug/sleep) | 503 { code: 'SANDBOX_V1_SLEEP_UNSUPPORTED' } — this seam is legacy-only |
11. Failure injection (disposable tenant only)
| Injection | Expected |
|---|---|
Corrupt state/openclaw.sqlite before boot | fails in seconds (not after the 300 s budget); doctor-repair queued with gateway-launch-state-database; boot re-queued after repair |
| Gateway exits non-zero before binding | fail-fast, terminal for that attempt, no duplicate launch |
| Pointer names a missing record | fail closed (record-absent), no onboarding over converted state |
| Restore over a live gateway | maintenance window first, gateway stopped before the swap, no EBUSY |
Intent read unknown | no legacy fallback, no state advance |
Never inject on tenant-client-sandbox-v1 (production-env) or any tenant with real data.
12. Operator migration path (Lerma)
Run against the deployed staging Lerma (https://lerma-dev.paso4.io) with the
provisioning application key — never the global control key, and never printed.
?dry_run=1 performs no control call and no container wake.
POST /api/provisioning/migrate/<slug>?dry_run=1
Authorization: Bearer $STAGING_PROVISIONING_API_KEY
X-FourIA-Application: fouria
POST /api/provisioning/migrate/<slug>?dry_run=1&instance=<cf_worker_name> # triage one instance
POST /api/provisioning/migrate/<slug> # enqueue (one-way), 202
GET /api/provisioning/migrate/<slug> # poll only (migration-status)
POST is the only trigger: it enqueues the durable conversion (migrate, an
idempotent enqueue that starts work from absent/failed). GET and the
Worker’s X-FourIA-Control: migration-status action it maps to are strictly
read-only — they call v1ConversionState() and never v1RequestConversion() —
so polling can never convert a tenant, start work or wake the container.
| Step | Action | Expected on staging |
|---|---|---|
| 12.1 | dry-run a converted client (e.g. client-sandbox-v1) | 200, eligible: false, reasons: ["already_converted"], warnings: ["already_converted"], already_converted: true, v1_record_id = the pointer’s recordId |
| 12.2 | same, pinned to the production-environment instance | 200, eligible: false, reasons: ["production_environment"], that instance already_converted: false unless it was converted too |
| 12.3 | dry-run client | 409, code: MIGRATION_NOT_ELIGIBLE, reasons: ["reserved_client"], no control call |
| 12.4 | dry-run on a production-platform Lerma | 403, reasons: ["platform_not_staging"] |
| 12.5 | idempotency: real run on an already-converted client | 200 in ~1 s: {state: "committed", already: true, record_id: <pointer id>, duration_ms: 0}; stable.json and conversion.json unchanged |
| 12.6 | pointer unreadable (already_converted: nil) | the GET still reads migration-status (200 {state} — absent if nothing was enqueued); only a POST enqueues |
| 12.7 | enqueue then poll a legacy dev tenant | POST answers 202 {state: "queued"|"running"} in seconds (no wait); repeated GET reads 200 {state} until it reports committed + record_id (a GET never enqueues) |
| 12.8 | poll a tenant whose job failed | the GET answers 502 {state: "failed", error} (the Worker’s read is a 200; Lerma maps it); only a fresh POST (or --yes) starts the next attempt |
Nothing here holds a request open for the conversion: the POST only enqueues
and the GET reads the durable job’s state, so a proxy 524 cannot make an
in-flight conversion ambiguous. Verify durably, both sides of a real run:
sandbox-migration/<tenant>/stable.json (recordId, committedAt) and
conversion.json (recordId, createdAt) must be identical before and
after an already: true response — a changed pointer means a conversion
actually ran. Poll until terminal (or run mix lerma.sandbox_v1.migrate --client <slug> --yes, which enqueues and polls with --wait/--poll-interval)
before doing anything else. Never retry with a different instance.
11. Born-v1 provisioning (fresh tenant, no conversion)
Provision a new dev instance on a staging client with a free dev slot (never
the reserved client slug) and pass sandbox_runtime: "v1". Do not convert
anything, and do not touch an existing tenant.
| Step | Action | Expected |
|---|---|---|
| 11.1 | POST /api/clients/<id>/instances with {"instance":{"environments":["dev"],"sandbox_runtime":"v1","version":"unstable-<sha>"}} | 202 {run_id, sandbox_runtime: "v1"} |
| 11.2 | Read the run’s provisioning_step_events | sandbox_runtime_marker (succeeded) immediately before dispatch_deploy (succeeded) |
| 11.3 | GET /accounts/<acct>/r2/buckets/<bucket>/objects?prefix=sandbox-migration/<slug>/ | runtime.json with {version:1, tenant:<slug>, runtime:"v1", mode:"fresh", provisionedAt} |
| 11.4 | Dispatch-namespace script settings | SANDBOX_RUNTIME=v1, SANDBOX_V1_CUTOVER=true, SandboxV1 DO binding, WORKER_PLATFORM=staging |
| 11.5 | Poll GET /api/health with the health-monitor token | 200 running (a fresh onboarding takes ~5 min; the container may report stopped while it boots) |
| 11.6 | POST /_control/<slug>/migration-status and …/migrate | 200 {state:"absent"} and 409 MIGRATION_ALREADY_V1 — no conversion can start |
| 11.7 | Poll sandbox-migration/<slug>/stable.json (≤ 5 min) | a recordId within seconds of readiness (the initial snapshot), with records/<id>.json and <id>.tar.zst.enc + .meta.json; no plaintext archive |
| 11.8 | POST /_control/<slug>/backup | 200; the pointer advances (operator snapshots work) |
| 11.9 | GET /api/provisioning/migrate/<client-slug> | already_v1: true, reason sandbox_runtime_v1, duration_ms: 0 — no control call |
If the SandboxV1 container application stays at zero instances while the DO
reports running: true (probes time out, /control …/backup hangs), the
background health check for a fresh, pointer-less provision now recreates the
application once and retries the probe
(TenantProvisioner.verify_health/3 → Containers.recreate_v1_application/1).
If it still stays empty, deprovision the instance and re-provision: the marker
is create-only and survives, so the tenant is still v1 and the new container app
starts normally. Triage with the container applications API
(health.instances.active) before suspecting the image — the same tag boots
elsewhere.
Acceptance checklist
[ ] Gate: 403/409/409 observed, no R2 write, no DO/container addressing
[ ] Gate: `client` refuses migrate only; freeze/backup/checkpoint unchanged
[ ] Conversion: 200 committed + recordId; repeat is idempotent
[ ] Barrier: stable.json names a shape-valid record (verifyRecoveryPoint: complete)
[ ] Snapshot: catalog entry + pointer + ciphertext + sidecar, no plaintext
[ ] Retention: <= BACKUP_MAX_VERSIONS, newest protected
[ ] Freeze: snapshot committed BEFORE destroy; cold boot restores and serves
[ ] Restore-by-id: promotes by default, ?setCurrent=0 does not; 404 on unknown
[ ] Park: 409 TENANT_PARKED on writes; reads served; resume releases
[ ] Park: sessions drained/restored; Telegram notice sent and acked
[ ] Promotion: serving_version converged on both instances; zero error instances
[ ] Promotion: park log + backup_id before rollout; resume log after
[ ] Cadence: committedAt advances without started_at changing
[ ] Bridges: OTLP accepted, R2 bridges tenant-scoped, no ECONNRESET
[ ] Health: passive never wakes; real readiness; /healthz sub-second
[ ] Debug tools: gateway-logs path+runtime, backup recordId, restart/backup/destroy 200 on v1
[ ] Debug tools: /debug/sleep answers 503 SANDBOX_V1_SLEEP_UNSUPPORTED (no stop claimed)
[ ] Lerma migrate: dry-run refuses a converted tenant with `already_converted` + the pointer record id
[ ] Lerma migrate: real run on a converted tenant is `200 already: true` with an unchanged pointer/conversion record
[ ] Lerma migrate: production-environment instance and `client` slug refuse; only `dev` tenants are eligible
[ ] No legacy artefacts written for the converted tenant (no new .sqsh)Rollback and abort guidance
- Before converting. Abort is free: leave the tenant legacy. Unselected tenants are byte-identical to today and every legacy suite still gates them.
- During a park. An incomplete park un-arms the exec lock, closes the maintenance window and restores service; the 15-minute TTL is the crash backstop. Do not delete the R2 park/record objects by hand while a rollout is in flight.
- After converting. No flag-based downgrade:
SANDBOX_V1_CUTOVER=falsedoes not move a converted tenant back, because the intent is the authority. Use the audited operator levers (restore an older catalogued record, or an explicit rebuild). Never deleteintent.json,conversion.json,records/*orstable.jsonto “reset” — that is how two divergent sources of truth are created. - Staging Lerma. Each deployment in the observations doc records its own compose backup (
/opt/lerma/docker-compose.yml.bak-*); restore the most recent anddocker compose up -d --no-deps --force-recreate app. The branch Lerma must drive v1 upgrades — adevelopLerma promotion re-uploads the legacy binding set and drops the v1 bindings. - Bundle rollback. There is no automatic downgrade. Re-publishing an earlier
unstable-<sha>is a forward promotion of that build and goes through the same park → rollout → resume contract.
Result reporting template
Environment: staging (Paso4) — client client-sandbox-v1 (id 20)
Instance(s): tenant-client-sandbox-v1-dev [ , tenant-client-sandbox-v1 ]
Commit: 806a2580 (or the promoted unstable-<sha>)
Bundle / images: unstable-<sha> (legacy + v1-unstable-<sha>)
Lerma image: ghcr.io/0xcab0/lerma:v1-sandbox-<sha> (or :latest-pre)
Drills run: gate, conversion, snapshot, freeze/restore, restore-by-id,
park/resume, promotion, cadence, bridges, failure injection
Result per drill: PASS / FAIL / BLOCKED (reason)
Durable evidence: record ids, stable.json committedAt, backup ids, instance
started_at, request ids, log event names
Open issues: <id + one line>, or "none"
Not measured: <what you deliberately did not claim>Verification log
Latest deployment — Lerma migration + v1 debug tools (2026-11-07, unstable-6602f1fc…, from sandbox-v1-staging-observations.md):
| Item | Value |
|---|---|
| Bundle + images | unstable-6602f1fc98fc41692678ea05718adde8505f6b56 (fouria:unstable-…, fouria:v1-unstable-…) |
| Artifacts | R2 fouria-bundles/unstable-…/{fouria.mjs,container-manifest.json} |
| Staging Lerma | ghcr.io/0xcab0/lerma:v1-sandbox-6602f1fc, /health 200, compose backup .bak-pre-6602f1fc |
| Promotion | 2 running instance(s) accepted → client-sandbox-v1 is running unstable-6602f1fc… |
| Status gate | both instances deployed_version = serving_version = unstable-6602f1fc…; running; healthy 3/2; zero errors |
| Park / resume | v1-dev parked 21:34:57Z (556d5773-…) → resumed 21:40:02Z; v1 parked 21:40:31Z (5eb28f82-…) → resumed 21:44:10Z |
| Dry-run, converted tenant | eligible: false, reasons: ["already_converted"], already_converted: true, v1_record_id 556d5773-… |
| Dry-run, prod-environment | eligible: false, reasons: ["production_environment"], already_converted: false (that instance is still legacy) |
| Idempotency drill | 200 in 1.16 s → {state: "committed", already: true, record_id: 556d5773-…}; pointer + conversion.json (77cadb66-…) unchanged |
| Idle cadence | first live tick: trigger: "backup" record 4437c278-… @ 2026-11-07T21:18:55Z, no operator action |
| Tenant health | /api/health 200, running, startup.ok + readiness.ok on both hosts (health-monitor service token) |
| ROOT debug tools | not verified live: /debug/* answers 302 to Cloudflare Access on tenant hostnames (service_token_status: false) |
Tests from this run: unit/format/lint/typecheck 4037 passed, 8 skipped (199
files); business-rules corpus 596/596; scripts/staging-validation.test.ts
83/83; dispatch unit 127; plan:sandbox-v1:check, mix check,
mix license_audit pass; Lerma 1305 passed, 4 excluded. No tenant was
converted by this deployment.
Previous deployment — CDP bridge fix (2026-11-07, unstable-e56ac9c8…, from sandbox-v1-staging-observations.md):
| Item | Value |
|---|---|
| Bundle + images | unstable-e56ac9c8d79f062ee6b66bfed1f221e1e82f102f (fouria:unstable-…, fouria:v1-unstable-…) |
| Artifacts | R2 fouria-bundles/unstable-…/{fouria.mjs,container-manifest.json} |
| Staging Lerma | ghcr.io/0xcab0/lerma:v1-sandbox-e56ac9c8, /health 200, compose backup .bak-pre-e56ac9c8 |
| Promotion | 2 running instance(s) accepted → client-sandbox-v1 is running unstable-e56ac9c8… |
| Status gate | both instances deployed_version = serving_version = unstable-e56ac9c8…; errors: 0; healthy: 2 / 3; rollout_active: false |
checkpoint-park (dev) | 200, parked: true, backupId 8f4e88bc-368a-48c5-888e-30317895f6f8, sessionsPatched: 0 |
resume (dev) | 200, resumed: true; a following backup returned 200 (park released) |
| Recovery point | recordId 0e426f0e-577b-40a1-a4a6-c0abcafaa20e, committedAt 2026-11-07T20:12:55Z |
| Lerma park/resume | parked 20:02:50Z / resumed 20:07:16Z (v1-dev), parked 20:07:39Z / resumed 20:11:01Z (v1) |
cdp.internal | registered on both runtimes: both tenant scripts list all 6 hosts in SANDBOX_OUTBOUND_ROUTES, incl. [CDP_INTERNAL_HOST] |
| Idle cadence | unit-verified; live observation pending the next hourly tick (not waited on) |
cdp.internal status. The merge-resolution fix (e56ac9c8) is now the
deployed bundle on both tenants: the shared SANDBOX_OUTBOUND_ROUTES map carries
[CDP_INTERNAL_HOST]: handleCdpRequest, so the legacy class statics and the v1
per-instance interceptOutboundHttp registration expose the same 6 hosts
(releases/docs/leads/software.r2.internal, otel.internal,
cdp.internal). The registration is [live] (read from the deployed script
content on both tenant scripts); the browser round-trip through the container’s
cdp-proxy.mjs remains not measured — test/e2e/r_browser_remote_profile.txt
is cloud-profile only and was deliberately not run.
Tests from this run: unit/format/lint/typecheck 4022 passed, 8 skipped (199
files); business-rules corpus 593/593; scripts/staging-validation.test.ts
83/83; plan:sandbox-v1:check, mix check, mix license_audit pass; Lerma
1259 passed, 4 excluded.
Previous combined deployment (2026-11-07, unstable-795f1dab…, from sandbox-v1-staging-observations.md):
| Item | Value |
|---|---|
| Bundle + images | unstable-795f1dab0d6b2d4bce7b6a98c9b5afb3be089e1f (fouria:unstable-…, fouria:v1-unstable-…) |
| Artifacts | R2 fouria-bundles/unstable-…/{fouria.mjs,container-manifest.json} |
| Staging Lerma | ghcr.io/0xcab0/lerma:v1-sandbox-795f1dab, /health 200, compose backup .bak-pre-795f1dab |
| Promotion | 2 running instance(s) accepted → client-sandbox-v1 is running unstable-795f1dab… |
| Status gate | both instances deployed_version = serving_version = unstable-795f1dab…; errors: 0; healthy: 3 / 2; rollout_active: false |
checkpoint-park (dev) | 200, parked: true, backupId 63f79239-9d43-4824-96b0-83f37e34b71e, sessionsPatched: 0 |
resume (dev) | 200, resumed: true; a following backup returned 200 (park released) |
| Recovery point | recordId 1a169e07-cb50-444c-8ae5-5976bfe72a20, committedAt 2026-11-07T08:50:12Z |
| Lerma park/resume | parked 08:46:35Z, resumed 08:48:39Z (parked=true / resumed=true) |
| Idle cadence | unit-verified; live observation pending the next hourly tick |
Tests from that run: unit/format/lint/typecheck 3974 passed, 8 skipped; business-rules corpus 582/582; scripts/staging-validation.test.ts 83/83; plan:sandbox-v1:check, mix check, mix license_audit pass; Lerma 1242 passed, 4 excluded.
Merge check (806a2580) and repair. bun run test on the merged tree
reported 4020 passed, 8 skipped, 2 failed: both were
apps/fouria/src/index.test.ts > Sandbox.outboundByHost (“registers the release,
docs, leads, software, OTLP and CDP bridge hosts” and “answers the CDP bridge
health probe”). The develop-side CDP bridge (#563) adds CDP_INTERNAL_HOST /
handleCdpRequest in src/sandbox-cdp-bridge.ts, but the v1-side refactor moved
the map to src/sandbox-outbound.ts and the merge resolution did not add a
cdp.internal route, so TenantSandbox.outboundByHost['cdp.internal'] was
undefined. format:check, lint and typecheck passed (the map is generic),
so it was a merge-resolution regression, not a type error — and because both
runtimes consume the same map, v1 was missing the intercept as well.
Fixed by the follow-up commit fix(fouria): register the cdp.internal bridge on the shared outbound host map (PR #565): the CDP handler joins
SANDBOX_OUTBOUND_ROUTES, the dead bridge imports are dropped from
src/index.ts, and the host is pinned in sandbox-outbound.test.ts /
sandbox-v1.test.ts (plus the two node-pool suites now stub
@cloudflare/sandbox-legacy, which the CDP bridge pulls in transitively).
Measured on the fixed tree: unit/format/lint/typecheck 4022 passed, 8 skipped,
0 failed; business-rules corpus 593/593. Live behaviour of cdp.internal
on a converted tenant remains not measured (same status as the other
outbound hosts).
Earlier drills cited above (conversion 77cadb66…, snapshot 79165e52…, freeze 17:18:17, scheduler 2c97395d…, park 2455d8c8…, restores d1c9f143… / d1c02116… / 26c78dac…) are in the observation record with dates and bundle versions; cite that file as the primary source.
References
| Document | Purpose |
|---|---|
sandbox-v1-cutover-contracts.md | Normative invariants C1–C10 and the slice map |
sandbox-v1-decisions.md | Append-only ADR log (ADR-SV1-001…009) |
sandbox-v1-evaluation-plan.json / .md | 166 features, 14 milestones, dispositions, five-step protocol |
sandbox-v1-scaffold.md | Reference note: what SandboxV1 is and is not |
sandbox-v1-staging-observations.md | The live evidence record this report cites |
apps/fouria/docs/WORKER-DEVELOPMENT.md | Runtime deep-dive; §“Sandbox v1 …” per slice |
apps/fouria/docs/OPERATOR-GUIDE.md | §2.5 rollout park, §2.6 migration gate |
apps/lerma/README.md | v1 provisioning, promotion gate, zero-loss upgrades |
docs/TESTING.md | Test surface; “Sandbox v1 migration plan” |
docs/ENV-VARIABLES.md | WORKER_PLATFORM, SANDBOX_V1_CUTOVER, SANDBOX_V1_SNAPSHOT_INTERVAL_MS |