F FourIA staging 2026.10.29 GitHub ↗

Sandbox v1 adversarial + fuzz suites

On this page

Two seeded suites attack the Sandbox v1 (Sandbox SDK 1.0) critical pipelines instead of exercising curated examples. Every random case is reproducible from its seed, and a failure message names the seed and the case index.

SuiteScope
apps/fouria/src/sandbox-migration/fuzz.test.tsPure state machines (no DO): tri-state reads, validators, C8 barrier, retention, lease, runtime marker, intent gate, catalog listing
apps/fouria/src/sandbox-v1-adversarial.test.tsThe SandboxV1 Durable Object: encryption, secrets, snapshots, cadence, boot, process map, transport
bun run --cwd apps/fouria test src/sandbox-migration/fuzz.test.ts
bun run --cwd apps/fouria test src/sandbox-v1-adversarial.test.ts

Attack matrix

PipelineAttackInvariant assertedResult
Zero data lossRandom pointer / catalog record / conversion-absent / malformed / transport-error combinations (1 000 cases)verifyRecoveryPoint agrees with an independent oracle; unknown never falls through to a legacy readholds
Zero data lossRandom retention sizes, duplicate timestamps, shuffled bucket order (1 000 cases)The protected record is never pruned, pruning is order-independent and deduplicatedholds
Zero data lossSDK backup / pointer write / post-encryption failures (60 fuzz errors)A failed snapshot never advances stable.json; the previous point stays completeholds
Zero data lossSimulated stopped container (running = false)Snapshot returns skipped, never starts the container, never archives an empty homeholds
Zero data lossCatalog listing reports truncation with a repeated/absent cursorListing is bounded (page cap + non-advancing-cursor stop)fixed
Zero data lossPost-commit catalog listing/retention failureThe snapshot is still reported committed (the point already advanced)fixed
Zero data lossRecovery point dated in the future (clock skew / corruption)The idle cadence keeps snapshotting instead of suppressing the loss bound foreverfixed
Zero data lossRandom 80-snapshot sequence with changing BACKUP_MAX_VERSIONS and injected failuresEvery committed point is restorable; retention stays bounded; a failure never moves itholds
SecretsEncryption-required × absent/empty/malformed/short/valid key matrixRequired + unusable key uploads nothing; a broken optional key is never ignoredholds
Secrets60 random rejected key materialsThe key never appears in a thrown error or any console logholds
SecretsBoot attempt with sentinel token/key envContainer env is never loggedholds
SecretsPath-ish tenant names (../../etc/passwd, %2e%2e, …)The backup prefix stays one encoded key segmentholds
MemoryMAX_TRACKED_PROCESSES + 144 execsThe DO process map is bounded; exited handles are dropped oldest-firstfixed
MemoryOne still-running handle plus a flood of exited handlesA running handle is never dropped (gateway stays killable)holds
BootConversion record present without its stable pointerBoot refuses to onboard over converted state; never launches the setup bodyholds
Boot40 corrupt stable.json bodiesBoot refuses on an unreadable pointer instead of onboardingholds
BootBrand-new tenant (no record, no pointer)Launch runs without OPENCLAW_RESTORED (the setup body owns the state)holds
BootMalformed X-FourIA-Sandbox-Port (1e3, 0x50, 3.5, …)Rejected with 400 before any container accessfixed
BootStopped container request503 SANDBOX_V1_SLEEPING, never wakes the containerholds
Tenant isolationRuntime marker copied under another tenant’s scope; intent gate random statesMarker authority is scope-bound; an intent never falls back to legacyholds
SecretsSnapshot with sentinel gateway token / AI key / backup key in envNo R2 object written by the snapshot contains a secret valuefixed (scan)
SecretsConversion failure whose error embeds a credential (signed-URL shape)The persisted status and the log contain [redacted], never the valuefixed
MemoryA process whose output() returns 3× the stream capThe decoded output is truncated at MAX_PROCESS_OUTPUT_BYTES and flagged truncatedfixed
Zero data lossA snapshot taken while an operator restore is swapping the directoryskipped: 'restoring', no archive, the previous point staysfixed
Zero data lossA restore that fails mid-swap, then a retryThe point never moves on failure and the in-progress barrier is clearedholds
Zero data lossA second restore (and a boot) while a restore is in flightRefused; the boot never launches over the swapholds
Zero data loss12 snapshots → 24 randomized restores of surviving records → boot the point (fuzz)Every surviving record restores, the point matches, and the boot skips the setup bodyholds
Memory (disk)A conversion whose extractor / upload failsThe streamed archive and partial extraction are removed best-effort; the error winsfixed
Bootstart-openclaw.sh start exits 0 with no gateway process anywhereThe boot fails after the zero-exit grace instead of waiting the whole cold-start budgetfixed
Bootstart-openclaw.sh start exits 0 while the lock winner is still bootingThe DO keeps waiting and adopts the peer’s readinessholds

Defects found and fixed

  1. Unbounded catalog listing loop (OOM). listRecordEntries followed page.cursor without bounding the iteration. A bucket (or a proxy) that reports truncated: true with a non-advancing cursor made the loop read the same page forever; the fuzz run reproduced it as a real FATAL ERROR: Ineffective mark-compacts near heap limit in the test worker. Fixed with MAX_RECORD_LIST_PAGES, repeated-cursor detection and per-recordId deduplication — an incomplete list can only retain more records, which is the safe direction for retention (C6).
  2. A committed snapshot reported as failed. v1CreateSnapshot wrote the catalog entry and advanced stable.json, then let a listing/retention error escape, so the caller saw a failed snapshot whose recovery point had in fact moved (and retention never ran). Retention is now best-effort after the commit and logs sandbox.v1.snapshot.retention_failed.
  3. Future-dated recovery point suppressed the cadence forever. The dedupe test was Date.now() - committedAt < interval, which is true for any future timestamp. It now only dedupes a non-negative age inside the interval.
  4. Unbounded process-handle map. Every v1Exec retained its ExecProcess handle for the DO’s lifetime. The map is now capped at MAX_TRACKED_PROCESSES (256), dropping exited handles oldest-first and never dropping a running one.
  5. Port-header coercion. The DO transport accepted Number() coercions (1e3, 0x50). The header must now be canonical decimal digits before the range check.

Observations left as code, not changed here

  • Conversion disk amplification. materializeLegacyArchive writes the archive to /tmp/fouria-legacy.sqsh, extracts it to /tmp/fouria-legacy-extract, then copies the staged roots into /home/openclaw. Peak disk is roughly twice the archive plus the restored home (~3× archive) on a 4–12 GB container. A preflight free-space check (or extracting through a pipe) is the follow-up; it is a deploy/runtime decision, not a pure invariant.
  • Process cap vs. running handles. The bound only evicts exited handles. A pathological burst of still-running processes can exceed it rather than losing the ability to kill the gateway.
  • Backup key scope. directoryBackupPrefix is derived from the DO id (sandbox-v1/openclaw/ for the fixed production container id). Tenant isolation comes from the per-tenant fouria-backup-<slug> bucket, not from the key; keep that binding invariant when changing provisioning.

Second run: critical-pipeline hardenings (2026-11-09)

The second attack pass targeted the four incident pipelines again, this time around the restore/snapshot/boot interactions and persisted text:

  1. A snapshot could race an operator restore. v1CreateSnapshot only checked the container’s liveness; the worker’s maintenance window prevented the overlap for route-driven calls, but the primitive itself would archive a half-swapped directory and then advance the recovery point to it. v1CreateSnapshot now returns skipped: 'restoring' while #restoring.
  2. A boot could launch over a swap. startGatewayOwner checked #restoring only at entry (through v1EnsureGateway); a restore that began while the state probe ran still launched. The owner now re-checks before the launch and inside the readiness loop, and reports a retryable failure instead of readiness for a gateway the restore is about to kill.
  3. Unbounded process output. v1ProcessOutput decoded whatever the process wrote, after the SDK had already buffered it in the DO isolate. Reads are now truncated at MAX_PROCESS_OUTPUT_BYTES (1 MiB per stream) with truncated: true. The SDK’s own buffer is still bounded only by the command, so callers must keep shell commands bounded (the system does: tail -n 200, JSON CLI output).
  4. Zero-exit boots burned the full cold-start budget. A wrapper that exits 0 (the lost-startup-lock path) was only judged by the health poll, so a launch nobody owned waited timeoutMs (up to 5 min). After SANDBOX_V1_ZERO_EXIT_GRACE_MS (default 15s, 0 disables) the DO proves no gateway (or setup wrapper) process exists (pgrep -f on neither) and fails fast; a booting peer (lock winner) resets the grace and is adopted normally.
  5. A failed conversion leaked its scratch disk. The streamed squashfs and the partial extraction were removed only on the success path, so every failed attempt (a truncated archive, a corrupt superblock) left up to ~2× the archive in /tmp on a 4–12 GB disk. Failures now clean up best-effort and rethrow the original error.
  6. A conversion error could persist a credential. The durable migration:convert-status record and the failure log stored error.message verbatim; SDK/transport errors can embed a signed URL or a header value. redactSecrets (known env values, ≥ 8 chars, longest-first) now scrubs both before they are written.

Third run: billing + client isolation before the develop merge (2026-11-16)

A third pass targeted the three workflows that must not remain a security or consistency risk when the v1-sandbox track merges: data consistency, billing, and client isolation. It lives in apps/fouria/src/sandbox-v1-critical-adversarial.test.ts (17 deterministic cases) and complements the existing persistence fuzz and DO lifecycle suites.

PipelineAttackInvariant assertedResult
BillingA legacy worker with only WORKER_URL sends both a trace and a metric/v1/metrics and /v1/traces attribute to the same tenant, never splitting it with defaultfixed
BillingA billable instance with CUSTOMER_SLUG='bad slug' posts a metricFails closed with 503 TENANT_UNRESOLVED, exactly like the traces pathfixed
BillingA metric claims a foreign customer through customer.slug / customer_id / customerRejected with 403; no row is writtenfixed
BillingA trace reuses a padded/case/separator/homoglyph variant of the resolved tenantOnly the exact identity emits; padding that normalizes to the authority still emits that tenantholds
BillingOne batch names a billing-frozen customer and one unfrozen customerOnly the unfrozen customer emits; the frozen rows are dropped before any side effectholds
BillingThe same action id is submitted twice in one isolateEmitted once, never double-chargedholds
BillingAn adversarial customer (' OR 1=1, quote, newline, `, path) reaches the cost summaryRejected by the shared sanitizer; no Analytics Engine SQL is issuedholds
BillingA legacy WORKER_URL-only worker reads its cost summaryThe read side resolves the same identity the write side emittedfixed
Client isolationCUSTOMER_SLUG / WORKER_NAME / WORKER_URL carry default or a padded/path formResolves to null, never the sentinelholds
Client isolationPlatform/environment affixes (staging-, -dev) are stripped once; loopback/IP URLs are refusedExactly one affix removed; local instances have no tenant identityholds
Client isolationThe global leads bucket is reached through the bare fixed sandbox id (openclaw) with no env identityRefused (400); a global bucket must never use the container id as a tenant namespacefixed
Client isolationPer-tenant prefixes for lookalike slugs (a, acme, acme-corp, acme.corp)Pairwise disjoint; no prefix relationshipholds
Data consistencyA crafted record id (../victim, a/b, ../../openclaw/stable) is cataloguedPercent-encoded; the key stays under the tenant prefixholds
Data consistencyRetention runs across 1–12 records × 1–5 versionsThe just-committed recovery point is never pruned; the catalog stays boundedholds
Data consistencyLeads keys/prefixes carry traversal (../, .., //)Rejected; a relative key can never escape the tenant rootholds

Fixes landed with the suite

  1. One OTLP identity authority. resolveCostIdentity(env) (src/sandbox-tenant.ts) is now the single resolver for /v1/metrics, /v1/traces, and the cost-summary read. It prefers the explicit trusted CUSTOMER_SLUG (or the WORKER_NAME it mirrors), then falls back to the routing resolver for a legacy WORKER_URL-only worker. A billable instance with no resolvable identity answers 503 TENANT_UNRESOLVED; a metric that names a foreign customer through any tenant key answers 403.
  2. Global leads bucket refuses the fixed-id fallback. The leads bucket is shared across tenants, so openclaw can never be a tenant namespace. When no env identity resolves, the bridge answers 400 instead of scoping reads and writes to openclaw/.

Residual (documented, not changed). resolveTenantSlug still accepts a bare container id as a last-resort slug for the per-tenant docs/software bridges (their buckets are per-tenant, so the fallback is harmless there). Only the global leads bridge needed the refusal. The migration paths themselves are already barred from the context fallback by contract C10.