Sandbox v1 adversarial + fuzz suites
On this page
Two seeded suites attack the Sandbox v1 (Sandbox SDK 1.0) critical pipelines instead of exercising curated examples. Every random case is reproducible from its seed, and a failure message names the seed and the case index.
| Suite | Scope |
|---|---|
apps/fouria/src/sandbox-migration/fuzz.test.ts | Pure state machines (no DO): tri-state reads, validators, C8 barrier, retention, lease, runtime marker, intent gate, catalog listing |
apps/fouria/src/sandbox-v1-adversarial.test.ts | The SandboxV1 Durable Object: encryption, secrets, snapshots, cadence, boot, process map, transport |
bun run --cwd apps/fouria test src/sandbox-migration/fuzz.test.ts
bun run --cwd apps/fouria test src/sandbox-v1-adversarial.test.tsAttack matrix
| Pipeline | Attack | Invariant asserted | Result |
|---|---|---|---|
| Zero data loss | Random pointer / catalog record / conversion-absent / malformed / transport-error combinations (1 000 cases) | verifyRecoveryPoint agrees with an independent oracle; unknown never falls through to a legacy read | holds |
| Zero data loss | Random retention sizes, duplicate timestamps, shuffled bucket order (1 000 cases) | The protected record is never pruned, pruning is order-independent and deduplicated | holds |
| Zero data loss | SDK backup / pointer write / post-encryption failures (60 fuzz errors) | A failed snapshot never advances stable.json; the previous point stays complete | holds |
| Zero data loss | Simulated stopped container (running = false) | Snapshot returns skipped, never starts the container, never archives an empty home | holds |
| Zero data loss | Catalog listing reports truncation with a repeated/absent cursor | Listing is bounded (page cap + non-advancing-cursor stop) | fixed |
| Zero data loss | Post-commit catalog listing/retention failure | The snapshot is still reported committed (the point already advanced) | fixed |
| Zero data loss | Recovery point dated in the future (clock skew / corruption) | The idle cadence keeps snapshotting instead of suppressing the loss bound forever | fixed |
| Zero data loss | Random 80-snapshot sequence with changing BACKUP_MAX_VERSIONS and injected failures | Every committed point is restorable; retention stays bounded; a failure never moves it | holds |
| Secrets | Encryption-required × absent/empty/malformed/short/valid key matrix | Required + unusable key uploads nothing; a broken optional key is never ignored | holds |
| Secrets | 60 random rejected key materials | The key never appears in a thrown error or any console log | holds |
| Secrets | Boot attempt with sentinel token/key env | Container env is never logged | holds |
| Secrets | Path-ish tenant names (../../etc/passwd, %2e%2e, …) | The backup prefix stays one encoded key segment | holds |
| Memory | MAX_TRACKED_PROCESSES + 144 execs | The DO process map is bounded; exited handles are dropped oldest-first | fixed |
| Memory | One still-running handle plus a flood of exited handles | A running handle is never dropped (gateway stays killable) | holds |
| Boot | Conversion record present without its stable pointer | Boot refuses to onboard over converted state; never launches the setup body | holds |
| Boot | 40 corrupt stable.json bodies | Boot refuses on an unreadable pointer instead of onboarding | holds |
| Boot | Brand-new tenant (no record, no pointer) | Launch runs without OPENCLAW_RESTORED (the setup body owns the state) | holds |
| Boot | Malformed X-FourIA-Sandbox-Port (1e3, 0x50, 3.5, …) | Rejected with 400 before any container access | fixed |
| Boot | Stopped container request | 503 SANDBOX_V1_SLEEPING, never wakes the container | holds |
| Tenant isolation | Runtime marker copied under another tenant’s scope; intent gate random states | Marker authority is scope-bound; an intent never falls back to legacy | holds |
| Secrets | Snapshot with sentinel gateway token / AI key / backup key in env | No R2 object written by the snapshot contains a secret value | fixed (scan) |
| Secrets | Conversion failure whose error embeds a credential (signed-URL shape) | The persisted status and the log contain [redacted], never the value | fixed |
| Memory | A process whose output() returns 3× the stream cap | The decoded output is truncated at MAX_PROCESS_OUTPUT_BYTES and flagged truncated | fixed |
| Zero data loss | A snapshot taken while an operator restore is swapping the directory | skipped: 'restoring', no archive, the previous point stays | fixed |
| Zero data loss | A restore that fails mid-swap, then a retry | The point never moves on failure and the in-progress barrier is cleared | holds |
| Zero data loss | A second restore (and a boot) while a restore is in flight | Refused; the boot never launches over the swap | holds |
| Zero data loss | 12 snapshots → 24 randomized restores of surviving records → boot the point (fuzz) | Every surviving record restores, the point matches, and the boot skips the setup body | holds |
| Memory (disk) | A conversion whose extractor / upload fails | The streamed archive and partial extraction are removed best-effort; the error wins | fixed |
| Boot | start-openclaw.sh start exits 0 with no gateway process anywhere | The boot fails after the zero-exit grace instead of waiting the whole cold-start budget | fixed |
| Boot | start-openclaw.sh start exits 0 while the lock winner is still booting | The DO keeps waiting and adopts the peer’s readiness | holds |
Defects found and fixed
- Unbounded catalog listing loop (OOM).
listRecordEntriesfollowedpage.cursorwithout bounding the iteration. A bucket (or a proxy) that reportstruncated: truewith a non-advancing cursor made the loop read the same page forever; the fuzz run reproduced it as a realFATAL ERROR: Ineffective mark-compacts near heap limitin the test worker. Fixed withMAX_RECORD_LIST_PAGES, repeated-cursor detection and per-recordIddeduplication — an incomplete list can only retain more records, which is the safe direction for retention (C6). - A committed snapshot reported as failed.
v1CreateSnapshotwrote the catalog entry and advancedstable.json, then let a listing/retention error escape, so the caller saw a failed snapshot whose recovery point had in fact moved (and retention never ran). Retention is now best-effort after the commit and logssandbox.v1.snapshot.retention_failed. - Future-dated recovery point suppressed the cadence forever. The dedupe
test was
Date.now() - committedAt < interval, which is true for any future timestamp. It now only dedupes a non-negative age inside the interval. - Unbounded process-handle map. Every
v1Execretained itsExecProcesshandle for the DO’s lifetime. The map is now capped atMAX_TRACKED_PROCESSES(256), dropping exited handles oldest-first and never dropping a running one. - Port-header coercion. The DO transport accepted
Number()coercions (1e3,0x50). The header must now be canonical decimal digits before the range check.
Observations left as code, not changed here
- Conversion disk amplification.
materializeLegacyArchivewrites the archive to/tmp/fouria-legacy.sqsh, extracts it to/tmp/fouria-legacy-extract, then copies the staged roots into/home/openclaw. Peak disk is roughly twice the archive plus the restored home (~3× archive) on a 4–12 GB container. A preflight free-space check (or extracting through a pipe) is the follow-up; it is a deploy/runtime decision, not a pure invariant. - Process cap vs. running handles. The bound only evicts exited handles. A pathological burst of still-running processes can exceed it rather than losing the ability to kill the gateway.
- Backup key scope.
directoryBackupPrefixis derived from the DO id (sandbox-v1/openclaw/for the fixed production container id). Tenant isolation comes from the per-tenantfouria-backup-<slug>bucket, not from the key; keep that binding invariant when changing provisioning.
Second run: critical-pipeline hardenings (2026-11-09)
The second attack pass targeted the four incident pipelines again, this time around the restore/snapshot/boot interactions and persisted text:
- A snapshot could race an operator restore.
v1CreateSnapshotonly checked the container’s liveness; the worker’s maintenance window prevented the overlap for route-driven calls, but the primitive itself would archive a half-swapped directory and then advance the recovery point to it.v1CreateSnapshotnow returnsskipped: 'restoring'while#restoring. - A boot could launch over a swap.
startGatewayOwnerchecked#restoringonly at entry (throughv1EnsureGateway); a restore that began while the state probe ran still launched. The owner now re-checks before the launch and inside the readiness loop, and reports a retryable failure instead of readiness for a gateway the restore is about to kill. - Unbounded process output.
v1ProcessOutputdecoded whatever the process wrote, after the SDK had already buffered it in the DO isolate. Reads are now truncated atMAX_PROCESS_OUTPUT_BYTES(1 MiB per stream) withtruncated: true. The SDK’s own buffer is still bounded only by the command, so callers must keep shell commands bounded (the system does:tail -n 200, JSON CLI output). - Zero-exit boots burned the full cold-start budget. A wrapper that exits 0
(the lost-startup-lock path) was only judged by the health poll, so a launch
nobody owned waited
timeoutMs(up to 5 min). AfterSANDBOX_V1_ZERO_EXIT_GRACE_MS(default 15s,0disables) the DO proves no gateway (or setup wrapper) process exists (pgrep -fon neither) and fails fast; a booting peer (lock winner) resets the grace and is adopted normally. - A failed conversion leaked its scratch disk. The streamed squashfs and
the partial extraction were removed only on the success path, so every failed
attempt (a truncated archive, a corrupt superblock) left up to ~2× the archive
in
/tmpon a 4–12 GB disk. Failures now clean up best-effort and rethrow the original error. - A conversion error could persist a credential. The durable
migration:convert-statusrecord and the failure log storederror.messageverbatim; SDK/transport errors can embed a signed URL or a header value.redactSecrets(known env values, ≥ 8 chars, longest-first) now scrubs both before they are written.
Third run: billing + client isolation before the develop merge (2026-11-16)
A third pass targeted the three workflows that must not remain a security or
consistency risk when the v1-sandbox track merges: data consistency,
billing, and client isolation. It lives in
apps/fouria/src/sandbox-v1-critical-adversarial.test.ts (17 deterministic
cases) and complements the existing persistence fuzz and DO lifecycle suites.
| Pipeline | Attack | Invariant asserted | Result |
|---|---|---|---|
| Billing | A legacy worker with only WORKER_URL sends both a trace and a metric | /v1/metrics and /v1/traces attribute to the same tenant, never splitting it with default | fixed |
| Billing | A billable instance with CUSTOMER_SLUG='bad slug' posts a metric | Fails closed with 503 TENANT_UNRESOLVED, exactly like the traces path | fixed |
| Billing | A metric claims a foreign customer through customer.slug / customer_id / customer | Rejected with 403; no row is written | fixed |
| Billing | A trace reuses a padded/case/separator/homoglyph variant of the resolved tenant | Only the exact identity emits; padding that normalizes to the authority still emits that tenant | holds |
| Billing | One batch names a billing-frozen customer and one unfrozen customer | Only the unfrozen customer emits; the frozen rows are dropped before any side effect | holds |
| Billing | The same action id is submitted twice in one isolate | Emitted once, never double-charged | holds |
| Billing | An adversarial customer (' OR 1=1, quote, newline, `, path) reaches the cost summary | Rejected by the shared sanitizer; no Analytics Engine SQL is issued | holds |
| Billing | A legacy WORKER_URL-only worker reads its cost summary | The read side resolves the same identity the write side emitted | fixed |
| Client isolation | CUSTOMER_SLUG / WORKER_NAME / WORKER_URL carry default or a padded/path form | Resolves to null, never the sentinel | holds |
| Client isolation | Platform/environment affixes (staging-, -dev) are stripped once; loopback/IP URLs are refused | Exactly one affix removed; local instances have no tenant identity | holds |
| Client isolation | The global leads bucket is reached through the bare fixed sandbox id (openclaw) with no env identity | Refused (400); a global bucket must never use the container id as a tenant namespace | fixed |
| Client isolation | Per-tenant prefixes for lookalike slugs (a, acme, acme-corp, acme.corp) | Pairwise disjoint; no prefix relationship | holds |
| Data consistency | A crafted record id (../victim, a/b, ../../openclaw/stable) is catalogued | Percent-encoded; the key stays under the tenant prefix | holds |
| Data consistency | Retention runs across 1–12 records × 1–5 versions | The just-committed recovery point is never pruned; the catalog stays bounded | holds |
| Data consistency | Leads keys/prefixes carry traversal (../, .., //) | Rejected; a relative key can never escape the tenant root | holds |
Fixes landed with the suite
- One OTLP identity authority.
resolveCostIdentity(env)(src/sandbox-tenant.ts) is now the single resolver for/v1/metrics,/v1/traces, and the cost-summary read. It prefers the explicit trustedCUSTOMER_SLUG(or theWORKER_NAMEit mirrors), then falls back to the routing resolver for a legacyWORKER_URL-only worker. A billable instance with no resolvable identity answers503 TENANT_UNRESOLVED; a metric that names a foreign customer through any tenant key answers403. - Global leads bucket refuses the fixed-id fallback. The leads bucket is
shared across tenants, so
openclawcan never be a tenant namespace. When no env identity resolves, the bridge answers400instead of scoping reads and writes toopenclaw/.
Residual (documented, not changed). resolveTenantSlug still accepts a
bare container id as a last-resort slug for the per-tenant docs/software
bridges (their buckets are per-tenant, so the fallback is harmless there). Only
the global leads bridge needed the refusal. The migration paths themselves are
already barred from the context fallback by contract C10.