Sandbox v1 / F4E1 adversarial & fuzz report
On this page
Status: adversarial coverage added; four hardening changes landed on
v1-sandbox, one of which also applies to the legacy runtime and must ship ondevelop. This note covers the attacks run againstSandboxV1(apps/fouria/src/sandbox-v1.ts) and the shared at-rest codec (apps/fouria/src/crypto/backup-encryption.ts), the defects found, and which findings apply to the legacySandboxpath.
How to run
# Codec / lifecycle adversarial suite (deterministic, part of `bun run test`)
bun run --cwd apps/fouria test src/sandbox-v1-codec-adversarial
# The broader DO adversarial suite (recovery point, boot refusal, process map)
bun run --cwd apps/fouria test src/sandbox-v1-adversarial
# Seeded fuzz harness over the F4E1 codec (many more shapes)
bun run --cwd apps/fouria test:fuzz:sandbox-v1 # 500 iterations, fixed seed
bun run --cwd apps/fouria test:fuzz:sandbox-v1 5000 987654
The harness exits non-zero only when an invariant breaks; documented weaknesses
are printed as WARN with the seed so a finding is reproducible.
Findings and applicability
| # | Severity | Defect | Applies to | Fix |
|---|---|---|---|---|
| 1 | critical (data loss) | startContainer() destroyed the container on any setup failure, including a container that was already running (DO restart against a live container, or a retry). A transient intercept()/setInactivityTimeout() error destroyed tenant filesystem state. | SandboxV1 only (legacy ensureGateway kills the gateway process, never the live container) | Destroy only a container this call started (isNewContainer). |
| 2 | high (corruption) | ensureRunning() guarded on #setup === null || !container.running, and container.running is not guaranteed to flip inside start(). A burst of first requests could call startContainer() several times: duplicate start() and a second intercept(), which the SDK documents as replacing a running backup operation’s grant. | SandboxV1 only (legacy boot is owned by start-openclaw.sh’s container lock + the Worker’s per-key start promise) | Single in-flight attempt (#starting); concurrent callers join it, and only the initiator clears #setup on failure. |
| 3 | high (data loss) | Concurrent restores of one record shared the transient plaintext key: one restore could delete it while the other’s SDK restore was still queued, failing an intact backup with BACKUP_NOT_FOUND. | SandboxV1 only (legacy restores are serialized by the shared R2 backup-lock lease) | Per-record restore queue (withRestoreSlot). |
| 4 | high (OOM) | An F4E1 header accepted a chunk size up to 256 MiB. Decryption buffers chunkSize + tag before authenticating it, so a corrupt or maliciously written object could make a restore buffer ~256 MiB and OOM the Worker mid-recovery — far above the isolate memory limit. | Shared: SandboxV1 and legacy (r2-backup.ts decryptBackupArchive decrypts .sqsh.enc with the same decryptStream/chunkSizeFromHeader) | MAX_CHUNK_SIZE (16 MiB) enforced on read; encryptBytes clamps its single-chunk size so it can never emit a header a reader rejects. |
Regression evidence: with sandbox-v1.ts at its pre-fix logic, the lifecycle /
restore-race tests fail; with the fixes they pass. The crypto bound has explicit
unit tests (default accepted, MAX_CHUNK_SIZE accepted, 0 and
MAX_CHUNK_SIZE + 1 rejected, a 256 MiB header rejected from the 17 header
bytes alone).
Conclusion for develop (legacy)
Only finding 4 exists in the runtime develop serves. develop carries the
same src/crypto/backup-encryption.ts, and the legacy restore path
(src/crypto/r2-backup.ts#decryptBackupArchive) calls chunkSizeFromHeader +
decryptStream on every encrypted recovery point. A corrupted/forged
data.sqsh.enc header can therefore OOM the Worker exactly when a tenant is
being recovered. Findings 1–3 are v1-only code paths and stay on this branch.
Documented weaknesses (not changed)
- Chunk-boundary truncation is length-preserving. AES-GCM authenticates
each chunk but F4E1 has no total-length commitment, so truncating a
multi-chunk ciphertext exactly on a chunk boundary decrypts to a shorter
plaintext without error.
SandboxV1is protected bywithDecryptedDirectoryBackup’sdecryptedLength(...) === record.sizecheck before the SDK restore (covered by a test); legacy.sqsh.enchas no recorded plaintext length, so a truncated archive reachessandbox.restoreBackup()and the squashfs/size validation there is the only guard. A future format version should authenticate a total length/EOF marker. - A container replaced underneath a live DO is not re-instrumented. If the
platform restarts the container while the DO incarnation survives,
container.runningstays true, soensureRunning()short-circuits and the new container keeps the intercepts registered by the previous instance (the SDK drops per-instance outbound intercepts). The DO cannot observe a container generation through the 1.0 surface; the cutover should add an identity probe. - Off-workerd uploads buffer the object.
putKnownLengthStreamfalls back tocollectStreamwhenFixedLengthStreamis absent (Node tests only). Production (workerd) takes the streamed path, which the suite asserts. - The idle policy trusts
WORKER_URL. Fuzz over arbitrary hostnames, dev-mode values, and durations never produced a non-positive/NaN/over-6h timeout, and billable production always resolves the keep-alive maximum; a malformed hostname degrades to keep-alive (the safe direction).