F FourIA latest 2026.10.29 GitHub ↗

Snapshot step budget — evaluating the remaining overrun risk

On this page

Date: 2026-10-22 Context: follow-up to the checkpointed-jobs change (docs/plans/durable-jobs-availability-audit.md, P0 #4) and its remaining risk: the archive and finalize steps are single alarm invocations with no explicit timeout.


1. The risk, precisely scoped

After the checkpointed-jobs change, every durable job runs one bounded step per alarm invocation. The platform budget for an alarm invocation is a 15-minute wall limit (Durable Objects: Alarm handler invocations have a maximum wall time of 15 minutes), with 30 s of active CPU by default (5 min if limits.cpu_ms is raised).

StepWorst case todayBounded?
prepare (stage)LEGACY_STAGE_TIMEOUT_MS = 10 min, maxAttempts: 1 inside a durable stepyes
archive (createBackup + F4E1 encrypt)container squashfs + R2 upload + Worker-side whole-object encryptionno
finalize (config capture + promote + prune)one small file write, one pointer write, ≤ maxVersions R2 deleteseffectively yes (< 1 min)
restoreRESTORE_BACKUP_TIMEOUT_MS = 3 minyes
launchSTARTUP_TIMEOUT_MS = 5 minyes
stopcontainer stopyes

⇒ The risk is concentrated in exactly one step: archive. finalize was suspected but is bounded by small R2 operations; it does not need splitting.

createBackup has no timeout parameter in the Sandbox SDK (BackupOptions = dir, name, ttl, gitignore, excludes, compression, multipart, localBucket), so the caller can only bound it by racing it.


2. What actually happens when an archive step is killed

Verified against the installed SDK (@cloudflare/sandbox-legacy):

doCreateBackup():
  backupId  = crypto.randomUUID()
  archivePath = /container-server/<id>.sqsh        (container side)
  r2Key     = backups/<id>/data.sqsh
  metaKey   = backups/<id>/meta.json
  await control.backup.createArchive(...)           (container: mksquashfs)
  await uploadBackupMultipart/Presigned(...)        (container → R2, presigned)
  await bucket.put(metaKey, {... name, sizeBytes, createdAt ...})   (Worker side)

Consequences:

Kill pointR2 stateReconcile behaviourCost
during squashfsnothinginvisiblecontainer .sqsh may linger
during uploadmultipart parts only, no objectinvisible (no meta.json)parts billed until R2 aborts them (7 days)
after upload, before meta.jsonbackups/<id>/data.sqsh without meta.jsoninvisible — the reconciler keys on meta.jsonpermanent leak: never catalogued, never pruned
after meta.json, before our encryptionarchive + meta.json (no encrypted: true)adopted into the catalogan unencrypted archive becomes listed and restorable
during encryptiondata.sqsh.enc truncated (the code verifies the length and aborts, deleting the truncated object)archive still plainthe step retries

So a killed archive step is not a corruption risk — it is wasted work plus two leaks — and the retry re-uploads from scratch, because the SDK only writes meta.json (and therefore only makes the id discoverable) after a successful upload.

Two defects found while evaluating (independent of the timing risk)

D1 — an unencrypted archive can be adopted and listed. reconcileOrphanedBackups (src/persistence.ts) reads only dir and createdAt from meta.json; it never checks the encrypted flag that our F4E1 step writes. An archive whose upload completed but whose encryption never ran is therefore merged into the catalog with encrypted unset. The restore path then decrypts only when the flag is set (src/persistence.ts:1301-1311), so the archive restores without at-rest encryption. That is a compliance hole in the F4E1 promise (“encrypted at rest”), independent of any timeout.

D2 — an archive without meta.json leaks forever. Retention (cleanupOldBackups) deletes only catalogued entries, and the reconciler only sees directories that have a meta.json. The “uploaded but not catalogued” object is invisible to both, so it is never pruned and never counted.

D3 — a killed step stalls for the rest of the lease. The job lease is now + 14 min (BACKUP_JOB_LEASE_MS, BOOT_JOB_LEASE_MS, MAINTENANCE_JOB_LEASE_MS), but the alarm is killed at 15 min. If a step dies at minute 3 (CPU limit, transport drop, eviction), the record is still running with a live lease, so the next tick waits up to 11 more minutes before retrying. The lease must be sized to the step, not to the old whole-operation model.


3. Options

#OptionEffectEffortRisk
1Lease = step budget (e.g. 6 min instead of 14)D3 gone: a killed step retries in minutesSlow — a step is idempotent; the lease only guards a duplicate run
2Encryption-aware reconcile: never adopt an unencrypted orphan when a master key is configured; mark it needs-encryption and let the durable job encrypt it (or delete it)D1 closed; at-rest encryption becomes enforcedS–Mlow
3Sweep for meta-less objects (backups/*/data.sqsh without a sibling meta.json) and delete themD2 closed; stops the storage/billing leakSlow (delete only objects with no metadata)
4Per-step deadline (step.deadlineAt) + a distinct outcome-unknown state instead of a retry-from-zeroa killed step is labelled, not silently retried; the UI/alert can distinguish “slow” from “failed”S–Mlow
5Split archive into archive (upload) + encrypt (F4E1)the worst case per invocation becomes max(upload, encrypt) instead of upload + encrypt; the CPU-bound part is isolated from the IO-wait partMmedium — creates a plaintext window between the two steps, so it must ship with (2)
6Adopt-by-name correlation: pass createBackup({ name: jobId }); after a kill, scan backups/*/meta.json for name === jobId and adopt the id instead of re-uploadinga completed upload is never re-done (saves the 350 MB re-upload)Mmedium — meta.json is only written on success, so adoption is only possible when the upload completed; a failed upload still re-uploads (correct)
7Chunked, resumable encryption: persist the F4E1 base nonce + a chunk cursor in the job record and encrypt N MiB per invocationthe encryption step becomes bounded and resumable; removes the last unbounded Worker-side workL (2–3 days)medium-high — touches the crypto format; the base nonce must be persisted (a random per-call nonce cannot be resumed) and must never be reused for different plaintext
8Own chunked upload (container mksquashfs + Worker-side multipart with a persisted offset), replacing sandbox.createBackupthe upload becomes bounded and resumable; the wall risk disappears entirelyXL (≥ 1 week)high — reimplements the SDK backup path, incl. local-bucket mode and presigned-URL handling
9Control-plane Workflow (audit §2, option B) driving the phases over the dispatch bindingunlimited per-step wall time, native retries/observabilityL/XLhigh — new deployable, phase protocol, and the tenant worker still has to await the container RPC somewhere
10limits.cpu_ms = 300000 on the tenant scriptraises the CPU budget from 30 s to 5 min for the encryption stepXS if the WFP upload accepts it (unverified — the dispatch upload metadata currently sets only main_module/bindings/compatibility_*/migrations/containers/observability)unknown until verified

Side finding: the local cctr was not the pinned fork

The repo pins the Paso4 fork 0xCAB0/cctr 0.31.0-paso4.1 (CI + Dockerfile), but the local machine had stock cctr 0.31.0 installed. Stock cctr stops collecting tests at a block whose expected output is followed by an empty where section (a trailing ---): everything after that block silently disappears from the run. Three corpus files had such a block, so a local run executed 235 checks instead of 471 — and the local run was the only place the gap showed up, because CI always used the fork.

Two fixes:

  • The corpus no longer uses the truncating shape: the affected blocks carry an explicit constraint ({{ count }} + where * count > 0), so the fork and stock cctr now run the same 471 checks.
  • scripts/check-corpus-comments.mjs — the CI guard that already caught the top-level # comment quirk — now also catches the empty where shape, so a file can never silently lose its tail again.

The corpus README documents the fork pin (with the SHA-verified install command) and both format gotchas.

Rejected

  • Raise the alarm wall: not configurable (15 min is the platform maximum).
  • Move the archive into a request: a request’s wall time is unlimited while the client is connected, but the client here is the platform’s own durable job — the request would die with the same alarm.
  • Retries with a longer backoff: does not address a step that legitimately needs more than 15 min; it only delays the inevitable.

4. Measure before choosing the budget

The numbers needed to size the step budget are already collected:

  • r2-operation Analytics Engine rows carry durationMs (the createBackup duration) and bytesTransferred (the encrypted archive size) per snapshot.
  • lastSnapshotDurationMs (new, in the catalog) carries the end-to-end duration.

So the per-tenant p95/p99 of createBackup is queryable today. Proposed rule:

step_budget = clamp(3 × p99(createBackup + encrypt), 6 min, 12 min)

with 12 min as the hard ceiling (3 min of headroom under the 15-minute wall). The per-step duration should also be written to the job record (cursor.startedAt is already there) and emitted as a structured event, so a tenant that approaches the budget is visible before it starts failing.


Phase 1 — hardening (small, high value, no format change) — implemented 2026-10-22

  1. (1) Lease = step budget: BACKUP_JOB_LEASE_MS / BOOT_JOB_LEASE_MS / MAINTENANCE_JOB_LEASE_MS are now STEP_LEASE_MS (6 min) — the lease guards a duplicate step, not a whole operation.
  2. (2) Encryption-aware reconcile: while a master key is configured, an archive whose meta.json lacks encrypted: true is catalogued with unencrypted: true and queued in pendingEncryption. It is not dropped (it stays restorable — dropping it would lose a recovery point) and it is never promoted to the recovery point. The durable repair step encrypts it in place.
  3. (3) Sweep backups/* for archive objects without a sibling meta.json and delete them — they are invisible to the reconciler (which keys on meta.json) and to the retention prune (which only knows catalogued entries), so they would leak forever.
  4. (4) Per-step deadline: an interrupted step is reported (detail: 'step-interrupted', or 'step-deadline-exceeded' past the 12-minute soft deadline) with a job-step.interrupted event, and every step emits job-step.completed / job-step.failed with its durationMs — the measurement the step budget is derived from.

Phase 2 — bounding the step (medium)

  1. (5) Split archive into archive (upload) + encrypt (F4E1), shipping with (2) so the plaintext window is never adopted.
  2. (6) Adopt-by-name correlation (name: jobId) so an interrupted upload that actually completed is adopted instead of re-uploaded.
  3. Archive-size reduction: pass SDK excludes for regenerable paths (node_modules, *.log, caches) and prune the container before staging, so the upload (and the encryption) shrink. Measure the effect on the archive size first — the legacy stage must stay inside the archive.

Phase 3 — only if the measured p99 still approaches the budget

  1. (7) chunked resumable encryption, or (9) the control-plane Workflow.

Verification for Phase 1/2

  • Unit: reconcile refuses an unencrypted orphan; the sweeper deletes a meta-less object but keeps a catalogued one; a killed step is retried after the (short) lease, not after 14 minutes; the archive/encrypt split leaves the catalog consistent when the encryption step fails.
  • Corpus (test/business-rules/operational-modes.txt): an unencrypted orphan is never catalogued; a killed step resumes within the step budget.
  • Staging: force-kill the DO during archive and during encrypt and confirm (a) the retry is prompt, (b) no unencrypted archive is ever listed, (c) no object without meta.json survives.