Snapshot step budget — evaluating the remaining overrun risk
On this page
Date: 2026-10-22
Context: follow-up to the checkpointed-jobs change (docs/plans/durable-jobs-availability-audit.md, P0 #4) and its remaining risk: the archive and finalize steps are single alarm invocations with no explicit timeout.
1. The risk, precisely scoped
After the checkpointed-jobs change, every durable job runs one bounded step per
alarm invocation. The platform budget for an alarm invocation is a 15-minute
wall limit (Durable Objects: Alarm handler invocations have a maximum wall
time of 15 minutes), with 30 s of active CPU by default (5 min if
limits.cpu_ms is raised).
| Step | Worst case today | Bounded? |
|---|---|---|
prepare (stage) | LEGACY_STAGE_TIMEOUT_MS = 10 min, maxAttempts: 1 inside a durable step | yes |
archive (createBackup + F4E1 encrypt) | container squashfs + R2 upload + Worker-side whole-object encryption | no |
finalize (config capture + promote + prune) | one small file write, one pointer write, ≤ maxVersions R2 deletes | effectively yes (< 1 min) |
restore | RESTORE_BACKUP_TIMEOUT_MS = 3 min | yes |
launch | STARTUP_TIMEOUT_MS = 5 min | yes |
stop | container stop | yes |
⇒ The risk is concentrated in exactly one step: archive. finalize was
suspected but is bounded by small R2 operations; it does not need splitting.
createBackup has no timeout parameter in the Sandbox SDK
(BackupOptions = dir, name, ttl, gitignore, excludes, compression,
multipart, localBucket), so the caller can only bound it by racing it.
2. What actually happens when an archive step is killed
Verified against the installed SDK (@cloudflare/sandbox-legacy):
doCreateBackup():
backupId = crypto.randomUUID()
archivePath = /container-server/<id>.sqsh (container side)
r2Key = backups/<id>/data.sqsh
metaKey = backups/<id>/meta.json
await control.backup.createArchive(...) (container: mksquashfs)
await uploadBackupMultipart/Presigned(...) (container → R2, presigned)
await bucket.put(metaKey, {... name, sizeBytes, createdAt ...}) (Worker side)
Consequences:
| Kill point | R2 state | Reconcile behaviour | Cost |
|---|---|---|---|
| during squashfs | nothing | invisible | container .sqsh may linger |
| during upload | multipart parts only, no object | invisible (no meta.json) | parts billed until R2 aborts them (7 days) |
after upload, before meta.json | backups/<id>/data.sqsh without meta.json | invisible — the reconciler keys on meta.json | permanent leak: never catalogued, never pruned |
after meta.json, before our encryption | archive + meta.json (no encrypted: true) | adopted into the catalog | an unencrypted archive becomes listed and restorable |
| during encryption | data.sqsh.enc truncated (the code verifies the length and aborts, deleting the truncated object) | archive still plain | the step retries |
So a killed archive step is not a corruption risk — it is wasted work
plus two leaks — and the retry re-uploads from scratch, because the SDK only
writes meta.json (and therefore only makes the id discoverable) after a
successful upload.
Two defects found while evaluating (independent of the timing risk)
D1 — an unencrypted archive can be adopted and listed. reconcileOrphanedBackups
(src/persistence.ts) reads only dir and createdAt from meta.json; it never
checks the encrypted flag that our F4E1 step writes. An archive whose upload
completed but whose encryption never ran is therefore merged into the catalog
with encrypted unset. The restore path then decrypts only when the flag is set
(src/persistence.ts:1301-1311), so the archive restores without at-rest
encryption. That is a compliance hole in the F4E1 promise (“encrypted at
rest”), independent of any timeout.
D2 — an archive without meta.json leaks forever. Retention
(cleanupOldBackups) deletes only catalogued entries, and the reconciler only
sees directories that have a meta.json. The “uploaded but not catalogued”
object is invisible to both, so it is never pruned and never counted.
D3 — a killed step stalls for the rest of the lease. The job lease is
now + 14 min (BACKUP_JOB_LEASE_MS, BOOT_JOB_LEASE_MS,
MAINTENANCE_JOB_LEASE_MS), but the alarm is killed at 15 min. If a step dies
at minute 3 (CPU limit, transport drop, eviction), the record is still
running with a live lease, so the next tick waits up to 11 more minutes
before retrying. The lease must be sized to the step, not to the old
whole-operation model.
3. Options
| # | Option | Effect | Effort | Risk |
|---|---|---|---|---|
| 1 | Lease = step budget (e.g. 6 min instead of 14) | D3 gone: a killed step retries in minutes | S | low — a step is idempotent; the lease only guards a duplicate run |
| 2 | Encryption-aware reconcile: never adopt an unencrypted orphan when a master key is configured; mark it needs-encryption and let the durable job encrypt it (or delete it) | D1 closed; at-rest encryption becomes enforced | S–M | low |
| 3 | Sweep for meta-less objects (backups/*/data.sqsh without a sibling meta.json) and delete them | D2 closed; stops the storage/billing leak | S | low (delete only objects with no metadata) |
| 4 | Per-step deadline (step.deadlineAt) + a distinct outcome-unknown state instead of a retry-from-zero | a killed step is labelled, not silently retried; the UI/alert can distinguish “slow” from “failed” | S–M | low |
| 5 | Split archive into archive (upload) + encrypt (F4E1) | the worst case per invocation becomes max(upload, encrypt) instead of upload + encrypt; the CPU-bound part is isolated from the IO-wait part | M | medium — creates a plaintext window between the two steps, so it must ship with (2) |
| 6 | Adopt-by-name correlation: pass createBackup({ name: jobId }); after a kill, scan backups/*/meta.json for name === jobId and adopt the id instead of re-uploading | a completed upload is never re-done (saves the 350 MB re-upload) | M | medium — meta.json is only written on success, so adoption is only possible when the upload completed; a failed upload still re-uploads (correct) |
| 7 | Chunked, resumable encryption: persist the F4E1 base nonce + a chunk cursor in the job record and encrypt N MiB per invocation | the encryption step becomes bounded and resumable; removes the last unbounded Worker-side work | L (2–3 days) | medium-high — touches the crypto format; the base nonce must be persisted (a random per-call nonce cannot be resumed) and must never be reused for different plaintext |
| 8 | Own chunked upload (container mksquashfs + Worker-side multipart with a persisted offset), replacing sandbox.createBackup | the upload becomes bounded and resumable; the wall risk disappears entirely | XL (≥ 1 week) | high — reimplements the SDK backup path, incl. local-bucket mode and presigned-URL handling |
| 9 | Control-plane Workflow (audit §2, option B) driving the phases over the dispatch binding | unlimited per-step wall time, native retries/observability | L/XL | high — new deployable, phase protocol, and the tenant worker still has to await the container RPC somewhere |
| 10 | limits.cpu_ms = 300000 on the tenant script | raises the CPU budget from 30 s to 5 min for the encryption step | XS if the WFP upload accepts it (unverified — the dispatch upload metadata currently sets only main_module/bindings/compatibility_*/migrations/containers/observability) | unknown until verified |
Side finding: the local cctr was not the pinned fork
The repo pins the Paso4 fork 0xCAB0/cctr 0.31.0-paso4.1 (CI + Dockerfile),
but the local machine had stock cctr 0.31.0 installed. Stock cctr stops
collecting tests at a block whose expected output is followed by an empty
where section (a trailing ---): everything after that block silently
disappears from the run. Three corpus files had such a block, so a local run
executed 235 checks instead of 471 — and the local run was the only place the
gap showed up, because CI always used the fork.
Two fixes:
- The corpus no longer uses the truncating shape: the affected blocks carry an
explicit constraint (
{{ count }}+where * count > 0), so the fork and stock cctr now run the same 471 checks. scripts/check-corpus-comments.mjs— the CI guard that already caught the top-level#comment quirk — now also catches the emptywhereshape, so a file can never silently lose its tail again.
The corpus README documents the fork pin (with the SHA-verified install command) and both format gotchas.
Rejected
- Raise the alarm wall: not configurable (15 min is the platform maximum).
- Move the archive into a request: a request’s wall time is unlimited while the client is connected, but the client here is the platform’s own durable job — the request would die with the same alarm.
- Retries with a longer backoff: does not address a step that legitimately needs more than 15 min; it only delays the inevitable.
4. Measure before choosing the budget
The numbers needed to size the step budget are already collected:
r2-operationAnalytics Engine rows carrydurationMs(thecreateBackupduration) andbytesTransferred(the encrypted archive size) per snapshot.lastSnapshotDurationMs(new, in the catalog) carries the end-to-end duration.
So the per-tenant p95/p99 of createBackup is queryable today. Proposed rule:
step_budget = clamp(3 × p99(createBackup + encrypt), 6 min, 12 min)
with 12 min as the hard ceiling (3 min of headroom under the 15-minute wall).
The per-step duration should also be written to the job record
(cursor.startedAt is already there) and emitted as a structured event, so a
tenant that approaches the budget is visible before it starts failing.
5. Recommended plan
Phase 1 — hardening (small, high value, no format change) — implemented 2026-10-22
- (1) Lease = step budget:
BACKUP_JOB_LEASE_MS/BOOT_JOB_LEASE_MS/MAINTENANCE_JOB_LEASE_MSare nowSTEP_LEASE_MS(6 min) — the lease guards a duplicate step, not a whole operation. - (2) Encryption-aware reconcile: while a master key is configured, an archive
whose
meta.jsonlacksencrypted: trueis catalogued withunencrypted: trueand queued inpendingEncryption. It is not dropped (it stays restorable — dropping it would lose a recovery point) and it is never promoted to the recovery point. The durablerepairstep encrypts it in place. - (3) Sweep
backups/*for archive objects without a siblingmeta.jsonand delete them — they are invisible to the reconciler (which keys onmeta.json) and to the retention prune (which only knows catalogued entries), so they would leak forever. - (4) Per-step deadline: an interrupted step is reported
(
detail: 'step-interrupted', or'step-deadline-exceeded'past the 12-minute soft deadline) with ajob-step.interruptedevent, and every step emitsjob-step.completed/job-step.failedwith itsdurationMs— the measurement the step budget is derived from.
Phase 2 — bounding the step (medium)
- (5) Split
archiveintoarchive(upload) +encrypt(F4E1), shipping with (2) so the plaintext window is never adopted. - (6) Adopt-by-name correlation (
name: jobId) so an interrupted upload that actually completed is adopted instead of re-uploaded. - Archive-size reduction: pass SDK
excludesfor regenerable paths (node_modules,*.log, caches) and prune the container before staging, so the upload (and the encryption) shrink. Measure the effect on the archive size first — the legacy stage must stay inside the archive.
Phase 3 — only if the measured p99 still approaches the budget
- (7) chunked resumable encryption, or (9) the control-plane Workflow.
Verification for Phase 1/2
- Unit: reconcile refuses an unencrypted orphan; the sweeper deletes a
meta-less object but keeps a catalogued one; a killed step is retried after
the (short) lease, not after 14 minutes; the
archive/encryptsplit leaves the catalog consistent when the encryption step fails. - Corpus (
test/business-rules/operational-modes.txt): an unencrypted orphan is never catalogued; a killed step resumes within the step budget. - Staging: force-kill the DO during
archiveand duringencryptand confirm (a) the retry is prompt, (b) no unencrypted archive is ever listed, (c) no object withoutmeta.jsonsurvives.