# Snapshot step budget — evaluating the remaining overrun risk

**Date:** 2026-10-22
**Context:** follow-up to the checkpointed-jobs change (`docs/plans/durable-jobs-availability-audit.md`, P0 #4) and its remaining risk: the `archive` and `finalize` steps are single alarm invocations with no explicit timeout.

---

## 1. The risk, precisely scoped

After the checkpointed-jobs change, every durable job runs **one bounded step per
alarm invocation**. The platform budget for an alarm invocation is a **15-minute
wall limit** (Durable Objects: _Alarm handler invocations have a maximum wall
time of 15 minutes_), with **30 s of active CPU** by default (5 min if
`limits.cpu_ms` is raised).

| Step                                          | Worst case today                                                           | Bounded?                  |
| --------------------------------------------- | -------------------------------------------------------------------------- | ------------------------- |
| `prepare` (stage)                             | `LEGACY_STAGE_TIMEOUT_MS` = 10 min, `maxAttempts: 1` inside a durable step | yes                       |
| `archive` (createBackup + F4E1 encrypt)       | container squashfs + R2 upload + Worker-side whole-object encryption       | **no**                    |
| `finalize` (config capture + promote + prune) | one small file write, one pointer write, ≤ `maxVersions` R2 deletes        | effectively yes (< 1 min) |
| `restore`                                     | `RESTORE_BACKUP_TIMEOUT_MS` = 3 min                                        | yes                       |
| `launch`                                      | `STARTUP_TIMEOUT_MS` = 5 min                                               | yes                       |
| `stop`                                        | container stop                                                             | yes                       |

⇒ **The risk is concentrated in exactly one step: `archive`.** `finalize` was
suspected but is bounded by small R2 operations; it does not need splitting.

`createBackup` has **no timeout parameter** in the Sandbox SDK
(`BackupOptions` = `dir`, `name`, `ttl`, `gitignore`, `excludes`, `compression`,
`multipart`, `localBucket`), so the caller can only bound it by racing it.

---

## 2. What actually happens when an `archive` step is killed

Verified against the installed SDK (`@cloudflare/sandbox-legacy`):

```
doCreateBackup():
  backupId  = crypto.randomUUID()
  archivePath = /container-server/<id>.sqsh        (container side)
  r2Key     = backups/<id>/data.sqsh
  metaKey   = backups/<id>/meta.json
  await control.backup.createArchive(...)           (container: mksquashfs)
  await uploadBackupMultipart/Presigned(...)        (container → R2, presigned)
  await bucket.put(metaKey, {... name, sizeBytes, createdAt ...})   (Worker side)
```

Consequences:

| Kill point                               | R2 state                                                                                           | Reconcile behaviour                                | Cost                                                     |
| ---------------------------------------- | -------------------------------------------------------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------------- |
| during squashfs                          | nothing                                                                                            | invisible                                          | container `.sqsh` may linger                             |
| during upload                            | multipart **parts** only, no object                                                                | invisible (no `meta.json`)                         | parts billed until R2 aborts them (7 days)               |
| after upload, before `meta.json`         | `backups/<id>/data.sqsh` **without** `meta.json`                                                   | **invisible** — the reconciler keys on `meta.json` | **permanent leak**: never catalogued, never pruned       |
| after `meta.json`, before our encryption | archive + `meta.json` (no `encrypted: true`)                                                       | **adopted into the catalog**                       | an **unencrypted** archive becomes listed and restorable |
| during encryption                        | `data.sqsh.enc` truncated (the code verifies the length and aborts, deleting the truncated object) | archive still plain                                | the step retries                                         |

So a killed `archive` step is **not** a corruption risk — it is _wasted work_
plus two leaks — and the retry re-uploads from scratch, because the SDK only
writes `meta.json` (and therefore only makes the id discoverable) after a
successful upload.

### Two defects found while evaluating (independent of the timing risk)

**D1 — an unencrypted archive can be adopted and listed.** `reconcileOrphanedBackups`
(`src/persistence.ts`) reads only `dir` and `createdAt` from `meta.json`; it never
checks the `encrypted` flag that our F4E1 step writes. An archive whose upload
completed but whose encryption never ran is therefore merged into the catalog
with `encrypted` unset. The restore path then decrypts only when the flag is set
(`src/persistence.ts:1301-1311`), so the archive restores **without at-rest
encryption**. That is a compliance hole in the F4E1 promise ("encrypted at
rest"), independent of any timeout.

**D2 — an archive without `meta.json` leaks forever.** Retention
(`cleanupOldBackups`) deletes only catalogued entries, and the reconciler only
sees directories that have a `meta.json`. The "uploaded but not catalogued"
object is invisible to both, so it is never pruned and never counted.

**D3 — a killed step stalls for the rest of the lease.** The job lease is
`now + 14 min` (`BACKUP_JOB_LEASE_MS`, `BOOT_JOB_LEASE_MS`,
`MAINTENANCE_JOB_LEASE_MS`), but the alarm is killed at 15 min. If a step dies
at minute 3 (CPU limit, transport drop, eviction), the record is still
`running` with a live lease, so the next tick **waits up to 11 more minutes**
before retrying. The lease must be sized to the _step_, not to the old
whole-operation model.

---

## 3. Options

| #   | Option                                                                                                                                                                          | Effect                                                                                                                                           | Effort                                                                                                                                                                                   | Risk                                                                                                                                                                   |
| --- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 1   | **Lease = step budget** (e.g. 6 min instead of 14)                                                                                                                              | D3 gone: a killed step retries in minutes                                                                                                        | S                                                                                                                                                                                        | low — a step is idempotent; the lease only guards a duplicate run                                                                                                      |
| 2   | **Encryption-aware reconcile**: never adopt an unencrypted orphan when a master key is configured; mark it `needs-encryption` and let the durable job encrypt it (or delete it) | D1 closed; at-rest encryption becomes enforced                                                                                                   | S–M                                                                                                                                                                                      | low                                                                                                                                                                    |
| 3   | **Sweep for meta-less objects** (`backups/*/data.sqsh` without a sibling `meta.json`) and delete them                                                                           | D2 closed; stops the storage/billing leak                                                                                                        | S                                                                                                                                                                                        | low (delete only objects with no metadata)                                                                                                                             |
| 4   | **Per-step deadline** (`step.deadlineAt`) + a distinct `outcome-unknown` state instead of a retry-from-zero                                                                     | a killed step is _labelled_, not silently retried; the UI/alert can distinguish "slow" from "failed"                                             | S–M                                                                                                                                                                                      | low                                                                                                                                                                    |
| 5   | **Split `archive` into `archive` (upload) + `encrypt` (F4E1)**                                                                                                                  | the worst case per invocation becomes `max(upload, encrypt)` instead of `upload + encrypt`; the CPU-bound part is isolated from the IO-wait part | M                                                                                                                                                                                        | medium — creates a plaintext window between the two steps, so it **must** ship with (2)                                                                                |
| 6   | **Adopt-by-name correlation**: pass `createBackup({ name: jobId })`; after a kill, scan `backups/*/meta.json` for `name === jobId` and adopt the id instead of re-uploading     | a completed upload is never re-done (saves the 350 MB re-upload)                                                                                 | M                                                                                                                                                                                        | medium — `meta.json` is only written on success, so adoption is only possible when the upload completed; a failed upload still re-uploads (correct)                    |
| 7   | **Chunked, resumable encryption**: persist the F4E1 base nonce + a chunk cursor in the job record and encrypt N MiB per invocation                                              | the encryption step becomes bounded _and_ resumable; removes the last unbounded Worker-side work                                                 | L (2–3 days)                                                                                                                                                                             | medium-high — touches the crypto format; the base nonce must be persisted (a random per-call nonce cannot be resumed) and must never be reused for different plaintext |
| 8   | **Own chunked upload** (container `mksquashfs` + Worker-side multipart with a persisted offset), replacing `sandbox.createBackup`                                               | the upload becomes bounded _and_ resumable; the wall risk disappears entirely                                                                    | XL (≥ 1 week)                                                                                                                                                                            | high — reimplements the SDK backup path, incl. local-bucket mode and presigned-URL handling                                                                            |
| 9   | **Control-plane Workflow** (audit §2, option B) driving the phases over the dispatch binding                                                                                    | unlimited per-step wall time, native retries/observability                                                                                       | L/XL                                                                                                                                                                                     | high — new deployable, phase protocol, and the tenant worker still has to await the container RPC somewhere                                                            |
| 10  | **`limits.cpu_ms = 300000`** on the tenant script                                                                                                                               | raises the CPU budget from 30 s to 5 min for the encryption step                                                                                 | XS **if** the WFP upload accepts it (unverified — the dispatch upload metadata currently sets only `main_module`/`bindings`/`compatibility_*`/`migrations`/`containers`/`observability`) | unknown until verified                                                                                                                                                 |

### Side finding: the local cctr was not the pinned fork

The repo pins the Paso4 fork `0xCAB0/cctr` **0.31.0-paso4.1** (CI + Dockerfile),
but the local machine had **stock cctr 0.31.0** installed. Stock cctr stops
collecting tests at a block whose expected output is followed by an **empty
`where` section** (a trailing `---`): everything after that block silently
disappears from the run. Three corpus files had such a block, so a local run
executed 235 checks instead of 471 — and the local run was the only place the
gap showed up, because CI always used the fork.

Two fixes:

- The corpus no longer uses the truncating shape: the affected blocks carry an
  explicit constraint (`{{ count }}` + `where * count > 0`), so **the fork and
  stock cctr now run the same 471 checks**.
- `scripts/check-corpus-comments.mjs` — the CI guard that already caught the
  top-level `#` comment quirk — now also catches the empty `where` shape, so a
  file can never silently lose its tail again.

The corpus README documents the fork pin (with the SHA-verified install
command) and both format gotchas.

### Rejected

- **Raise the alarm wall**: not configurable (15 min is the platform maximum).
- **Move the archive into a request**: a request's wall time is unlimited while
  the client is connected, but the client here is the platform's own durable job
  — the request would die with the same alarm.
- **Retries with a longer backoff**: does not address a step that legitimately
  needs more than 15 min; it only delays the inevitable.

---

## 4. Measure before choosing the budget

The numbers needed to size the step budget are **already collected**:

- `r2-operation` Analytics Engine rows carry `durationMs` (the `createBackup`
  duration) and `bytesTransferred` (the encrypted archive size) per snapshot.
- `lastSnapshotDurationMs` (new, in the catalog) carries the end-to-end
  duration.

So the per-tenant p95/p99 of `createBackup` is queryable today. Proposed rule:

```
step_budget = clamp(3 × p99(createBackup + encrypt), 6 min, 12 min)
```

with 12 min as the hard ceiling (3 min of headroom under the 15-minute wall).
The per-step duration should also be written to the job record
(`cursor.startedAt` is already there) and emitted as a structured event, so a
tenant that approaches the budget is visible _before_ it starts failing.

---

## 5. Recommended plan

**Phase 1 — hardening (small, high value, no format change)** — **implemented 2026-10-22**

1. (1) Lease = step budget: `BACKUP_JOB_LEASE_MS` / `BOOT_JOB_LEASE_MS` /
   `MAINTENANCE_JOB_LEASE_MS` are now `STEP_LEASE_MS` (6 min) — the lease
   guards a duplicate _step_, not a whole operation.
2. (2) Encryption-aware reconcile: while a master key is configured, an archive
   whose `meta.json` lacks `encrypted: true` is catalogued with
   `unencrypted: true` and queued in `pendingEncryption`. It is **not** dropped
   (it stays restorable — dropping it would lose a recovery point) and it is
   never promoted to the recovery point. The durable `repair` step encrypts it
   in place.
3. (3) Sweep `backups/*` for archive objects without a sibling `meta.json` and
   delete them — they are invisible to the reconciler (which keys on
   `meta.json`) and to the retention prune (which only knows catalogued
   entries), so they would leak forever.
4. (4) Per-step deadline: an interrupted step is reported
   (`detail: 'step-interrupted'`, or `'step-deadline-exceeded'` past the
   12-minute soft deadline) with a `job-step.interrupted` event, and every step
   emits `job-step.completed` / `job-step.failed` with its `durationMs` — the
   measurement the step budget is derived from.

**Phase 2 — bounding the step (medium)**

5. (5) Split `archive` into `archive` (upload) + `encrypt` (F4E1), shipping
   **with** (2) so the plaintext window is never adopted.
6. (6) Adopt-by-name correlation (`name: jobId`) so an interrupted upload that
   actually completed is adopted instead of re-uploaded.
7. Archive-size reduction: pass SDK `excludes` for regenerable paths
   (`node_modules`, `*.log`, caches) and prune the container before staging, so
   the upload (and the encryption) shrink. Measure the effect on the archive
   size first — the legacy stage must stay inside the archive.

**Phase 3 — only if the measured p99 still approaches the budget**

8. (7) chunked resumable encryption, or (9) the control-plane Workflow.

**Verification for Phase 1/2**

- Unit: reconcile refuses an unencrypted orphan; the sweeper deletes a
  meta-less object but keeps a catalogued one; a killed step is retried after
  the (short) lease, not after 14 minutes; the `archive`/`encrypt` split leaves
  the catalog consistent when the encryption step fails.
- Corpus (`test/business-rules/operational-modes.txt`): an unencrypted orphan is
  never catalogued; a killed step resumes within the step budget.
- Staging: force-kill the DO during `archive` and during `encrypt` and confirm
  (a) the retry is prompt, (b) no unencrypted archive is ever listed, (c) no
  object without `meta.json` survives.
