Persistent workspace policy — Phase 0 (measurement)
On this page
Date: 2026-10-30
Status: Implemented (instrumentation only)
Design: https://design.paso4.io → Backup policy page (12-board proposal)
Next: Phase 1 (snapshot on evict) is gated on the numbers below.
1. Why this phase exists
The current policy treats the archive as both the routine save path and the
disaster-recovery artifact. Every save is a full squashfs: staged in the
container, encrypted in the Worker, uploaded to R2, pruned to three versions.
BACKUP_INTERVAL_MINUTES is a floor that stretches to twice the last observed
snapshot, so a slow tenant snapshots back-to-back.
The proposed rework splits persistence into two tiers:
- Tier 1 — container snapshot (
ctx.container.snapshotContainer()): the routine save path, seconds, image-pinned, 30-day TTL. - Tier 2 — portable archive (today’s F4E1 archive): the recovery point, portable across images, encrypted at rest.
Choosing the Tier 1 cadence, the dirty-window bound, and the archive ceiling requires data we do not have. Phase 0 collects it without changing behaviour: no snapshot, restore, eviction, or retention decision reads these events.
2. What is measured
| Event | Emitted from | Answers |
|---|---|---|
persistence-snapshot | finalizeSnapshot (success); ActivityScheduler (fail/skip) | How big is an archive? How long does each stage take? |
persistence-restore | restoreIfNeeded, restoreBackupById | How long does recovery actually take, and why? |
backup-change-window | BackupWatch tick | How long is a tenant continuously dirty before capture? |
persistence-snapshot
double1 = end-to-end (prepare start → finalize commit), double2 =
sandbox.createBackup() alone, double3 = captured workspace bytes (from the
archive’s meta.json). Blobs: customer, trigger (scheduled / manual /
deprovision), outcome (succeeded / failed / skipped), detail
(failure name or skip reason), platform, environment.
Skip rows are the denominator: interval-not-elapsed, unchanged,
not-running, backup-lock-busy. Without them a quiet series looks healthy.
persistence-restore
double1 = duration, double2 = archive bytes. Blobs: customer, source
(recovery-point for a cold boot, explicit for an operator restore),
outcome, detail, platform, environment. An archive that is present but
unusable is recorded as skipped / archive-unusable, not as a generic failure.
backup-change-window
double1 = continuous dirty time at the observation (dirtySince → now),
double2 = container-side scan duration. Blobs: customer, status
(unchanged / changed / active / unknown), decision (job-active,
unchanged, deferred-active, deferred-floor, requested, busy,
request-failed, disarmed), platform, environment.
Two decisions are taken before any container RPC, from the
ActivityScheduler’s own job record (snapshotGate): job-active (a snapshot
already owns the slot) and deferred-floor (inside the cadence floor). Both
report status = unknown with scanMs = 0 — no scan ran — so a query must not
read them as “the tenant was inspected and found clean”.
The watch is feature-flagged (BACKUP_CHANGE_WATCH), so this series only exists
where the watch runs. That is intentional: the watch is the mechanism whose
cadence the rework changes.
3. Queries
Archive volume per tenant per day:
SELECT blob2 AS customer,
SUM(double3) AS bytes,
SUM(double1) AS snapshot_ms,
SUM(double2) AS archive_ms,
COUNT(*) AS snapshots
FROM fourai_cost_metrics
WHERE blob1 = 'persistence-snapshot'
AND blob4 = 'succeeded'
AND timestamp > NOW() - INTERVAL 1 DAY
GROUP BY customer
ORDER BY bytes DESC
Dirty-window distribution (p50/p95 by status):
SELECT blob2 AS customer, blob3 AS status,
COUNT(*) AS ticks,
SUM(double1) AS dirty_ms_total,
MAX(double1) AS dirty_ms_max,
SUM(double2) AS scan_ms_total
FROM fourai_cost_metrics
WHERE blob1 = 'backup-change-window'
AND timestamp > NOW() - INTERVAL 7 DAY
GROUP BY customer, status
ORDER BY customer
Why snapshots did not run:
SELECT blob2 AS customer, blob4 AS outcome, blob5 AS detail, COUNT(*) AS n
FROM fourai_cost_metrics
WHERE blob1 = 'persistence-snapshot'
AND blob4 != 'succeeded'
AND timestamp > NOW() - INTERVAL 7 DAY
GROUP BY customer, outcome, detail
ORDER BY n DESC
Restore cost (cold boots vs operator restores):
SELECT blob2 AS customer, blob3 AS source, blob4 AS outcome,
COUNT(*) AS n, SUM(double1) AS total_ms, MAX(double1) AS max_ms,
SUM(double2) AS bytes
FROM fourai_cost_metrics
WHERE blob1 = 'persistence-restore'
AND timestamp > NOW() - INTERVAL 30 DAY
GROUP BY customer, source, outcome4. Gates for Phase 1 (snapshot on evict)
Phase 1 may start when the Phase 0 series shows, for at least three tenants over two weeks:
- Archive size is stable and material. A p95 archive above roughly 100 MB justifies a snapshot-based save path; a p95 below ~10 MB means the archive is already cheap and the rework should be deferred.
- The archive step dominates.
double2 / double1above ~0.5 meanscreateBackup(not staging or bookkeeping) is the cost to remove. - Dirty windows are bounded. The p95
dirtyWindowMsforchangedticks is the RPO the checkpoint cadence must beat. If it is already under the proposed checkpoint interval, Tier 1 buys resume latency (not freshness), and Phase 1 is justified on cold-start time rather than RPO. - Skips are explained. Every non-succeeded row carries a reason. An
unexplained
failedcluster blocks the phase.
5. Explicitly out of scope
- No new binding, env var, or migration.
- No container snapshot call: Tier 1 does not exist yet, so
archiveDurationMsis thecreateBackuptime, not a snapshot time. - No change to retention, the schedule gate, the lease, or the restore ladder.
- The
backup-change-windowevent is emitted only whereBACKUP_CHANGE_WATCHis enabled; enabling the watch for measurement is a separate operator choice.
6. Implementation
src/metrics.ts—trackPersistenceSnapshot,trackPersistenceRestore,trackBackupChangeWindow(row layouts documented above).src/persistence.ts—SnapshotPhaseContext.trigger,SnapshotArchive.sizeBytes(onemeta.jsonread atarchive), and the success/failure/skip emissions.src/activity/scheduler.ts— trigger labelling, skip rows (gate, sleeping, busy lock), failure rows per failed attempt.src/backup-watch/watch.ts(+durable-object.ts) — the pure tick returns an observation and the DO writes the row, so the state machine stays metric-free.- Corpus:
test/business-rules/cost-tracking.txtandtest/business-rules/fixture/expected-cost-schema.json.