# Lerma

## 0.17.0

### Minor Changes

- feat(observability): record per-instance job runs and surface their performance

  Every instance job lerma initiates or observes is now persisted as an
  `instance_job_events` row — `backup_create` (freeze snapshot and verified
  checkpoint/park), `restore` (the durable boot's restore-from-recovery-point
  step) and `gateway_start` (the boot's launch step, until `/api/health` answers
  ready). The instance Observability section gains a 24-hour **Job activity**
  milestone timeline (one lane per job, hoverable icon with status, duration,
  trigger, timestamp and error) and the **Worker telemetry** widget gains a **Job
  performance** tab: run counts, success rate, average/p95/max duration, the last
  run and a bar graph of the selected job's duration evolution across the 24 h /
  7 d / 30 d window.

  Recording is instance-scoped and best-effort: it never fails the job, never
  probes a sleeping container on its own (it folds the boot/restore fields of the
  non-passive health poll lerma already runs, plus a background poll after a
  manual start/restart), and stores only metadata. ([7c58e75](https://github.com/0xCAB0/fouria/commit/7c58e756df094d6cc48d54fb96fb0722ebc58c96))

### Patch Changes

- Backups: make a retained snapshot restorable for as long as it is retained, stop archiving regenerable paths, and gate scheduled snapshots on change by default.

  - **Archive TTL is retention-neutral.** The Sandbox SDK refuses to restore an archive past `createdAt + ttl` (`BackupExpiredError`), so the previous 7-day value silently turned a retained recovery point into a billed-but-unrestorable object — `cleanupOldBackups` prunes by count, so an idle or frozen tenant could keep `BACKUP_MAX_VERSIONS` archives that no boot could apply, and the stable pointer was then dropped, failing the boot closed with `TenantStateUnavailableError`. New snapshots use `SNAPSHOT_ARCHIVE_TTL_SECONDS` (10 years) and the durable `repair` step heals existing tenants via `extendArchiveTtl` (reported as `ttlExtended`/`ttlFailed` from `POST /api/admin/storage/reconcile`).
  - **`SNAPSHOT_EXCLUDES`** drops `.npm`, `*.log`, `*.tmp`, `.cache` and `workspace/*node_modules*` before `mksquashfs`, behind `BACKUP_ARCHIVE_EXCLUDES` (**off by default**). Excluding paths makes an archive-restored tenant differ from its pre-eviction tree (and from a Tier 1 workspace snapshot), so the recovery tier stays a complete image unless an operator opts in per instance. Plugin `extensions/<id>/node_modules`, SQLite databases and session transcripts are never excluded, because a restore has no reinstall step of its own.
  - **`BACKUP_CHANGE_DETECTION` defaults to on.** A snapshot re-uploads the whole archive and re-streams it through the Worker for F4E1 encryption; measured on `tenant-tarjetero-dev`, two snapshots 37 minutes apart differed by 0.08 % (348 KB of 428 MB) while both were uploaded, encrypted and stored in full. Only an explicit `false` opts out, and a scan failure still reports `changed`, so the gate can only cost an extra snapshot — never a missed one.
  - **A restore forgets the change baseline.** Restored files keep their archived (older) mtimes, so the `-newer` scan would have called the replaced tree unchanged and skipped the first post-restore snapshot.
  - **Idle eviction still stops inline.** Change detection on by default means a quiet tenant takes no scheduled snapshot, so its newest archive ages past the idle window and the snapshot-age test alone would re-archive the whole tenant on every sleep cycle. `onActivityExpired()` now also accepts `unchanged` from the same change scan as proof that an existing archive covers the current state. ([4e93f63](https://github.com/0xCAB0/fouria/commit/4e93f6314a43ac18a5dbcbec204791c854396b1c))
- fix(lerma): prepare container images before rollouts and gate promotion on live health ([dae0487](https://github.com/0xCAB0/fouria/commit/dae0487f97f595235a54faa3827028a7596f0151))
- fix(provisioning): private OTLP and promotion transports (sandbox bridge + R2 pull)

  OTLP ingest moves to the `otel.internal` sandbox bridge and promotion to an R2
  stage/pull between the sibling environment buckets, so neither feature exposes a
  public tenant route. Lerma binds `PROMOTION_SOURCE_BUCKET` on production
  instances and ensures the sibling dev bucket exists before the deploy. ([4e171bb](https://github.com/0xCAB0/fouria/commit/4e171bb4238342cb165a6e36292ae3c7d0e52107))
- fix(billing): stop deploying the tenant AE read-back token; emission uses the ANALYTICS binding ([673b1cd](https://github.com/0xCAB0/fouria/commit/673b1cdc7a259d8dfb8ecbc9d72e3ec5c795f4e4))
- Default new instances to `standard-1` instead of `basic`.

  `basic` is 1 GiB of memory, below the OpenClaw gateway's ~1.1 GiB steady-state
  RSS (and ~2.6 GiB peak), so an instance provisioned without an explicit type
  OOM-looped on first use. `Lerma.Instances.InstanceTypes.default/1`, the
  `Instance` schema default, and `Containers.deploy_script_and_container/1`'s
  fallback now return `standard-1` (4 GiB / 0.5 vCPU / 8 GB), the practical floor
  for an agent-running tenant. Existing instances keep their stored type; a resize
  is a Cloudflare `full_auto` rollout (one cold start + R2 restore per tenant). ([2902de0](https://github.com/0xCAB0/fouria/commit/2902de063931c50df06049d9166c8ab90b8d7954))
- Zero-loss container rollouts: park the tenant before an image rollout and resume
  it after the replacement.

  A container replacement always starts from a fresh disk, so the replacement
  restores the tenant from the checkpoint taken immediately before the rollout.
  Anything written after that archive was produced was lost. The upgrade now
  **parks** the tenant instead of restarting it:

  - `Cloudflare.Containers.checkpoint/2` gained `park: true`, which uses the
  tenant Worker's new `checkpoint-park` control action: the same verified
  snapshot, but the tenant is left drained and write-blocked (mutating admin and
  debug requests answer `409 TENANT_PARKED`, and container exec is refused)
  until the control plane resumes it.
  - `Cloudflare.Containers.resume/1` closes the park and lets the replacement
  container restore the checkpoint and boot the gateway. A no-op resume reports
  `resumed: false` instead of failing.
  - `@upgrade_pipeline`'s `:checkpoint` step now parks and is **fail-closed**: a
  checkpoint that cannot be taken aborts the upgrade _before_ the container is
  replaced (a sleeping instance is exempt — it has nothing live to lose). A
  `:resume` step runs immediately after `:dispatch_deploy`, and both error paths
  resume best-effort so a failed upgrade cannot leave a tenant parked.

  The worker-only `Instances.upgrade_instance/3` path keeps the non-parking
  checkpoint: it does not change the container image, so there is nothing to lose.

  Zero-loss upgrades depend on this ordering, not on the Sandbox 1.0 migration:
  Cloudflare's container snapshots are image-locked (not portable across an image
  change) and require the `durable_object` scheduling policy, so they cannot carry
  state across an upgrade. ([9f6124c](https://github.com/0xCAB0/fouria/commit/9f6124c064ba2cb27b3f085cd90af5aa9cdee84c))
- fix(lerma): keep container-backed DO classes enabled across redeploys ([994a087](https://github.com/0xCAB0/fouria/commit/994a0876a2c78744ce74a781d3e021f7186e9656))

## 0.16.0

### Minor Changes

- feat: add a client documentation widget with a source tree and live preview ([3054026](https://github.com/0xCAB0/fouria/commit/3054026c8cf891418c0041a8d0661761a68cadb6))
- Redesign the instance page around dedicated **Observability** and **Tools** sections.

  The instance page is now a full-width, single-page layout instead of a narrow
  two-column grid that squeezed every widget into a third of the screen:

  - **Overview** — instance details in a full-width definition grid (worker name,
  type, environment, deployed version, region, created, AI Gateway).
  - **Observability** — uptime (production) and worker telemetry side by side at
  roughly twice their previous width, plus a new **Health events** timeline
  (`LermaWeb.HealthEventsWidget`) listing the state transitions recorded by the
  background probes. The timeline works for every instance, not only production.
  The two uptime figures are now labelled for what they measure — **probe
  availability (time-weighted)** vs **invocation success (request-weighted)** —
  and the Uptime widget surfaces the window it actually observed, so a short
  probe history no longer reads as a 24 h outage.
  - **Health events** — a compact 24 h **time graph** (up / down / unknown) instead
  of an event list: hovering a segment animates it and reveals that window's log
  (probe OK, or the HTTP status and error message). The header **Inspect** action
  opens the Infrastructure health panel, which moved here from the Tools grid.
  - **Billable actions** — a new section embedding the client's Usage widget
  (credit spend per action, action log, stats) for **production** instances and
  for any instance whose client has an active billing plan. Paso4 instances are
  never billable and omit it. The widget is labelled _Production only_ with a
  two-tone environment legend: dev/staging are non-billable by contract and
  their cost lives in development expense, so the credits chart has no dev
  series.
  - **Tools** — a sticky Swiss-knife toolbar at the top of the page (Deploy,
  Start/Stop, Environment, Deployments, AI Gateway, Gateway token, Debug routes,
  Deprovision). Primary actions keep a label, the other blades are icons with a
  tooltip and state dot; **Unfold** opens the detailed tool cards in a scrollable
  drawer, and folding clears a revealed gateway token. The bar condenses to icons
  on mobile. The icon-only action row in the details header and the cramped
  "Ops" workbench/management cards are gone.

  Visually the page drops the nested bordered cards in favour of calmer
  `rounded-2xl` surfaces, quieter section headings with one-line descriptions, and
  soft secondary buttons.

  `LermaWeb.UptimeWidget` keeps only the aggregate uptime ratios; the recent
  changes list moved to the new widget. No backend changes. ([35f07eb](https://github.com/0xCAB0/fouria/commit/35f07ebfffc14a07cdc5f205d07c6c4b6a6ef12a))
- feat: price semantic billing actions (telegram_query, queue_reevaluation, signal_analysis, email_draft, review_analysis) with success-only consumption, business-unit action_key idempotency, and trace-level precedence ([85f8701](https://github.com/0xCAB0/fouria/commit/85f8701ecfe0fce2b92fadd24468c09d0633e337))

### Patch Changes

- fix: lazy-load client documentation on demand ([5fbcabe](https://github.com/0xCAB0/fouria/commit/5fbcabe7a5324db22be5e2783fc564a6b67e401e))
- fix(lerma): retry dispatch uploads that re-declare migrated DO classes (#516) ([d3c2f60](https://github.com/0xCAB0/fouria/commit/d3c2f60fe78bda2930ddf470840c3d69e0ce0b53))

## 0.15.2

### Patch Changes

- Decompress gzip OTLP/HTTP JSON batches before parsing so Cloudflare
  Observability destinations are accepted instead of retrying on a `400`. ([7bd792b](https://github.com/0xCAB0/fouria/commit/7bd792b461cfd4fdc0cf4de42bf7e88a71178816))

## Historical changes (imported from `apps/fouria/AGENTS.md`)

These entries predate the generated release sections above. They were maintained
by hand in `apps/fouria/AGENTS.md` and are kept here verbatim for traceability.
New entries belong in a changeset (see `docs/VERSIONING.md`); the sections above
are generated by Changesets (`scripts/changeset-changelog.mjs`).

- **Container-backed tenant redeploys keep the `Sandbox` DO container-enabled.** A tenant script that is updated after a working deploy could later fail with `Sandbox.jsrpc → "Containers have not been enabled for this Durable Object class"` (observed on `tenant-tarjetero-dev`): the DO keeps the code it was instantiated with, so the missing enablement stayed latent until the object was evicted (here, after the `v2026.10.29` upgrade and the Dev instance’s idle stop). `Lerma.Cloudflare.DispatchNamespace` now accepts either class names or full container entries in `containers`, and `Lerma.Cloudflare.Containers.deploy_script_and_container/1` always sends the full entry (`class_name`, `scheduling_policy`, `image`, `instance_type`, `max_instances`) — a class-name-only entry is enough for the first upload but **not** for an update. Regression tests: `apps/lerma/test/lerma/cloudflare/dispatch_namespace_test.exs` (metadata carries the full config) and `containers_test.exs` (deploy passes it through); docs `apps/lerma/README.md` (“Container-backed tenant deploys”).
- **Lerma deploys are idempotent around Cloudflare DO migration conflicts (#516).** A tenant Worker's Durable Object class set was derived from the script's _bindings_, but Cloudflare validates `new_sqlite_classes` against the script's _migration history_, so the two can diverge: an upgrade redeploy that binds fewer classes (or a rollback that deletes the script while leaving its classes migrated) made every subsequent deploy re-declare `TelegramIngress` and fail with `400 code 10074` — deterministically, so provisioning could not recover on its own. `Lerma.Cloudflare.DispatchNamespace.upload_script/6` now retries the upload without any class Cloudflare reports as already migrated (`10074` names it), switching an implicit full first-deploy set to an explicit `new_sqlite_classes` list; each retry removes at least one class, so it terminates, and the retry is logged. `Lerma.Instances.instance_bindings/2` now re-declares the same DO set as provisioning (`Sandbox`, `ACTIVITY`, `TELEGRAM_INGRESS`, plus `BACKUP_WATCH` while the flag is on) so an upgrade cannot detach a migrated class from its binding. Provisioning cleanup and a failed atomic deploy tear the classes down first via `delete_durable_object_namespaces/2` (never on an upgrade rollback, whose Durable Objects hold the live tenant's state: `preserve_do_namespaces`). Tests: `apps/lerma/test/lerma/cloudflare/dispatch_namespace_test.exs` (migrated-but-unbound re-deploy, resurrected first deploy, unrelated error not retried), `containers_test.exs`, `tenant_provisioner_pipeline_test.exs`; docs `apps/lerma/README.md`.
- Cloudflare Observability **destination** ingestion (issue #337). Account-level Observability → Destinations now push Workers Logs and Workers Traces to Lerma over OTLP/HTTP JSON, giving event-driven worker diagnostics and uptime statistics that scale with traffic instead of tenant count. **(1) Config:** `apps/fouria/wrangler.jsonc` (`env.staging` → `lerma-worker-logs`/`lerma-worker-traces`, `env.production` → `lerma-worker-logs-prod`/`lerma-worker-traces-prod`) and `Lerma.Cloudflare.Observability.worker_metadata/0` reference the environment's destination names, with per-signal head sampling (10% at rollout) and `persist: false` when a destination is configured (Lerma is the destination of record). The Lerma CD passes the matching names to staging and production, and the token to both; it also reconciles the account-level destination pairs (`mix lerma.observability.destinations --warn-only`) before each deploy so a fresh environment self-heals once the Cloudflare credential has Workers Observability Write. New `WORKER_OBSERVABILITY_LOGS/TRACES_DESTINATIONS`, `*_SAMPLING_RATE` and `*_PERSIST` env vars. **(2) Ingest:** new `POST /api/ingest/otel/v1/logs` and `/v1/traces` (bearer `OTEL_INGEST_TOKEN`, constant-time compare, fail closed; OTLP/HTTP JSON only — protobuf → `415` + hint; `202 {accepted, skipped}` so Cloudflare never retry-loops). `Lerma.Observability.Otlp` is a pure allowlist parser: it keeps script/dispatch namespace/outcome/handler/status/exception class names/cpu+wall/trace+span+invocation ids/timestamps, drops log text, stacks, URLs/query strings, user agents, request/response bodies and DB query text, and keeps only invocation root spans. Rows land in `observability_events` with a unique `dedupe_key` + `ON CONFLICT DO NOTHING`, so replayed batches (at-least-once delivery) add no rows. Cloudflare gzips each OTLP/HTTP JSON batch while keeping `Content-Type: application/json`; `LermaWeb.Plugs.OtlpBodyReader` decompresses it before `Plug.Parsers` (otherwise the destination retries on a `400` forever). **(3) Uptime:** `GET /api/analytics/v1/uptime/:script_name?window=24h|7d|30d|<n>h|<n>d` computes uptime/failure ratios from pushed root spans only; no events observed (including under sampling) → `null` ratios + `status: "unknown"`, never downtime. The instance page gained a **Worker telemetry** widget (window selector + uptime/failure/invocation/CPU/wall), and `Lerma.Observability.Pruner` sweeps 90-day retention daily. `HealthMonitor` and billing flows are untouched. Tests: `test/lerma/observability/otlp_test.exs`, `telemetry_test.exs`, `test/lerma/cloudflare/observability_test.exs`, `test/lerma_web/controllers/api/otlp_ingest_controller_test.exs`, `observability_controller_test.exs`, `test/lerma_web/components/worker_telemetry_widget_test.exs`, `test/lerma_web/live/instance_live_test.exs`; docs `docs/CLOUD-ARCHITECTURE.md`, `docs/ENV-VARIABLES.md`, `docs/TESTING.md`, `apps/fouria/public/articles/PRIVACY.md`, `apps/fouria/docs/WORKER-DEVELOPMENT.md`, `SECRETS.example.yaml`, `docs/schemas/secrets-manifest.schema.json`, `apps/lerma/.env.example`.
- Health endpoints no longer require the ROOT role at the edge. A tenant ADMIN opening `/_admin` was 302'd to the Cloudflare Access login for `GET /api/health` and `GET /api/health/ws` (the dashboard's health stream), because the per-tenant `FourIA Health - <slug>` Access app on `<slug>.fouria.io/api/health` allowed only the `Paso4 Root Users` group plus lerma's health-monitor service token. The health app (and the platform-wide `*.fouria.io/api/health` / `*.dev.fouria.io/api/health` IaC apps) now carries **no role requirement**: an `allow` policy on `everyone` (any authenticated user) plus the existing `non_identity` service token, so the dashboard and lerma's headless probe both work. `AccessPolicy.health_policies/1` drops the root-group policy (the host app keeps it); `access.tf` `public_health` matches. Tests: `test/lerma/cloudflare/access_policy_test.exs`, `apps/fouria/iac/tests/access_app.tftest.hcl`; docs `apps/lerma/README.md`, `apps/fouria/iac/README.md`, `apps/fouria/docs/WORKER-DEVELOPMENT.md`, `docs/ENV-VARIABLES.md`. Existing tenants were reconciled in place via the Cloudflare API (aud preserved).
- Paso4-exclusive instances + agent: lerma can now provision internal `scope: "paso4"` instances (nullable `client_id`, no billing plan, `@paso4.io`-only Access), and moltlazy ships an opt-in `paso4-dev` agent (`PASO4_AGENTS`, gated by `MOLTLAZY_PASO4_AGENTS=true` and named in `MOLTLAZY_AGENTS`) that picks up GitHub issues via the bundled `github` skill (`gh` CLI, installed in the image) authenticated by `GITHUB_TOKEN`. The Dispatch Worker resolves paso4 tenants from `instances` (no `clients` row) and rejects non-`@paso4.io` emails. See [apps/lerma/README.md](/staging/llms/platform/apps/lerma/index.md#paso4-exclusive-instances) and `docs/ENV-VARIABLES.md`.
