Agent Guidelines
On this page
- Project Overview
- Project Structure
- Key Patterns
- Commands
- Contribution Guidelines
- Code Style
- Documentation
- Architecture
- Cost Observability
- Local Development
- Docker Image Caching
- Gateway Configuration
- OpenClaw Config Schema
- Common Tasks
- R2 Backup Encryption (F4E1)
- R2 Storage Notes
- Cloudflare Tunnel Setup (Manual Flow)
- Active Technologies
- Recent Changes
Guidelines for AI agents working on this codebase.
Project Overview
This is a Cloudflare Worker that runs OpenClaw (formerly Moltbot/Clawdbot) in a Cloudflare Sandbox container. It provides:
- Proxying to the OpenClaw gateway (web UI + WebSocket)
- Admin UI at
/_admin/for device management - API endpoints at
/api/*for device pairing - Debug endpoints at
/debug/*for troubleshooting
Note: The CLI tool and npm package are now named openclaw. Config files use .openclaw/openclaw.json. Legacy .clawdbot paths are supported for backward compatibility during transition.
Paso4 Extensions (moltlazy/)
The moltlazy/ module is fully owned by the Paso4 developers and provides two systems:
- Config Module (
config/+moltlazy/index.ts) — filesystem patcher used bystart-openclaw.shat container startup to writeopenclaw.jsonbefore the gateway launches. - Features Module (
sdk/) — RPC-based config applied to a running gateway via the Admin HTTP API. Supported feature packs: Knowledge Graph, CF Unified Billing.
Project Structure
src/
├── index.ts # Main Hono app, route mounting
├── types.ts # TypeScript type definitions (AUTOGENERATED by `bun run types`, do not apply changes directly here, instead re-run the command after implementing the rest of the features)
├── config.ts # Constants (ports, timeouts, paths)
├── auth/ # Cloudflare Access authentication
│ ├── jwt.ts # JWT verification
│ ├── jwks.ts # JWKS fetching and caching
│ └── middleware.ts # Hono middleware for auth
├── gateway/ # OpenClaw gateway management
│ ├── process.ts # Process lifecycle (find, start)
│ ├── env.ts # Environment variable building
│ ├── r2.ts # R2 bucket mounting
│ ├── sync.ts # R2 backup sync logic
│ └── utils.ts # Shared utilities (waitForProcess)
├── routes/ # API route handlers
│ ├── api.ts # /api/* endpoints (devices, gateway)
│ ├── admin.ts # /_admin/* static file serving
│ └── debug.ts # /debug/* endpoints
└── client/ # React admin UI (Vite)
├── App.tsx
├── api.ts # API client
└── pages/
moltlazy/ # Paso4 proprietary configuration utilities
├── cli.ts # CLI entry point for patching config
├── index.ts # Main logic for config patching
├── types.ts # Type definitions for OpenClaw config
├── config/
│ └── include.ts # per-section $include injection (session/tools/logging)
├── config/
├── config/ # Configuration templates
├── agents/ # Agent configuration templates
│ ├── defaults.ts # Default agent settings
│ └── premade/ # Pre-configured agent definitions
├── models/ # Provider-specific model configurations
├── sdk/ # SDK Module — typed RPC client over the Admin HTTP API
│ ├── client.ts # Typed RPC client
│ └── (agents, channels, config, cron, devices, models, gateway, update, …)
└── tests/ # Tests for configuration utilitiesKey Patterns
Environment Variables
DEV_MODE- Local-dev switch: skips CF Access auth AND bypasses device pairingOPENCLAW_DEV_MODE- Explicit override for the dev-agents flag. Normally derived by the Worker from the instance’s own-devhostname suffix (seesrc/dev-mode.ts), which enables the Fouria Builder/Q&A agents, client-doc materialization, and the opencode-credits automation without bypassing auth. Forwarded to the container asOPENCLAW_DEV_MODE; falls back toDEV_MODEfor local developmentDEBUG_ROUTES- Enables/debug/*routes (disabled by default)- See
src/types.tsfor fullMoltbotEnvinterface - See
docs/schemas/secrets-manifest.schema.jsonfor the secrets manifest
CLI Commands
On OpenClaw 2026.8.2 the CLI no longer accepts --url ws://localhost:18789
(and most local commands reject --token too). Commands run inside the container
(the gateway host) either operate on local state or auto-connect to the local
Gateway — auth resolves from the OPENCLAW_GATEWAY_TOKEN env SecretRef, so no
connection flags are passed. Commands that require an owner on a multi-agent
roster take --agent <id>. Never hardcode the id: resolve it at runtime via
openclaw agents list --json (prefer main when present, else the first
configured agent — e.g. the dev-only fouria-builder/fouria-qa premades when
running with DEV_MODE=true; see
resolveAgentFlag() in src/routes/api/shared.ts and resolveCliAgentId()
in moltlazy/sdk/cli.ts):
// Sandbox SDK 1.0: exec takes argv and resolves on launch, not completion.
startShell(sandbox, 'openclaw devices list --json');
// multi-agent commands need an explicit, dynamically-resolved owner:
startShell(sandbox, `openclaw models list --json${await resolveAgentFlag(sandbox)}`);
Gateway-connecting commands (devices, cron, secrets reload) may still accept
--url/--token when given together, but the 2026.8.2 guidance is to drop --url
and let the CLI use the configured local target.
CLI commands take 10-15 seconds due to WebSocket connection overhead. Use waitForProcess() helper in src/routes/api.ts.
Success Detection
The CLI outputs “Approved” (capital A). Use case-insensitive checks:
stdout.toLowerCase().includes('approved');Commands
bun run lint # Run linting (oxlint)
bun run test # Run tests (vitest)
bun run test:watch # Run tests in watch mode
bun run test:client # Run SPA component/page tests (jsdom, vitest.client.config.ts)
bun run test:ui # Run the Playwright UI suite (Vite dev server + mocked API)
bun run test:ui:headed # Same, with a visible browser
bun run test:ui:render:local # Local-only plwr admin-UI component render timings
bun run test:gateway:boot # Local gateway boot smoke test (Docker)
bun run test:gateway:boot:ai # Same, with real AI Gateway onboard path (.dev.vars.e2e)
bun run build # Build client (vite)
bun run build:all # Build client + moltlazy
bun run build:docker # Build all + install deps (pre-Docker build)
bun run deploy # Build all and deploy to production
bun run deploy:dev # Build all and deploy to dev
bun run dev # Admin SPA Vite dev server (port 8787)
bun run dev:worker # Worker dev server (Cloudflare Vite plugin, port 8788)
bun run start # Build all + Worker dev server
bun run typecheck # TypeScript checkContribution Guidelines
Validation Requirements
Before submitting any contribution:
-
Run lint and tests - All builds must pass:
bun run lint bun run test -
Environment variable changes - If your contribution adds, updates, or deletes an environment variable, you MUST update it in all three locations:
README.md- Add to the secrets reference tablesrc/gateway/env.ts- Add tobuildEnvVars()function if passed to container.dev.vars.example- Add to the mapping comments and example
Testing
Tests use Vitest. Test files are colocated with source files (*.test.ts).
Current test coverage:
auth/jwt.test.ts- JWT decoding and validationauth/jwks.test.ts- JWKS fetching and cachingauth/middleware.test.ts- Auth middleware behaviorgateway/env.test.ts- Environment variable buildinggateway/process.test.ts- Process finding logicgateway/r2.test.ts- R2 mounting logicgateway/sync.test.ts- R2 backup sync logicmoltlazy/tests/index.test.ts- per-section $include injection and config generation
When adding new functionality, add corresponding tests.
The admin SPA also has a Playwright UI suite (test/ui/, config
playwright.config.ts) run with bun run test:ui. It serves the real dashboard
via the Vite dev server (bun run dev) and mocks every /api/* call per test,
so it needs no Worker, container, or Cloudflare credentials. It covers the app
shell, role-based navigation/redirects, every admin page’s render, and the
notification bell. On NixOS the browsers come from PLAYWRIGHT_BROWSERS_PATH
(see flake.nix); playwright.config.ts selects the newest chromium-* there
so the Nix bundle can lag the npm playwright version without breaking the run.
A separate local-only cctr+plwr suite (test/e2e/ui_components_render.txt,
bun run test:ui:render:local) validates the admin (root) UI components against
the local worker and records their render times (openMs/componentMs/waitMs
plus Navigation Timing/FCP) via fixture/ui-render-measure; it self-skips
outside E2E_LOCAL=true. See docs/TESTING.md.
Code Style
- Use TypeScript strict mode
- Prefer explicit types over inference for function signatures
- Keep route handlers thin - extract logic to separate modules
- Use Hono’s context methods (
c.json(),c.html()) for responses
Documentation
README.md- User-facing documentation (setup, configuration, usage)AGENTS.md- This file, for AI agents
Development documentation goes in AGENTS.md, not README.md.
Architecture
Browser
│
▼
┌─────────────────────────────────────┐
│ Cloudflare Worker (index.ts) │
│ - Starts OpenClaw in sandbox │
│ - Proxies HTTP/WebSocket requests │
│ - Passes secrets as env vars │
└──────────────┬──────────────────────┘
│
▼
┌─────────────────────────────────────┐
│ Cloudflare Sandbox Container │
│ ┌───────────────────────────────┐ │
│ │ OpenClaw Gateway │ │
│ │ - Control UI on port 18789 │ │
│ │ - WebSocket RPC protocol │ │
│ │ - Agent runtime │ │
│ └───────────────────────────────┘ │
└─────────────────────────────────────┘Key Files
| File | Purpose |
|---|---|
src/index.ts | Worker that manages sandbox lifecycle and proxies requests |
src/metrics.ts | Cost and usage metrics instrumentation for Analytics Engine |
Dockerfile | Container image based on cloudflare/sandbox with Node 22 + OpenClaw |
start-openclaw.sh | Startup entry point: start is the production cold-start (acquire the container-wide startup lock before setup, hold fd 9 through the Gateway exec — a single owner covers setup and launch), post-start (dev automation + boot-log reset), setup/launch (standalone smoke/manual modes only), or full (legacy: setup + background gateway launch). The Worker drives production cold-start as one start-openclaw.sh start call then a detached post-start (never awaited, so it cannot block readiness); it no longer invokes setup/launch separately. setup/post-start must stay child-free (no tail -f/cloudflared) — the Worker waitForExits on them. start/setup/full serialize the slow setup body on the exclusive /tmp/openclaw-setup.lock, and start holds it through the Gateway exec (no release before launch); the standalone launch takes the dedicated /tmp/openclaw-gateway-launch.lock, held for the gateway’s whole lifetime (fd inherited by the exec’d gateway). Either way exactly one openclaw gateway run owner can exist — losers poll /health and exit 0 without launching a duplicate; the serial test/e2e/ac_gateway_startup_concurrency.txt asserts one owner and the absence of the migration-input-change and SQLite offline-maintenance errors. When the Worker passes OPENCLAW_RESTORED=true (a backup was restored), the setup body is skipped entirely and the restored state is launched directly. See docs/WORKER-DEVELOPMENT.md |
wrangler.jsonc | Cloudflare Worker + Container configuration |
Cost Observability
The Worker instruments cost metrics using Cloudflare Analytics Engine for per-client cost tracking:
| Event Type | Dimensions | Metrics |
|---|---|---|
ai-query | model, provider, customer, actionType | tokensIn, tokensOut, costUsd |
container-active | customer, instanceType, action | heartbeat count |
r2-operation | customer, opType, opClass | bytesTransferred, durationMs |
cron-trigger | customer, triggerReason | wokeContainer flag |
ws-session | customer, sessionId | durationSeconds, messagesCount |
Analytics Engine Configuration
The ANALYTICS binding writes to dataset fourai_cost_metrics. Query via GraphQL:
query CostMetrics($accountTag: String!) {
viewer {
accounts(filter: { accountTag: $accountTag }) {
analyticsEngineDatasets(filter: { name: "fourai_cost_metrics" }) {
data(
query: "SELECT blob1 as event_type, blob2 as customer, blob3 as model, SUM(double1) as tokens_in, SUM(double2) as tokens_out FROM fourai_cost_metrics WHERE timestamp > NOW() - INTERVAL 1 DAY GROUP BY blob1, blob2, blob3"
) {
result
}
}
}
}
}Cost API Endpoint
GET /api/admin/costs returns pricing constants and query examples for cost dashboards.
Local Development
npm install
cp .dev.vars.example .dev.vars
# Edit .dev.vars with your CLOUDFLARE_AI_GATEWAY_API_KEY + CLOUDFLARE_API_TOKEN + CF_AI_GATEWAY_ACCOUNT_ID + CF_AI_GATEWAY_GATEWAY_ID
npm run startEnvironment Variables
For local development, create .dev.vars:
CLOUDFLARE_AI_GATEWAY_API_KEY=your-provider-api-key
CLOUDFLARE_API_TOKEN=your-ai-scoped-api-token # REST transport; use the same AI-scoped credential locally
CF_AI_GATEWAY_ACCOUNT_ID=your-account-id
CF_AI_GATEWAY_GATEWAY_ID=your-gateway-id
DEV_MODE=true # Skips CF Access auth + device pairing
DEBUG_ROUTES=true # Enables /debug/* routesWebSocket Limitations
Local development with the Vite dev server has issues proxying WebSocket connections through the sandbox. HTTP requests work but WebSocket connections may fail. Deploy to Cloudflare for full functionality.
Docker Image Caching
The Dockerfile includes a cache bust comment. When changing start-openclaw.sh, bump the version:
# Build cache bust: 2026-02-06-v28-openclaw-upgradeGateway Configuration
OpenClaw configuration is built at container startup:
- R2 backup is restored if available (with migration from legacy
.clawdbotpaths) - If no config exists,
openclaw onboard --non-interactivecreates one based on env vars - Feature flags come from the per-customer
MOLTLAZY_FEATURE_FLAGSbinding (managed via the FourIA API) start-openclaw.sh startrunsmoltlazy patch(after the restore/onboard above) which generates per-section include files (moltlazy-session/tools/logging.json) and injects them as per-section$includedirectives (no root$include);gateway.*/agents.*are written inline and the gateway token is an env SecretRef- The Worker calls
/usr/local/bin/start-openclaw.sh startonce: that mode acquires the container-wide startup lock before running the setup body, runs the patch while holding it, thenexecs the supervised Gateway under the same lock (theexecreplaces the wrapper, so the tracked process is the gateway) and runsstart-openclaw.sh post-startafterward once the port is ready. The Worker no longer invokessetupthenlaunchseparately; those remain standalone smoke/manual modes only.
Feature Flag Configuration
Feature flags are managed via the FourIA API and applied as a per-customer binding (MOLTLAZY_FEATURE_FLAGS, a JSON string). The Worker reads the binding and passes it to the container via buildEnvVars(). The legacy CONFIG_BUCKET / config-cache.ts (R2 + ETag) mechanism was removed.
Feature Flags Example:
{
"knowledgeGraph": true,
"unifiedBilling": false,
"softwareProvisioning": false
}
The container consumes MOLTLAZY_FEATURE_FLAGS at startup (moltlazy patch generates the per-section include files and injects them as per-section $include into openclaw.json). There is no worker-side cache — the binding is read directly.
Software provisioning (softwareProvisioning)
Software that agent skills install at runtime (plugins, apt/npm packages,
downloaded binaries) lives in the container filesystem only, so it is lost on a
container-version upgrade (fresh image) or idle-sleep (container destroyed).
The per-customer softwareProvisioning flag fixes this:
- The tenant’s manifest (
customers/<slug>/software.jsonin theBACKUP_BUCKET, uploaded viaPUT /api/admin/software/manifest) declares each package with its actual install command:{ id, check, install, verify?, timeoutSec? }. - On cold start,
ensureGatewayImplmaterializes the manifest into/home/openclaw/.openclaw/software.jsonand — only when the flag is on — spawnsmoltlazy software provisionas an independent supervised process through the Sandbox SDK (void spawnSoftwareProvision(...)), not a step instart-openclaw.sh. It runs in parallel with the gateway boot, never blocks readiness, and is non-fatal (per-package timeouts; failures recorded, not raised). - It writes live ticks to
/tmp/software-progress.log(read byGET /api/startup-progress//api/startup/wsassoftwareLog) and a structured report to/tmp/software-report.json(surfaced byGET /api/admin/software/reportas the homepage Software widget). Each run truncates the progress log first (resetProgressLog), because/tmpsurvives an in-place restart (/debug/restart-openclaw), which would otherwise interleave ticks from earlier boots. POST /api/admin/software/provision(ADMIN/ROOT) reconciles the manifest on demand, so a manifest uploaded or corrected mid-life does not need a restart. It re-materializes the current R2 manifest, runs the same CLI to completion bounded bySOFTWARE_PROVISION_RUN_TIMEOUT_MS(5 min), and returns the fresh report;409when the flag is off (disabled) or nothing is declared (no_manifest),502when the run fails.- Both
GETendpoints carryenabled(the flag), so the dashboard can tell “switched off”, “asleep”, “not run yet” and “nothing declared” apart instead of showing one empty state. The widget exposes a Run provisioning button to ADMIN/ROOT when the flag is on.
The manifest is arbitrary shell executed inside the tenant container, so the
upload endpoint is ADMIN/ROOT-only (never exposed to readers), timeoutSec is
per package, and the on-demand run is bounded by the route.
Key modules: src/gateway/software-provisioning.ts (flag + spawn + on-demand
run + report parsing), src/services/software-manifest/ (R2 store +
materialization), src/routes/api/software.ts (manifest read/upload, report,
on-demand provision), packages/moltlazy/src/software/ (the CLI).
AI Provider
Cloudflare AI Gateway is the only provisioned AI provider path. The startup script configures auth when all of these env vars are set:
- Cloudflare AI Gateway (native):
CLOUDFLARE_AI_GATEWAY_API_KEY+CF_AI_GATEWAY_ACCOUNT_ID+CF_AI_GATEWAY_GATEWAY_ID
Container Environment Variables
These are the env vars passed TO the container (internal names):
| Variable | Config Path | Notes |
|---|---|---|
CLOUDFLARE_AI_GATEWAY_API_KEY | (env var) | Native AI Gateway key |
CLOUDFLARE_API_TOKEN | (env var) | AI Gateway REST API Bearer token; tenant binding is derived from the AI key |
CF_AI_GATEWAY_ACCOUNT_ID | (env var) | Account ID for AI Gateway |
CF_AI_GATEWAY_GATEWAY_ID | (env var) | Gateway ID for AI Gateway |
OPENCLAW_GATEWAY_TOKEN | --token flag | Mapped from MOLTBOT_GATEWAY_TOKEN |
OPENCLAW_DEV_MODE | (dev-agents flag) | Derived from the -dev hostname or an OPENCLAW_DEV_MODE override; DEV_MODE local |
OPENCLAW_RESTORED | (zero-data-loss signal) | true when a cold start restored an R2 backup (hadBackup && restored); makes start-openclaw.sh skip the setup body |
TELEGRAM_BOT_TOKEN | channels.telegram.botToken | @deprecated — E2E only. Use integrations dashboard in production. |
DISCORD_BOT_TOKEN | channels.discord.token | @deprecated — E2E only. Use integrations dashboard in production. |
SLACK_BOT_TOKEN | channels.slack.botToken | @deprecated — E2E only. Use integrations dashboard in production. |
SLACK_APP_TOKEN | channels.slack.appToken | @deprecated — E2E only. Use integrations dashboard in production. |
OpenClaw Config Schema
OpenClaw has strict config validation. Common gotchas:
agents.defaults.modelmust be{ "primary": "model/name" }not a stringgateway.modemust be"local"for headless operation- No
webchatchannel - the Control UI is served automatically gateway.bindis not a config option - use--bindCLI flag
See OpenClaw docs for full schema.
Common Tasks
Adding a New API Endpoint
- Add route handler in
src/routes/api.ts - Add types if needed in
src/types.ts - Update client API in
src/client/api.tsif frontend needs it - Add tests
Adding a New Environment Variable
- Add to
MoltbotEnvinterface insrc/types.ts - If passed to container, add to
buildEnvVars()insrc/gateway/env.ts - Update
.dev.vars.example - Document in README.md secrets table
Debugging
# View live logs
npx wrangler tail
# Check secrets
npx wrangler secret list
Enable debug routes with DEBUG_ROUTES=true and check /debug/processes.
R2 Backup Encryption (F4E1)
When BACKUP_ENCRYPTION_KEY (base64 32-byte secret) is set, the Worker post-encrypts every
snapshot after sandbox.createBackup(): backups/<id>/data.sqsh is replaced by
data.sqsh.enc (chunked AES-256-GCM, per-object HKDF key) and meta.json is flagged
encrypted: true (kept readable — the SDK needs it). Restores decrypt transiently and delete
the plaintext in a finally block. Config versions under configs/ are encrypted with
customMetadata.encrypted='true'. Legacy plaintext backups remain restorable.
Set BACKUP_ENCRYPTION_REQUIRED=true to fail closed. Crypto lives in src/crypto/.
workerd’s R2Bucket.put() rejects ReadableStream bodies of unknown length, so uploads wrap
the ciphertext/plaintext stream in a FixedLengthStream whose size is computed from the source
object (encryptedLength) or the F4E1 header (decryptedLength). This keeps backup/restore
memory-bounded (no whole-archive buffering) while satisfying R2. See src/crypto/r2-backup.ts.
See docs/plans/r2-encryption-at-rest.md.
R2 Storage Notes
Persistence uses the Sandbox SDK createBackup() / restoreBackup() API with
the BACKUP_BUCKET binding. The snapshot root is /home/openclaw; it is not an
s3fs mount and there is no /data/moltbot runtime path.
Never delete tenant backup objects outside the retention or decommission procedure. Hard restarts must checkpoint before killing the gateway.
-
Process status: The sandbox API’s
proc.statusmay not update immediately after a process completes. Instead of checkingproc.status === 'completed', verify success by checking for expected output (e.g., timestamp file exists after sync). -
R2 prefix migration: Backups are now stored under
openclaw/prefix in R2 (wasclawdbot/). The startup script handles restoring from both old and new prefixes with automatic migration. -
Snapshot lease + admin snapshot mode (F5): the shared R2
backup-lockTTL isSTEP_DEADLINE_MS(12 minutes), with a serialized heartbeat every quarter-TTL and explicit renewals around slow snapshot/restore boundaries. Renewal events are structured; renewal failure is best-effort and does not abort work. The longer TTL intentionally favors avoiding false expiry during a live durable step, at the cost of a longer stale-holder takeover window.POST /api/admin/storage/syncenqueues a durable job by default whenACTIVITYis available; use?inline=1only when synchronous completion is required (?async=1keeps its prior behavior). WithoutACTIVITY, the route falls back to inline work.
Cloudflare Tunnel Setup (Manual Flow)
This section documents how to expose an internal service (e.g. the Obsidian sync port 18790) via a Cloudflare Tunnel with a public hostname. Follow this end-to-end when setting up a new tunnel route.
1. Register and Add a Domain
You need a domain zone in your Cloudflare account before any DNS record can be created.
- Register a domain (e.g. via Cloudflare Registrar or a third-party registrar like Spaceship/Namecheap).
- In the Cloudflare Dashboard, select the Paso4 account → Websites → Add a site.
- Enter the domain name and choose the Free plan.
- Cloudflare will assign two nameservers, e.g.:
albert.ns.cloudflare.compriscilla.ns.cloudflare.com
- At your registrar, replace the existing nameservers with the Cloudflare ones.
- Wait for DNS propagation (typically 5–30 minutes). The zone status will change from
pendingtoactive.
Current domain: paso4.io (zone ID: 6ce23f544f57a1909578032abcda18dd, account: Paso4)
2. Create the Cloudflare Tunnel
If the tunnel does not yet exist, create it with cloudflared:
cloudflared tunnel create obsidian-sync
This prints a tunnel UUID and writes credentials to ~/.cloudflared/<UUID>.json.
Current tunnel: obsidian-sync — UUID 90792969-7077-40a8-ae10-28e716290d9d
To get the tunnel token (needed for CLOUDFLARE_TUNNEL_TOKEN):
cloudflared tunnel token 90792969-7077-40a8-ae10-28e716290d9d
Set it as a Worker secret:
npx wrangler secret put CLOUDFLARE_TUNNEL_TOKEN3. Create the DNS CNAME Record
A CNAME record must point from your public subdomain to <UUID>.cfargotunnel.com.
Via Cloudflare Dashboard:
-
Go to paso4.io → DNS → Records → Add record.
-
Fill in:
Field Value Type CNAMEName obsidian-syncTarget 90792969-7077-40a8-ae10-28e716290d9d.cfargotunnel.comProxy status Proxied (orange cloud — required) TTL Auto -
Save. Public URL becomes
https://obsidian-sync.paso4.io.
Via MCP API (requires DNS:Edit on the API token):
cloudflare.request({
method: 'POST',
path: `/zones/6ce23f544f57a1909578032abcda18dd/dns_records`,
body: {
type: 'CNAME',
name: 'obsidian-sync',
content: '90792969-7077-40a8-ae10-28e716290d9d.cfargotunnel.com',
proxied: true,
comment: 'Cloudflare Tunnel route for obsidian-sync',
},
});
To grant DNS:Edit to the MCP token: https://dash.cloudflare.com/profile/api-tokens → edit the token → add Zone → DNS → Edit.
4. Configure the Tunnel Ingress
Create or update ~/.cloudflared/config.yml on the host running cloudflared:
tunnel: 90792969-7077-40a8-ae10-28e716290d9d
credentials-file: /root/.cloudflared/90792969-7077-40a8-ae10-28e716290d9d.json
ingress:
- hostname: obsidian-sync.paso4.io
service: http://localhost:18790
- service: http_status:404
Then run:
cloudflared tunnel run obsidian-sync5. Verify
# Check tunnel is connected
cloudflared tunnel info obsidian-sync
# Test the public endpoint
curl -I https://obsidian-sync.paso4.ioGotchas
- The
cfargotunnel.comCNAME only routes traffic for DNS records in the same Cloudflare account as the tunnel. Cross-account routing is blocked. - The zone must be active (nameservers delegated) for public DNS to resolve. A
pendingzone means nameserver propagation is not complete yet. - If the tunnel stops, the DNS record is not deleted — visitors will see a Cloudflare
1016error. - The
CLOUDFLARE_TUNNEL_TOKENsecret in the Worker is used bystart-openclaw.shto runcloudflaredinside the container, exposing port 18790 to the tunnel.
FourIA offers Private Cloud only (Business On Premise is not offered at this time). Pricing is credit-based: Credits are the exclusive billable unit, consumed by completed messages and automations. Credits do NOT include AI costs — AI expenses are managed separately via BYOL or AI Gateway billing. Reader users are unlimited and free. Builder seats (users who can create and edit agents) are plan-limited. The v1.0-beta is delivered through the Design Partner Program with a single engaged partner. Partners pay a fixed credit fee negotiated with sales. See docs/VISION.md for the full pricing definition.
Active Technologies
- OpenTofu 1.6+ (HCL) +
cloudflareprovider (Terraform/OpenTofu) (001-opentofu-deploy) - Cloudflare R2 (for FourIA data), OpenTofu Local/Remote State (for infrastructure tracking) (001-opentofu-deploy)
Recent Changes
-
Gateway memory is observable from the Worker, and a dev instance self-heals before the container limit. Investigating a
tenant-tarjetero-dev“reboot” required SSH-ing into the container to read/proc,dmesgand the gateway log, because nothing exposed the gateway’s RSS. (1) Observability (plan Phase 1):src/gateway/memory.tsreadsVmRSS+Threadsfrom/proc/<pid>/statusfor the gateway server (pgrep -x 'openclaw-gatewa', the same exactcommthe soft reload targets, zombies skipped) in ONE shell exec, throttled toGATEWAY_RESOURCE_SAMPLE_INTERVAL_MS(60s) so the frequent health poll adds at most one exec per interval.attachGatewayResources()(routes/public.ts) attaches it to/api/health,/api/health-snapshotand/api/health-wsasgateway: { pid, rssBytes, threads }— only while the snapshot isrunning, so a sleeping tenant is never woken — and writes agateway-resourcesAnalytics Engine row (double1= RSS bytes,double2= threads). (2) Watchdog (plan Phase 2.4): on a dev instance (isDevAgentsEnabled) atGATEWAY_RSS_RESTART_BYTES(3 GiB of the 4 GiB container) the gateway is replaced viakillGateway+resetStartupCache+ensureGateway({ skipRestore: true, bypassMaintenance: true })— the only lever that reclaims the per-worker_threadV8 heaps — with a 30-minute cooldown claimed before acting so concurrent health polls cannot double-restart, dispatched detached so a multi-minute replacement never delays a health response. Deliberate design: the threshold guards the container limit, not OpenClaw’s 1.5 GiBrss_thresholdwarning (which fires permanently on a healthy ~2.1 GiB dev gateway and would make the watchdog a restart loop), and there is no soft-reload step (a measured in-processSIGUSR2restart does not reclaim RSS: 1.14 → 1.38 → 1.18 GiB). Production is excluded. Tests:src/gateway/memory.test.ts,src/metrics.test.ts(trackGatewayResources),src/routes/public.test.ts(sample attached + AE row, one read per interval, no sample when not running, no watchdog on a non-dev instance); corpustest/business-rules/operational-modes.txt; docsdocs/plans/sandbox-memory-usage.md,apps/fouria/docs/WORKER-DEVELOPMENT.md,apps/fouria/docs/OPERATOR-GUIDE.md(§5.4d). -
OpenClaw migrated to 2026.10.8 (
catalog.openclaw+ the@openclaw/gateway-client/@openclaw/gateway-protocolpins in the rootpackage.json; the plugin repo tracks and publishes the same release). 2026.10.8 is a bug-fix/security release: plugin lifecycle repairs (older plugins can process incoming messages again after an upgrade, scheduled plugin work no longer stops after startup/reload, and Gateway startup repairs stale managed plugin links left by an external OpenClaw reinstall — the state a warm R2 restore can produce), session-store fixes (relative session-store paths no longer fork a second database;openclaw doctor --fixrepairs an already-split store), Doctor robustness on very large retained session histories (stack overflow / retained memory), and a trusted-proxy browser-auth fix that no longer writesgateway.auth.passwordor restarts the Gateway on first browser use. It also hardens POSIX exec allowlists (test -v/[ -vare no longer auto-trusted; moltlazy writes no exec allowlist, and the dev-onlysecurity.installPolicy allowis unaffected). No config-schema changes — the moltlazyopenclaw config validatecctr contract passes unchanged, so nopackages/moltlazysource change was needed — and the Node engine floor is unchanged at>=24.22.1 <25 || >=26.12.1, so no Dockerfile change. No component CalVer bump — the release train handles that. -
Software provisioning is now usable mid-life and honest about its state. Three gaps closed around the
softwareProvisioningfeature. (1) On-demand reconcile:POST /api/admin/software/provision(ADMIN/ROOT) re-materializes the current R2 manifest, runsmoltlazy software provisionto completion (bounded bySOFTWARE_PROVISION_RUN_TIMEOUT_MS, 5 min) and returns the fresh report, so a manifest uploaded or corrected mid-life no longer waits for the next cold start;409 { code: 'disabled' | 'no_manifest' }and502 { code: 'failed', error }are distinct outcomes. (2) No cross-boot tick bleed:runSoftwareProvisiontruncates/tmp/software-progress.logbefore every run (resetProgressLog) —/tmpsurvives an in-place/debug/restart-openclaw, so the boot terminal used to interleave ticks from earlier boots. (3) Truthful dashboard states:GET /api/admin/software/{manifest,report}now returnenabled(the flag), and the homepage widget separates “provisioning is disabled”, “instance is asleep”, “no report yet” and “no software declared”, with a Run provisioning button (ADMIN/ROOT) that renders the on-demand report and surfaces its error.parseSoftwareReportmoved intosrc/gateway/software-provisioning.tsso the report, run and route paths share one shape check. Tests:src/gateway/software-provisioning.test.ts,src/routes/api/software.test.ts,src/client/pages/__tests__/HomePage.test.tsx,packages/moltlazy/tests/software/run.test.ts(progress-log reset), E2Etest/e2e/zzz_software_provisioning.txt(on-demand run must install the new package and skip the already-installed one); corpustest/business-rules/dashboard-api-contract.txt,operational-modes.txt; docsapps/fouria/docs/WORKER-DEVELOPMENT.md. -
Lerma deploys are idempotent around Cloudflare DO migration conflicts (#516). A tenant Worker’s Durable Object class set was derived from the script’s bindings, but Cloudflare validates
new_sqlite_classesagainst the script’s migration history, so the two can diverge: an upgrade redeploy that binds fewer classes (or a rollback that deletes the script while leaving its classes migrated) made every subsequent deploy re-declareTelegramIngressand fail with400 code 10074— deterministically, so provisioning could not recover on its own.Lerma.Cloudflare.DispatchNamespace.upload_script/6now retries the upload without any class Cloudflare reports as already migrated (10074names it), switching an implicit full first-deploy set to an explicitnew_sqlite_classeslist; each retry removes at least one class, so it terminates, and the retry is logged.Lerma.Instances.instance_bindings/2now re-declares the same DO set as provisioning (Sandbox,ACTIVITY,TELEGRAM_INGRESS, plusBACKUP_WATCHwhile the flag is on) so an upgrade cannot detach a migrated class from its binding. Provisioning cleanup and a failed atomic deploy tear the classes down first viadelete_durable_object_namespaces/2(never on an upgrade rollback, whose Durable Objects hold the live tenant’s state:preserve_do_namespaces). Tests:apps/lerma/test/lerma/cloudflare/dispatch_namespace_test.exs(migrated-but-unbound re-deploy, resurrected first deploy, unrelated error not retried),containers_test.exs,tenant_provisioner_pipeline_test.exs; docsapps/lerma/README.md. -
OpenClaw migrated to 2026.10.6 (
catalog.openclaw+ the@openclaw/gateway-client/@openclaw/gateway-protocolpins in the rootpackage.json; the plugin repo tracks and publishes the same release). 2026.10.6 is a security patch release (eMudhra certificate-compromise remediation, Windows setup/Doctor/Gateway startup fixes, Discord command and iPhone Talk fixes) with no config-schema changes: the moltlazyopenclaw config validatecctr contract passes unchanged, so nopackages/moltlazysource change was needed. It does raise OpenClaw’s Node engine floor from>=24.16.0 <25 || >=26.1.0to>=24.22.1 <25 || >=26.12.1, soapps/fouria/Dockerfilenow installs Node v24.22.1 (from v24.20.0) and its version gate compares major/minor/patch (24.22.0 is rejected). No component CalVer bump — the release train handles that. -
Idle-sleep no longer re-archives an unchanged tenant, and deprovision snapshots reset the cadence.
TenantSandbox.onActivityExpired()used to enqueue a fulldeprovisionsnapshot on every idle expiry; on a 422 MB tenant the snapshot’s own RPCs renewed the DO activity window, sostopIfIdle()refused, the next expiry snapshotted again, and the container never stopped (observed ontenant-tarjetero-dev: three 422 MB archives at 02:35/02:39/02:44 UTC plus three more at 05:39/05:43/05:49, ~3 GB/day, containerrunningsince Oct 21). The hook now reads the catalog first (hasRecentSnapshot()insrc/persistence.ts, a pure read): a committed snapshot inside the idle window (resolveSleepAfterMs()— staging 15m, dev 2h, production 6h) already covers the unchanged filesystem, so it stops inline throughstopIfIdle()without a new archive; no bucket or a read failure falls back to the durable backup-then-stop job (fail closed). Deprovisionfinalizenow passesstampScheduled: truetoo, so a committed archive resetslastScheduledBackupand the next cron tick does not archive the same state again. Tests:src/sandbox-do.test.ts(recent snapshot stops inline, probe failure snapshots, no bucket skips the probe),src/persistence.test.ts(hasRecentSnapshot),src/activity/scheduler.test.ts(deprovision stamps),src/build-sandbox-options.test.ts(resolveSleepAfterMs); corpustest/business-rules/operational-modes.txt; docsapps/fouria/docs/WORKER-DEVELOPMENT.md. -
The dev Fouria Builder heartbeat is reconciled to its intended cadence. OpenClaw 2026.10.5 retired
agents.entries.*.heartbeat: heartbeats are ordinary automation jobs, and config reloads/restarts never rewrite an existing job. The migratedheartbeat-fouria-builderjob therefore kept the default 30-minute cadence for 9 days ontenant-tarjetero-devdespite moltlazy’sheartbeat.every: 12h, running full agent turns on the main session (~317k tokens each) until the OpenCode subscription hit 0% (402 Insufficient account funds→agent-runner-failure).start-openclaw.shgains a dev-onlyensure_dev_heartbeat_cadence(post-start) that readsopenclaw automations get <id> --json, comparesschedule.everyMs/sessionTarget/payload.lightContext, and only then edits the job to--every 12h --session isolated --light-context(split so a rejected--light-contextcannot lose the cadence fix). Idempotent: a matching job is never touched, so the schedule anchor is not reset on every boot. Docspackages/moltlazy/AGENTS.md,apps/fouria/docs/WORKER-DEVELOPMENT.md. -
Backup catalog reads are pure; catalog healing has an explicit entry point (audit finding F3).
GET /api/admin/storage/backups(getBackupHistory) used to runreconcileOrphanedBackupson every dashboard poll — listingbackups/, readingmeta.json, adopting orphans, deleting over-cap archives, and writingbackup-handle.jsonfrom aGET. It now performs three parallel reads (getPersistenceStore,getStableBackup,isRecoveryHeld) and nothing else. The write path isreconcileCatalogState(adoption + retention prune inside the etag-CAS mutation, reporting the ids it changed) →repairBackupCatalog(sweep metadata-less uploads + in-place encryption) → refreshed history, exposed asreconcileBackupCatalog; the durablerepairstep now runs it before thegatestep of every scheduled snapshot, so the catalog still self-heals after a killed invocation (issue #344). Restore/schedule keep their inline reconcile,debug/reprovisionreconciles before deciding nothing is restorable (a crashed upload must not look unrecoverable), and operators getPOST /api/admin/storage/reconcile(ADMIN/ROOT). The history panel pollsGET /api/admin/storage/jobs/:id(getBackupJobStatus) while a snapshot is in flight instead of re-listing the catalog every 5s, falling back to the history diff when no job id is returned. Tests:src/persistence.test.ts,src/persistence-encryption.test.ts,src/routes/api/storage.test.ts,src/activity/scheduler.test.ts,src/routes/debug.test.ts,src/client/components/__tests__/BackupHistoryPanel.test.tsx; corpustest/business-rules/operational-modes.txt; docsapps/fouria/docs/WORKER-DEVELOPMENT.md,apps/fouria/docs/OPERATOR-GUIDE.md,docs/plans/durable-agents-rules-audit.md. -
A config repair is deferred until a restore settles and the gateway is up on the restored state, and the restored config becomes the stable/latest config version. A restore replaces the writable layer but launches nothing, so a
doctor --fixapplied around it was silently overwritten by the next re-apply of the recovery point (suspected ontenant-tarjetero-devafter a config fix). (1) Every successful restore (restoreIfNeeded,restoreBackupById) writes a durablerestore-config-pending.jsonmarker. (2) The boot job, the durable restore job, and the sync/async-fallback admin restore route finish through the sharedlaunchRestoredState(src/post-restore.ts):ensureGateway({ waitForReady: true })on the restored state, thencaptureRestoredConfigVersionwritesconfigs/<ts>.json+stable/config.json(the newest/stable version) and clears the marker. A gateway-launch failure does not re-run the destructive restore: restore jobs succeed withdetail: "gateway-launch-pending", the sync route returnsgatewayReady: false, and the marker drives the next tick. (3)doctor-repairchecks an active restore job,getPersistenceBusyReason(maintenance window / restore in flight), and the pending marker before running; when blocked it throwsMaintenanceJobDeferredError, andrunMaintenanceJobAlarmreschedules it every 60s without consuming an attempt (previously three deferrals would have failed the job before the restore finished). Tests:src/activity/maintenance-jobs.test.ts,src/activity/scheduler.test.ts,src/persistence.test.ts,src/routes/api/storage.test.ts; corpustest/business-rules/operational-modes.txt; docsapps/fouria/docs/WORKER-DEVELOPMENT.md,apps/fouria/docs/OPERATOR-GUIDE.md. -
Diagnostics dashboard is skip-not-park and non-blocking.
DiagnosticDashboardonly dispatches its container reads (/api/admin/config/raw,/api/admin/config) while the gateway is actually running (useGatewayRunning()—snapshot.readiness.ok, ignoring the ROOT breaker bypass) and re-runs them when it comes back; the debug-routes flag (Worker-local) still loads. Every load is deferred after paint (scheduleAfterPaint→requestIdleCallback,setTimeout(0)fallback), so no widget waits on a container response:ConfigSectiongained agatewayReadyprop and now keeps its tabs/editor visible with a non-blocking “Loading configuration…” / “Waiting for the gateway to start…” notice instead of swapping the whole panel for a spinner; devices render a waiting state; config loads never clobber text typed before the response arrived. Gateway-only actions (device list/approve, config save, channel probe) are guarded, and the Debug Tools that only make sense against a live gateway (gateway-api,stop-gateway) are skipped with an inline explanation — container diagnostics (logs/processes/version/sqlite) and recovery actions (restart/rescue/destroy) stay available so a down gateway can still be inspected and revived. Repair stays available outside the editor:doctor --fixis the recovery action for a down gateway, sorunRepairno longer requires a running gateway, a dedicated Repair Environment bar (data-testid="repair-environment-button", ADMIN/ROOT) sits above the widgets and is disabled only while a repair is in flight, andConfigSection’s repair button stays enabled whilegatewayReadyis false. Tests:src/client/pages/__tests__/DiagnosticDashboard.test.tsx,src/client/components/__tests__/ConfigSection.test.tsx,src/client/GatewayHealthContext.test.tsx; corpustest/business-rules/dashboard-api-contract.txt; docsapps/fouria/docs/WORKER-DEVELOPMENT.md,docs/TESTING.md. -
Billable emission: staging always emits,
BILLABLE_EVENTSopts a dev instance in. Both emit sites (POST /api/otel/v1/tracesand theCRON_BILLABLE_DRAINcron drain) previously gated on the strict billable cost axis (resolveInstanceDimensions(env).billable= production platform + production environment), so staging and dev instances could never exercise the billing pipeline. They now callisBillableEventsEnabled(env)(src/dev-mode.ts): billable production always emits and cannot be opted out, the staging platform always emits in every environment (mirroring production so billing is validated before release; the rows land in the staging Analytics Engine dataset), and a dev environment emits only withBILLABLE_EVENTS=true. Emission only:getCostContextkeeps the realplatform/environmentdimensions (staging/devremain non-billable cost axes) andshouldAutoSleepis untouched, so staging keeps its 15m idle timeout and an opted-in dev instance its 2h one. The emitting path omits the guard’sbillable: falsemarker (the local-E2E skip, whose worker is a dev instance viaDEV_MODE, is unaffected). Tests:src/dev-mode.test.ts,src/routes/otel-traces.test.ts(staging emits, dev opt-in emits),src/index.test.ts(cron drain: dev drops, staging and dev-with-override emit),src/build-sandbox-options.test.ts(sleep policy unchanged); corpustest/business-rules/dev-expense.txt; docsdocs/ENV-VARIABLES.md,docs/TESTING.md,apps/fouria/README.md,apps/fouria/docs/WORKER-DEVELOPMENT.md,.dev.vars.example. -
Request paths never run a destructive archive restore, and a container stops only on verified inactivity (investigated on
tenant-tarjetero-dev, which appeared to “reboot” while in active use). (1) The container instance had not restarted (started_at00:18, stillrunning); what died was the gateway process ([Gateway] Gateway process … has terminal state: exited), and the follow-up boot ran a 350 MB archive restore under request execution contexts.sandbox.restoreBackupwas aborted at ~30.1 s (backup.restore error … The operation was aborted (30177ms), plussandbox.exec … The operation was aborted (30001ms)for health probes/umountand the workerd “promise resolved in a different request context” warning) because Cloudflare cancels a request’s background continuations shortly after the response — so the restore never finished and every reload re-attempted it (6 boot jobs + 8 restore errors in one hour). Fix:restoreIfNeeded(..., { restoreOwner: 'request' })(theensureGatewaydefault) performs only the cheap non-destructive checks (therestore-neededlatch +containerHasState()) and otherwise throwsRestoreDeferredToBootJobError;ensureGatewayrequests the durable boot job and throwsGatewayBootPendingError(proxy answers 503/loading). Durable callers (ActivitySchedulerboot/wake jobs, the queue consumer) passrestoreOwner: 'durable'. The proxy’s three inlinerestoreIfNeededcalls (proxy.request,proxy.websocket-recovery,proxy.fetch-recovery) were removed. (2) A container is now stopped only after a stop-time inactivity check:TenantSandbox.stopIfIdle()re-reads the DO’s own idle window (renewed by every request) and the process list, and the deprovision job calls it after its multi-minute snapshot instead ofsandbox.stop(); if the tenant became active meanwhile the job isskipped(active:<reason>). Refusals:activity,transient-activity,probe-failed,billable-production,not-running. Tests:src/sandbox-do.test.ts(stopIfIdlecases),src/activity/scheduler.test.ts(stop-time skip),src/gateway/process.test.ts(deferral +restoreOwner),src/persistence.test.ts(request deferral / state-present short-circuit),src/index.test.ts(proxy no longer restores); corpustest/business-rules/operational-modes.txt; docsapps/fouria/docs/WORKER-DEVELOPMENT.md,apps/fouria/docs/OPERATOR-GUIDE.md,webs/fouria-docs/.../worker-development.md,operator-guide.md. -
Fouria Builder build-method skill. The dev-only
fouria-builderagent now seedsfouria-product-buildas a per-agent workspace skill (packages/moltlazy/workspace/skills/fouria-product-build.ts→<workspace>/fouria-builder/skills/fouria-product-build/SKILL.md, plus an entry in itsskillsallowlist) that sequences the build method: frame the production roster → tool-first restriction model (each capability names the exact tool; no tool means no capability) → smallest MCP tool surface withpreview/applypairs → 3–6 channel quick actions (Telegram inline buttons; a callback is untrusted input, never authorization) → per-client tone matrix before writing a client-facingSOUL.md→ one vertical slice →cctracceptance corpus →POC.mdhandoff. Workspace skills are OpenClaw’s highest-precedence skill source and are per-agent, andworkspace/fouria-builderis excluded from promotions, so no production roster changes.seedWorkspaceFilesnow creates parent directories so workspace-file keys can be nested paths. This supersedes the pending Skill Workshop proposalfouria-product-build-20261021-bf543e21c1: a workspace copy of the same name shadows a workshop skill, so the repo copy is authoritative. Tests:packages/moltlazy/tests/agents.test.ts; docspackages/moltlazy/AGENTS.md,packages/moltlazy/src/agents/README.md,docs/DEVELOPMENT.md. -
Cloudflare Observability destination ingestion (issue #337). Account-level Observability → Destinations now push Workers Logs and Workers Traces to Lerma over OTLP/HTTP JSON, giving event-driven worker diagnostics and uptime statistics that scale with traffic instead of tenant count. (1) Config:
apps/fouria/wrangler.jsonc(env.staging→lerma-worker-logs/lerma-worker-traces,env.production→lerma-worker-logs-prod/lerma-worker-traces-prod) andLerma.Cloudflare.Observability.worker_metadata/0reference the environment’s destination names, with per-signal head sampling (10% at rollout) andpersist: falsewhen a destination is configured (Lerma is the destination of record). The Lerma CD passes the matching names to staging and production, and the token to both; it also reconciles the account-level destination pairs (mix lerma.observability.destinations --warn-only) before each deploy so a fresh environment self-heals once the Cloudflare credential has Workers Observability Write. NewWORKER_OBSERVABILITY_LOGS/TRACES_DESTINATIONS,*_SAMPLING_RATEand*_PERSISTenv vars. (2) Ingest: newPOST /api/ingest/otel/v1/logsand/v1/traces(bearerOTEL_INGEST_TOKEN, constant-time compare, fail closed; OTLP/HTTP JSON only — protobuf →415+ hint;202 {accepted, skipped}so Cloudflare never retry-loops).Lerma.Observability.Otlpis a pure allowlist parser: it keeps script/dispatch namespace/outcome/handler/status/exception class names/cpu+wall/trace+span+invocation ids/timestamps, drops log text, stacks, URLs/query strings, user agents, request/response bodies and DB query text, and keeps only invocation root spans. Rows land inobservability_eventswith a uniquededupe_key+ON CONFLICT DO NOTHING, so replayed batches (at-least-once delivery) add no rows. Cloudflare gzips each OTLP/HTTP JSON batch while keepingContent-Type: application/json;LermaWeb.Plugs.OtlpBodyReaderdecompresses it beforePlug.Parsers(otherwise the destination retries on a400forever). (3) Uptime:GET /api/analytics/v1/uptime/:script_name?window=24h|7d|30d|<n>h|<n>dcomputes uptime/failure ratios from pushed root spans only; no events observed (including under sampling) →nullratios +status: "unknown", never downtime. The instance page gained a Worker telemetry widget (window selector + uptime/failure/invocation/CPU/wall), andLerma.Observability.Prunersweeps 90-day retention daily.HealthMonitorand billing flows are untouched. Tests:test/lerma/observability/otlp_test.exs,telemetry_test.exs,test/lerma/cloudflare/observability_test.exs,test/lerma_web/controllers/api/otlp_ingest_controller_test.exs,observability_controller_test.exs,test/lerma_web/components/worker_telemetry_widget_test.exs,test/lerma_web/live/instance_live_test.exs; docsdocs/CLOUD-ARCHITECTURE.md,docs/ENV-VARIABLES.md,docs/TESTING.md,apps/fouria/public/articles/PRIVACY.md,apps/fouria/docs/WORKER-DEVELOPMENT.md,SECRETS.example.yaml,docs/schemas/secrets-manifest.schema.json,apps/lerma/.env.example. -
OpenClaw migrated to 2026.10.5 (
catalog.openclaw+ the@openclaw/gateway-client/@openclaw/gateway-protocolpins in the rootpackage.json; the plugin repo tracks the same release). One breaking change hit moltlazy: the 2026.10.5 config schema retiredagents.defaults.compaction.enabled(embedded auto-compaction is no longer toggleable) and retiredcompaction.truncateAfterCompaction(“byte-triggered compaction now opts in viamaxActiveTranscriptBytesalone”) —openclaw config validaterejects both keys withInvalid input, which broke the moltlazy cctrconfig-savecontract.AGENT_DEFAULTS.compactionnow emits onlymaxActiveTranscriptBytes: '20mb'+keepRecentTokens: 20000, preserving the sandbox OOM transcript bound. The runtime zod schema still accepts the retired keys, so only theconfig validatecontract (and the published schema) caught it. Tests:packages/moltlazy/tests/agents.test.ts(asserts neither retired key is emitted); docspackages/moltlazy/AGENTS.md. -
Local development runs on the Cloudflare Vite plugin instead of
wrangler dev.bun run dev:worker/bun startstart the Worker withvite dev(vite.config.ts), the same pluginbun run buildalready used;bun run devserves the admin SPA fromvite.ui.config.ts(:8787) and proxies to the Worker (:8788). Local E2E fixtures (test/e2e/fixture/server/start-local,start-local-pair) launchvite dev, pass the old--varoverrides through the plugin’sconfighook (E2E_WORKER_NAME,PERSIST_STATE_PATH,VITE_CACHE_DIR,VITE_INSPECTOR_PORT), installtest/e2e/.dev.varsinto the Worker config directory for the run (the plugin has no--env-file;stop-local*restores the developer’s file), and fire cron ticks through/cdn-cgi/local/scheduledinstead of/__scheduled.--envis gone from the deploy/dry-run scripts:CLOUDFLARE_ENV=…selects the environment at build time andwrangler deploydiscovers the generateddist/fouria/wrangler.jsonvia.wrangler/deploy/config.json. Docs:README.md,docs/DEVELOPMENT.md,docs/TESTING.md,apps/fouria/README.md,apps/fouria/docs/WORKER-DEVELOPMENT.md,apps/fouria/docs/OPERATOR-GUIDE.md,test/e2e*/README.md. -
/healthzis now a real, fast liveness route. It previously fell through to the gateway-proxy catch-all (app.all('*')), which ranrestoreIfNeeded+ensureGateway+ awaitForGatewayReadyport wait — on a cold/sleeping instanceGET /healthztook ~108 s and was killed by the platform before responding (Workers Observability ontenant-client-dev:GET /healthz→ 503 @ ~108 s avg, 348 requests in 2 h). The newGET /healthzdoes exactly one non-wakingisRuntimeActive()read and returns200 { ok, ts, status }(active|idle) sub-second, never warming the container or probing the gateway; readiness stays on/api/health. It sits on the tenant hostname, so it stays behind the host Access app (not public). Tests:src/routes/public.test.ts(/healthz200 + noensureGateway/exec/listProcesses;idlewhen the runtime is off); docsapps/fouria/README.md,apps/fouria/docs/WORKER-DEVELOPMENT.md,docs/WORKER-HEALTH.md. -
Browser health endpoints moved to Access-safe sibling paths (
/api/health-ws,/api/health-snapshot,/api/health-components). On tenant hostnames every browser WebSocket andfetch()under/api/healthor/api/health/*failed (WebSocket connection to 'wss://…/api/health/ws?passive=1' failed, plus CORS/redirect failures onGET /api/health/componentsandGET /api/health): the path-scopedFourIA Health - <slug>Access app shadows those paths and its AUD is distinct from the host app’s (Cloudflare ignores a requestedaudon create and update — verified against the live API — so the AUDs cannot be unified). A browser logged into the host app therefore gets a cross-origin 302 login redirect to*.cloudflareaccess.comon/api/health*, which WebSocket handshakes cannot follow and browserfetch()rejects on CORS. The sibling paths/api/health-ws,/api/health-snapshot, and/api/health-componentssit outside the health app’s path prefix, so the host-app session the browser already holds passes (verified against livestaging-client-dev.fouria.io:/api/health-*resolves to the host appkid, while/api/healthand/api/health/*resolve to the health appkid). The Worker mounts the handlers at both path sets (healthHandler,healthComponentsHandler,healthWsHandlerinsrc/routes/public.ts);api.ts(subscribeHealth,getHealth,getCloudflareComponents) andloading.htmluse the-sibling paths, while/api/health,/api/health/ws, and/api/health/componentsremain for headless service-token clients (lerma). Tests:src/routes/public.test.ts,src/assets/loading.test.ts, UI stubstest/ui/support/api.ts+test/ui/cloudflare-components.spec.ts; docsapps/fouria/README.md,apps/fouria/docs/WORKER-DEVELOPMENT.md,docs/WORKER-HEALTH.md,docs/TESTING.md; lermaaccess_policy.exdocuments the constraint. -
Telegram ingress queue observability: notification-bell entry + home status widget. The per-tenant
TelegramIngressDO now records every ingress event (bounded counters + last 20) —queued(a cold update was enqueued and woke the container),forwarded(hot path),delivered, andfailed— viatelegram-ingress/status.ts, wired from the webhook route (queued/forwarded) and the queue consumer (delivered/failed).GET /api/admin/telegram-ingress/statusreturns the snapshot andGET /api/admin/telegram-ingress/wsstreams it (a frame per change,TELEGRAM_INGRESS_WS_POLL_INTERVAL_MS, rotated). The admin shell’sTelegramIngressProvideropens the socket only while the feature is in use (an account is registered or activity exists) and raises a notification-bell entry (“Telegram message queued — waking your instance”, or a delivery-failure alert); the newTelegramIngressWidgeton the home page shows state + queued/delivered/failed + last event, hidden until in use. Tests:src/telegram-ingress/status.test.ts,src/routes/api/telegram-ingress.test.ts,src/client/__tests__/telegram-ingress-context.test.tsx,src/client/components/__tests__/TelegramIngressWidget.test.tsx,src/client/pages/__tests__/HomePage.test.tsx; corpustest/business-rules/dashboard-api-contract.txt; docsapps/fouria/docs/WORKER-DEVELOPMENT.md. -
Homepage no longer crashes with
Cannot read properties of undefined (reading 'length').GET /api/admin/software/reportrancat <report> 2>/dev/null || echo '{}', so a tenant whosemoltlazy software provisionhad never run got a truthyreport: {}; the homepage Software widget then readsoftware.packages.lengthand threw on render (only for ADMIN — ROOT lands on diagnostics). The route now returnsreport: nullunless the payload is a real report (an object with apackagesarray), andHomePagereadssoftware?.packages ?? []defensively. Tests:src/routes/api/software.test.ts(missing/partial report →null),src/client/pages/__tests__/HomePage.test.tsx(truthy{}renders the empty state instead of crashing). -
Release CD promotes the already-built artifacts instead of rebuilding.
deploy-fouria.ymlused todocker buildthe container image from source and re-build the wrangler bundle at release time; it now promotes what CI already built: it resolves the release’s develop-mainline build commit (git rev-list -n1 --first-parent --invert-grep --grep='^chore: release' <tag> -- apps/fouria/ .github/workflows/test.yml), retags the Cloudflare registry imagefouria:unstable-<sha>→fouria:<tag>+latest(minting short-lived pull+push registry credentials viaPOST /accounts/{id}/containers/registries/{host}/credentials, mirroring Lerma’sImageRegistry), and copies the CI bundlefouria-bundles:unstable-<sha>/fouria.mjs→<tag>/+latest/in R2.deploy-dashboard.ymlnow tags the develop build with an immutablelerma:sha-<sha>in addition to:latest-pre, and itspromotejob pins to that per-commit image (resolving the release’s lerma build commit the same way) instead of the floating:latest-pre(falling back only for the first release after the tag is introduced). Docs:docs/DEVELOPMENT.md,docs/TESTING.md,apps/lerma/DEPLOY.md,apps/fouria/docs/WORKER-DEVELOPMENT.md. Both workflows validated withactionlint. -
Durable
doctor-repair/SQLite jobs start immediately, and their status streams over a WebSocket instead of 2s polling. Adoctor-repairjob enqueued at 12:57:30 only began at 13:02:11 — exactly the previous boot job’s 14-minute lease — because theActivitySchedulerowns one alarm and a stale arm (the alarm handler’s trailing re-arm, or a warm-pathrefresh) could read “no pending job”, then execute itssetAlarm(later)/deleteAlarm()after the just-enqueued job’ssetAlarm(now), postponing it.scheduleCombinedAlarmnow writes througharmWithGenerationGuard(activity/alarm.ts): it bumps a stored generation, and a superseded arm abandons its stale write so the newest arm (fresh read) owns the alarm. The dashboard no longer pollsGET /api/admin/config/fix/jobs/:idevery 2s (one request + Worker log line per tick for the whole multi-minute repair): a newGET /api/admin/config/fix/jobs/:id/ws(and the/debug/sqlite-integrity/jobs/:id/wsvariant) streams the job (routes/api/maintenance-ws.ts) — a frame only on status change, close1000when terminal, and arearm()self-heal while active. The client’ssubscribeMaintenanceJobis WS-first with a slow HTTP fallback only while the socket is down, andDiagnosticDashboardrenders a liveMaintenanceJobModal(queued/running/attempt/elapsed, run-in-background) instead of a bare spinner, with a banner to reopen it after backgrounding. Tests:src/activity/alarm.test.ts,src/activity/scheduler.test.ts,src/routes/api/maintenance-ws.test.ts,src/client/__tests__/maintenance-job-subscription.test.tsx,src/client/pages/__tests__/DiagnosticDashboard.test.tsx; corpustest/business-rules/dashboard-api-contract.txt; docsapps/fouria/docs/WORKER-DEVELOPMENT.md. -
Sandbox memory OOM mitigation (dev instance). A dev instance grew RSS past OpenClaw’s 1.5 GiB
rss_threshold(worker-thread V8 heaps + native arenas, not the main isolate) driven by cron-replayed heavyagentTurnjobs. (1)packages/moltlazyAGENT_DEFAULTS.compactionsetsenabled: true,maxActiveTranscriptBytes: '20mb',keepRecentTokens: 20000so a provider-side-managed session can no longer grow its persisted transcript unbounded (thefouria-builder:main~407k-token session inflates the SQLite history and worker heaps). (2) The dev-only Fouria Builder entry setsheartbeat.every: '12h'plusisolatedSession: true/lightContext: true— OpenClaw’s default 30m heartbeat is a full agent turn ~48×/day that also triggers memory-core dreaming reconciliation. (3)start-openclaw.shslows memory-core’s dreaming sweep to weekly (frequency: "0 3 * * 1", dev-gated) — the ~6-min daily sweep is replayed on every wake (cron: running missed jobs after restart) and produced the 1.6 → 2.08 GiB RSS spike. (4)ensure_opencode_credits_automationdedupes viaopenclaw cron list --all --jsonin addition to the legacyautomations list(which missed a prior boot’s job, creating two duplicateopencode-credits-checkcron jobs). Tests:packages/moltlazy/tests/agents.test.ts; docspackages/moltlazy/AGENTS.md,apps/fouria/docs/WORKER-DEVELOPMENT.md. -
Boot log streams gateway-launch progress (fixes “Launching the gateway — no progress for 6 min”).
start-openclaw.shnow starts astart_boot_log_streammirror inlaunch_gateway_processthat tails the gateway’s own--verbosestdout (/tmp/gateway-stdout.log) into the curated boot log (/tmp/startup.log) — so the loading page can follow SQLite migration → plugin load → port bind instead of going blank the momentstart/launchexecs the gateway.post-start/fullstop the mirror (stop_boot_log_stream) and truncate the buffer at readiness. The Worker’s/api/startup/wsstream now reads the boot log whileprocessType === 'gateway'(not just'script'), andloading.htmlshows the log panel during gateway startup — restyled as a vintage CRT terminal (title bar + traffic lights + phosphor text + scanlines) — instead of hiding it behind the logo. Tests:src/assets/loading.test.ts,src/routes/public.test.ts; docsapps/fouria/docs/WORKER-DEVELOPMENT.md. -
Loading page no longer gives up on its own.
loading.htmlhad two client-side budget specs — a 5-error retry cap (maxConsecutiveErrors,Retrying... (n/5)) and an 8-minute wall-clock deadline (MAX_WAIT_MS/deadline→runDoctorAndShow()). Since the cold boot moved to the durableActivitySchedulerboot job (which owns retries and terminal failure), the page’s self-imposed give-up only interrupted a legitimately long startup: it now waits indefinitely (WebSocket push, with an unlimited 5s HTTP-poll fallback) and reports terminal failure solely fromboot.status === 'failed'and permanentdegradedcontainer errors. Removed the now-deadrunDoctorAndShow/showStructuredError/showErrorhelpers. Tests:src/assets/loading.test.ts; docsapps/fouria/docs/WORKER-DEVELOPMENT.md. -
Health endpoints no longer require the ROOT role at the edge. A tenant ADMIN opening
/_adminwas 302’d to the Cloudflare Access login forGET /api/healthandGET /api/health/ws(the dashboard’s health stream), because the per-tenantFourIA Health - <slug>Access app on<slug>.fouria.io/api/healthallowed only thePaso4 Root Usersgroup plus lerma’s health-monitor service token. The health app (and the platform-wide*.fouria.io/api/health/*.dev.fouria.io/api/healthIaC apps) now carries no role requirement: anallowpolicy oneveryone(any authenticated user) plus the existingnon_identityservice token, so the dashboard and lerma’s headless probe both work.AccessPolicy.health_policies/1drops the root-group policy (the host app keeps it);access.tfpublic_healthmatches. Tests:test/lerma/cloudflare/access_policy_test.exs,apps/fouria/iac/tests/access_app.tftest.hcl; docsapps/lerma/README.md,apps/fouria/iac/README.md,apps/fouria/docs/WORKER-DEVELOPMENT.md,docs/ENV-VARIABLES.md. Existing tenants were reconciled in place via the Cloudflare API (aud preserved). -
Restore decrypt no longer exhausts the Worker CPU budget, and interrupted restores are labelled honestly. A large recovery archive (
tenant-tarjetero-dev) was killed withWorker exceeded CPU time limitmid-Decrypting backup …and retried in a loop.encryptStream/decryptStream(src/crypto/backup-encryption.ts) grew each 4 MiB chunk withconcaton every ~64 KB R2 read, recopying the pending chunk ~64× over; aByteQueuenow copies each byte once (96 MiB decrypt: 1.5 s → 0.14 s at 64 KB reads, 5.6 s → 0.14 s at 16 KB). The R2 binding delivers the archive in 4 KiB reads (64 MiB → 16,385 reads), which is why the per-read copy dominated; each restore now logs[crypto] Decrypt read profile(reads, bytes, max/avg read, chunks) so the pattern can be checked on a live tenant. The loading bar also showed the last durable phase (Waiting for the backup lock 3%) as if still live: a failed attempt is now labelledRestore attempt failed during "<phase>" — retrying(recordRestoreFailure), and an idle record gets— no progress for N min(describeRestoreProgressIdle); the fraction never moves backward. Tests:src/crypto/backup-encryption.test.ts,src/activity/restore-progress.test.ts,src/activity/scheduler.test.ts; docsdocs/WORKER-DEVELOPMENT.md. -
Recovery progress bar synchronized with the durable object’s events. During a restore the loading page showed a generic spinner; it now shows a progress bar driven by the
ActivitySchedulerDO’s own phase events, not a client-side timer.restoreIfNeeded/restoreBackupByIdcall theironProgress(phase)hook on everypersistence.phase; the DO wires it torecordRestoreProgress, which maps the phase to a label + a monotonic fraction (describeRestoreProgressinactivity/restore-progress.ts, cumulative phase weights) and exposes it viagetRestoreProgress()./api/startup/ws(and non-passive/api/health, plus/api/admin/boot-state) pushrestoreProgress: { phase, label, fraction };loading.htmlrenders the bar and hides the stepper while a restore is in flight, then switches back to the container’sstartup-progress.jsonstepper once the gateway launch begins (an indeterminate bar covers a request-scoped restore with no DO feed). Also declaresgetBootJob/getRestoreProgressonActivitySchedulerStub(fixing thepublic.tstype error). Tests:src/activity/restore-progress.test.ts,src/activity/scheduler.test.ts,src/routes/public.test.ts,src/routes/api/misc.test.ts,src/assets/loading.test.ts; corpustest/business-rules/operational-modes.txt; docsapps/fouria/docs/WORKER-DEVELOPMENT.md,apps/fouria/docs/OPERATOR-GUIDE.md. -
Restore legibility: name the backup + date being loaded (and mark the recovery point). A cold restore showed only a generic “Restoring data from backup…” spinner, so an operator could not tell which snapshot was loading. (1) Durable recovery state.
restoreIfNeeded/restoreBackupByIdnow writerestore-target.json(the archive being applied) just beforesandbox.restoreBackupand clear it on success;readRecoveryState(bucket)returns{ restoring, committed }(committed=stable-backup.json). It is read from R2, not the per-isolaterestoreStatus, because the durable boot/restore runs in theActivitySchedulerDO whose in-memory state the request isolate cannot see. (2) Surfaced on the loading page./api/health(non-passive, not-yet-ready) and/api/startup/wsattachrecovery(bounded byRECOVERY_STATE_TIMEOUT_MS, fail-soft to null; lerma’s passive poll never pays the read);loading.htmlrenders “Restoring backup— ” (or “Recovery point …”) and its isRestoringalso honoursrecovery.restoring. (3) Dashboard.BackupsPagethreadsstableBackupIdtoBackupHistoryPanel, which marks the recovery point with a recovery point badge + a “A cold boot restores— ” summary. Tests: src/persistence.test.ts(readRecoveryState, restore-target publish/clear),src/assets/loading.test.ts,src/client/components/__tests__/BackupHistoryPanel.test.tsx,src/routes/public.test.ts; corpustest/business-rules/operational-modes.txt; docsapps/fouria/docs/WORKER-DEVELOPMENT.md,apps/fouria/docs/OPERATOR-GUIDE.md. -
Manual-restore hardening: recovery-point flag, writer quiesce, and no restore loop. Restoring an older archive from the Backups page could not be applied without either making it permanent or being silently undone. Three fixes in
src/persistence.ts+ the storage route/durable job. (1)setCurrentflag.POST /api/admin/storage/restore/:id?setCurrent=0(also?async=1&setCurrent=0) passes{ setRecoveryPoint: false }torestoreBackupByIdso the archive is applied without force-promoting it tostable-backup.json(the default still promotes; the durableBackupJobRequest/BackupJobRecordcarry the flag so a retried job keeps the intent). The Backups page exposes it as a Set as recovery point checkbox (checked by default). (2) Stop writers on the lease.restoreBackupByIdnow opens a cross-isolate maintenance window (maintenance/replacement.json, operationrestore) and stops the gateway the moment it takes the shared restore lease (the route no longer pre-kills before the lock), closing the window in afinally; a live gateway/automation/channel therefore cannot keep writing whilesandbox.restoreBackupreplaces the writable layer. (3) No remount loop.restoreIfNeededrefuses to run while any maintenance window is active (andensureGatewayalready refuses to launch), so a health/proxy refetch cannot restart the gateway and re-run the restore against the committed recovery point — the restore → stop → refetch → remount-the-latest loop. Tests:src/persistence.test.ts(set-recovery-point, maintenance window, deferred restore),src/persistence-encryption.test.ts,src/routes/api/storage.test.ts,src/activity/scheduler.test.ts,src/activity/backup-jobs.test.ts,src/client/components/__tests__/BackupHistoryPanel.test.tsx; corpustest/business-rules/operational-modes.txt; docsapps/fouria/docs/WORKER-DEVELOPMENT.md,apps/fouria/docs/OPERATOR-GUIDE.md. -
Durable cold-boot job + slim health probe (observed on
tenant-client-dev:start-openclaw.shnever ran;phase=restore, age=686365ms; 141waitUntil() tasks did not complete;/api/health?passive=1at error level ×98). (1) Durable boot. The cold boot (R2 restore → gateway launch) was driven from a request’sctx.waitUntil(), which Cloudflare cancels shortly after the response — so a restore that outlived it was killed mid-flight (Decrypting backup …×14) andstart-openclaw.shwas never reached while every poll parked on the stuck start promise. It is now a durable job on theActivitySchedulerDO (src/activity/boot-job.ts):/api/warmup, non-passive/api/health, and/api/startup/wsonly request it (deduped) and return; the DO alarm performsrestoreIfNeeded→ensureGatewayand retries on a bounded lease. Falls back to the request-scoped boot whenACTIVITYis absent. (2)sandbox.restoreBackupis bounded byRESTORE_BACKUP_TIMEOUT_MS(a hung control channel fails fast instead of holding therestorephase for 11 min). (3) The restore reports sub-phase progress soisGatewayStartStuckjudgesrestoreonGATEWAY_RESTORE_NO_PROGRESS_MSof no progress (setGatewayStartProgress) — a slow-but-advancing restore is not killed. (4)withLeasereleases the shared R2 lease when its request aborts (PersistenceDiagnosticContext.signal), so a cancelled operation can’t leaverestore-needed+ a heldbackup-lockre-triggering the loop. Also:/api/healthnow issues one gateway/healthexec shared by readiness + liveness (was a duplicate curl pair per poll) and the passive path skips the container + R2 component probe (lerma’s frequent probe never readscomponents). Tests:src/activity/boot-job.test.ts,src/gateway/process.test.ts,src/gateway/health-probes.test.ts,src/routes/public.test.ts,src/persistence.test.ts; corpustest/business-rules/operational-modes.txt; docsapps/fouria/docs/WORKER-DEVELOPMENT.md. -
Slimmer
/api/health(frequent lerma probe). Two per-poll costs removed. (1) Readiness and liveness each ran the same gateway/healthcurl, so every poll spawned a pair of identicalcurlprocesses (visible as paired entries in/debug/processes);getHealthSnapshotnow fetches the process scan and the/healthprobe once, in parallel, and reuses both across the probes (newHealthProbeOptions.health). (2) The component probe (anisRuntimeActiveread + an R2 HEAD ofbackup-handle.json) was built on every/api/health, including lerma’s frequent?passive=1poll — which never readscomponents; the passive path now skipscomponentsForHealthentirely (the non-passive path and the dedicatedGET /api/health/componentsare unchanged). Tests:src/gateway/health-probes.test.ts(single shared exec),src/routes/public.test.ts(passive omits components); corpustest/business-rules/operational-modes.txt; docsapps/fouria/docs/WORKER-DEVELOPMENT.md. -
Restore config guard + recovery-point promotion (observed on
tenant-tarjetero-dev). After thev2026.10.13container rollout, an explicit restore of an older known-good backup did not stick: the redeploy changed the container runtime mid-restore, and a concurrent scheduled snapshot captured the partially-restored state and was auto-promoted to the stable recovery point, so the next cold boot restored the wrong archive and launched the gateway with no config. Three fixes: (1)start-openclaw.shonly honoursOPENCLAW_RESTORED=truewhen$CONFIG_FILEexists — otherwise it runs the setup body — so a “restored” boot with no materialized config (interrupted restore, orrestoreIfNeeded’s “container already holds tenant state” short-circuit) can no longer launch OpenClaw configless. (2)restoreBackupByIdnow force-promotes the restored archive tostable-backup.json(promoteStableBackup(bucket, entry, { force: true })) so an explicit restore becomes the recovery point even when it predates the current one; the default pointer still only advances forward in time. (3) the pre-backup config cleanup used.${CONFIG_DIR}withCONFIG_DIR='.openclaw', building/home/openclaw/..openclaw/…(a never-existent path, so the cleanup silently no-oped) — it now uses${CONFIG_DIR}. Tests:src/persistence.test.ts(force-promote; cleanup path); corpustest/business-rules/operational-modes.txt; docsapps/fouria/docs/WORKER-DEVELOPMENT.md,apps/fouria/docs/OPERATOR-GUIDE.md. -
Cold-start latency + boot legibility (a full non-restored boot on
staging-client-devtook ~14 min). Two causes, both fixed, plus a loading-page alert. (1) Non-owner killed the booting gateway. A non-owner isolate that observed thestart-openclaw.sh startwrapper waited a freshSTARTUP_TIMEOUT_MS(300s) from its own observation, but the wrapper’s real boot issetup (~290s) + gateway bootstrap (~45s) ≈ 335s— so the waiter timed out ~35s early andkill()ed the process just as it hadexeced into the gateway, releasing the startup lock and making the waiting second wrapper re-run the whole setup. Confirmed via Workers Observability:Startup script present but gateway never became reachable … killing and restarting= 1 while the warm-probe start/catch = 0/0 (ruling out the 30s warm probe). Fixed withSTARTUP_WRAPPER_STUCK_BUDGET_MS(STARTUP_TIMEOUT_MS + GATEWAY_START_STUCK_GRACE_MS): the non-owner’s stuck check and readiness wait are measured from the wrapper’sstartedAt(mirroring the owner’s watchdog) and a/healthre-probe runs immediately before any kill. Tests:src/gateway/process.test.ts,src/gateway/process-extra.test.ts. (2) Plugin detection read the hanging registry.plugin_is_installedranopenclaw plugins list --json, which hangs for its fullCLI_TIMEOUTand reads as “not installed” on a hung/partial list, so the pre-installed guard fell through and triggered a needless network install (ai-gateway ~57s + otel ~74s). Replaced withplugin_extension_present(theextensions/<id>dir the gateway actually loads). (3) Bootup-UI alert. The loading page now shows a non-fatal slow-step notice (step name + elapsed seconds) once a step overrunsSLOW_STEP_MS(120s), so a slow-but-progressing boot is distinguishable from a stuck one without opening logs. Tests:src/assets/loading.test.ts. Corpustest/business-rules/operational-modes.txt; docsapps/fouria/docs/WORKER-DEVELOPMENT.md,docs/plans/cold-start-latency.md. -
The tenant Usage & Billing widget no longer 403s on Analytics Engine.
GET /api/admin/costs/v2/summaryresolved its AE SQL token asANALYTICS_ENGINE_API_TOKEN ?? CLOUDFLARE_API_TOKEN, but Lerma never provisioned the dedicated token, so the query fell back toCLOUDFLARE_API_TOKEN— the AI Gateway credential alias, which lacks Account Analytics:Read — and logged[cost-summary] Analytics Engine query failed with HTTP 403on every load (observed ontenant-client-dev), zeroing the widget. The token is now a Lerma platform secret (tenant_provisioner.ex@platform_secrets), read from Lerma’s environment and deployed per-tenant as asecret_textbinding; the deploy path wires it throughmix lerma.deploy, Ansibleapp-config,docker-compose.prod.yml/.pre.yml, anddeploy-dashboard.yml, and the secrets manifest. Optional: when unset the worker still degrades to an all-zero summary. Tests:tenant_provisioner_pipeline_test.exs,tenant_provisioner_integration_test.exs; docsdocs/ENV-VARIABLES.md,apps/lerma/README.md,apps/fouria/docs/cost-observability.md,SECRETS.example.yaml,docs/schemas/secrets-manifest.schema.json. -
Zero-data-loss: a restored container skips the setup body. The R2 restore runs in the Worker (
restoreIfNeeded→sandbox.restoreBackup()) beforestart-openclaw.sh; when that restore actually restored a backup,ensureGatewaynow passesOPENCLAW_RESTORED=truein the launch env (gated ongetRestoreStatus().hadBackup && restored, so a brand-new tenant with no backup still onboards), and the script skips the whole setup body — onboard → plugin installs →moltlazy patch→ config validate — emitting those phases as skipped and launching the gateway directly from the restored filesystem. The restored state is authoritative: the cold-start budget is spent launching it, not re-mutating it (which could diverge from or clobber the snapshot). Each boot rewrites a durable receipt at/tmp/openclaw-restored-setup-skipped(written on the skip path, removed on the setup path). Tests:src/gateway/process.test.ts; E2Etest/e2e-backup/zzz_backup_restore_skip_setup.txt(opt-inE2E_BACKUP_RESTORE=1; sync → destroy → restore, asserts the receipt + unchanged restored setup state); corpustest/business-rules/operational-modes.txt; docsapps/fouria/docs/WORKER-DEVELOPMENT.md,test/e2e-backup/README.md,test/e2e/BACKUP-E2E-DESIGN.md. -
The dashboard Usage & Billing widget now reflects emitted billable actions.
GET /api/admin/costs/v2/summarywas a stub returningtotalCostUsd: 0; it now aggregates the tenant’sbillable-actionAnalytics Engine rows via the AE SQL API (src/cost-summary.ts→getCostSummary()), returningtotalCostUsd,totalCredits,actionCount,byTriggerType, andbyCostCategory. Config mirrors Lerma:ANALYTICS_DATASET(defaultfourai_cost_metrics),ANALYTICS_ENGINE_BASE_URL, andANALYTICS_ENGINE_API_TOKEN(falls back toCLOUDFLARE_API_TOKEN; needs Account Analytics:Read). A missing credential/transport error/non-200 degrades to an all-zero summary, so the widget never breaks. The widget surfaces the billable action/credit rollup when data is present. Tests:src/cost-summary.test.ts,src/routes/api/costs.test.ts,src/client/pages/__tests__/HomePage.test.tsx,src/client/pages/__tests__/AccountPage.test.tsx,test/ui/billing-widget.spec.ts; corpustest/business-rules/dashboard-api-contract.txt; docsapps/fouria/docs/cost-observability.md,docs/ENV-VARIABLES.md,docs/TESTING.md. -
The version surfaces now report the Worker’s own build identity.
GET /api/admin/versionandGET /debug/versionreturnfouria_version(component CalVer fromapps/fouria/package.json) andfouria_commit(short git SHA) alongside the container’s OpenClaw/Node versions; the DiagnosticDashboard Version Info tool surfaces them. Both values are injected at build time byvite.config.ts(__FOURIA_VERSION__/__FOURIA_COMMIT__, consumed viasrc/build-info.ts;GITHUB_SHAwins in CI,git rev-parse --short HEADlocally). Tests:src/routes/debug.test.ts,src/routes/api/misc.test.ts; docsdocs/WORKER-DEVELOPMENT.md,docs/OPERATOR-GUIDE.md. -
Expected CLI non-zero exits are no longer logged as failures.
waitForProcess(likerunShell) now takesreportNonZeroExit; the knowledgewiki statusprobe sets itfalse(operationknowledge.wiki-status) becauseopenclaw wiki statusexits non-zero for an uninitialized vault — which/api/admin/knowledge/treehandles as a normalinitialized: falseresult — yet every load emittedError [SandboxProcessError]: sandbox.waitForProcess failed (non-zero-exit, exit=1). Timeouts still log. Tests:src/gateway/utils.test.ts; corpustest/business-rules/operational-modes.txt; docsdocs/WORKER-DEVELOPMENT.md. -
The prelaunch gateway port probe no longer logs a false failure.
isGatewayPortOpenrancurl -sf …/healththroughrunShell, whoselogCommandFailureemitted a structuredsandbox.process.failederror for the expected closed-port exit (curl7 = connection refused). Every cold start therefore surfaced a scaryError [SandboxProcessError]: shell failed (non-zero-exit, exit=7)for the normal “gateway down, start it” outcome. It now passesreportNonZeroExit: falsewithoperation: 'gateway.port-probe'(matching the existinggateway.health-probe), so a closed port is quiet while a probe timeout (a hung gateway) still logs. Test:src/gateway/process.test.ts(does not log a structured failure for the expected closed-port exit); corpustest/business-rules/operational-modes.txt; docsdocs/WORKER-DEVELOPMENT.md. -
OpenClaw is installed globally with bun, not npm.
bun install --global --trust openclawruns withBUN_INSTALL=/usr/local: bun’s default global prefix is$HOME/.bun/bin, which is not onPATHin this image, soBUN_INSTALL=/usr/localputs the binary at/usr/local/bin/openclaw(onPATH) and the package under/usr/local/install/global/node_modules/openclaw.bun pm cache rmresolves the current project and hard-fails without apackage.json— the base image’sWORKDIR /container-serverhas none — so it runs from/tmp(which holds the copied repopackage.json). The dead-weight trim now targets bun’s hoisted global layout (<prefix>/install/global/node_modules), andstart-openclaw.shresolves the package by following the installed binary (command -v openclaw+readlink -f) instead ofnpm root -g, so the runtime-version log stays accurate. -
post-startno longer blocks readiness. After the port wait succeeds the gateway is already serving/health200, butensureGatewayImplstillawaitedstart-openclaw.sh post-start(dev automation + boot-log reset, bounded byGATEWAY_POST_START_TIMEOUT_MS= 60s). On dev instances that phase runsopenclaw automations list/create(30–60s), so every request loggedGateway start already in progress … (phase=post-start)and waited on the shared start promise even though the gateway was up. It is now dispatched detached (void runGatewayPostStart(...), not awaited) right aftermarkGatewayReady;startShelldispatches thesandbox.execsynchronously before its firstawait, so the work still runs server-side if the isolate is evicted, andrunGatewayPostStartswallows its own errors. Thepost-startphase is no longer set on the start tracker (the start is complete). Test:src/gateway/process.test.ts(does not extend the shared start promise with the detached post-start phase); docsdocs/WORKER-DEVELOPMENT.md. -
ClawHub plugin pre-install is best-effort so the image still builds. The unpinned
clawhub:@openclaw/cloudflare-ai-gateway-providerresolved to2026.9.9, which requires a newer plugin API than the pinned2026.9.8runtime, soopenclaw plugins installfailed and aborted the whole Docker build (Plugin … requires plugin API >=2026.9.9, but this OpenClaw runtime exposes 2026.9.8). The Dockerfile now wraps each pre-install in a warn-and-continue guard, andstart-openclaw.sh’s single enablementconfig patchonly enables a plugin whose extension directory is present (CF_AI_GATEWAY_PLUGIN_PRESENT/UNIFIED_BILLING_PLUGIN_PRESENT), so a failed install can neither fail the build nor invalidate the patch. Root-cause follow-up: bumpcatalog.openclawto the plugin-compatible release (openclaw-migration skill) or pin the specs; until then a build on the pinned runtime ships without the Cloudflare provider plugins. Corpustest/business-rules/operational-modes.txt; docsdocs/WORKER-DEVELOPMENT.md,docs/OPENCLAW-PLUGINS.md. -
Gateway start watchdog + process-count hygiene. A cold start is one shared
ensureGatewaypromise (restore → supervisedstart-openclaw.sh start→ port wait); a wrapper that neverexecs the gateway, or a control-channel drop, left it “in progress” for ~20 min while every request loggedGateway start already in progress … waiting...and waited on it (observed ontenant-client-dev: 64 processes, port closed, curl health probes exiting 7 every ~5s,[PROXY] Failed to start gatewaystack-only). The start is now tracked per sandbox key with a phase (probe→restore→launch→port-wait;post-startruns detached after readiness, not tracked) and a per-phase budget;isGatewayStartStucktrips when a phase overruns its budget +GATEWAY_START_STUCK_GRACE_MS. Before resettingport-wait,recoverStuckGatewayStartrechecks/health: a late-positive response emitsgateway.start.late-readyand is adopted without killing, while a failed/unknown recheck emitsgateway.start.health-recheck-failedand defers reset. Confirmed stuck starts emitgateway.start.stuck, kill the process (killGateway), clear startup/readiness caches, and reprovision afterGATEWAY_START_STUCK_MAX_ATTEMPTSresets (recoverStaleSandbox).findExistingGatewayProcessalso uses the pureclassifyGatewayStartto kill astart-openclaw.sh startwrapper older thanSTARTUP_TIMEOUT_MSwith the port closed, instead of waiting out another readiness budget. BecauselistProcesses()includesexited/errorentries the Worker never GCs (the SDK exposes no cleanup), the raw count is append-only and must not gate health: the log now readslistProcesses returned <total> processes (<running> running)and every decision considers onlyrunningentries. Emitsgateway.start.begin/phase/stuck/late-ready/health-recheck-failed. Tests:src/gateway/classify.test.ts,src/gateway/process.test.ts; corpustest/business-rules/operational-modes.txt; docsdocs/WORKER-DEVELOPMENT.md,docs/OPERATOR-GUIDE.md. -
Legacy bridge excludes the image-provided shared OpenClaw package.
extensions/<id>/node_modules/openclawis a symlink to the image’s/usr/local/lib/node_modules/openclaw; archiving it is pointless and (when an old dereferencing stage expanded it into a full package copy) made the restore materialization a multi-minute copy that blocked the gateway launch, leftrestore-neededset, and re-ran the restore on every boot (observed onstaging-client-dev:cp -a … /root/openclaw/.openclaw/running 8m45s, 65 Worker probe processes, nostart-openclaw.sh/gateway). BothstageLegacyRootsForBackupandmaterializeLegacyRootsAfterRestorenow use a symlink-preservingtarpipe with--exclude='*/node_modules/openclaw'(a copy-then-rm -rfis equally slow — deleting a squashfs-lower tree requires copy-up), so old and new archives restore fast.restoreIfNeededalso deletes therestore-neededmarker immediately after a successful materialization (before the analytics bookkeeping) so an isolate eviction can no longer leave a stale marker. Tests:src/persistence.test.ts; corpustest/business-rules/operational-modes.txt. -
Legacy restore bridge preserves symlinks (backup/restore fix). The legacy state bridge staged
/root/openclaw/.openclawwithcp -aL, which dereferenced the pluginextensions/*/node_modules/openclawsymlinks (→/usr/local/lib/node_modules/openclaw) into full OpenClaw package copies. That bloated every archive (the package three times over), timed out the pre-snapshot stage (600s) — surfacing to the dashboard as a backup-createInternal Server Error— and made restore fail becausecpcannot overwrite the image’s symlink with a directory (cp: cannot overwrite non-directory … with directory), leaving the tenant looping on a failed restore (staging-client-dev). The stage and the restore materialization now usecp -a(preserve symlinks); the restore additionally removes any destination symlink the stage replaces with a real directory first, so backups created by the old dereferencing stage still restore. Tests:src/persistence.test.ts; corpustest/business-rules/operational-modes.txt. Also: the admin API client (src/client/api.ts) now reads the response body once and surfaces a plain-text platform error (e.g.Internal Server Error) instead of throwingUnexpected token 'I', "Internal S"... is not valid JSON; testssrc/client/api.test.ts. -
Recurring-error hardening: credential redaction, bounded-RPC cleanup, live-gateway logs, and idempotent WS teardown. Investigation of the recurring
ReadableStream received over RPC disconnected prematurely/[WS] Client errortelemetry found four concrete causes. (1) Credential logging: the catch-all proxy logged the full forwarded header map (Object.fromEntries(httpHeaders.entries())), writing CF Access JWT/assertions, cookies, and the tenant metadata blob to Workers Logs; it now logs only an allowlisted subset (safeHeaderLog,src/utils/logging.ts). (2) Late container RPCs: a boundedcontainerFetch/wsConnectwhose timeout won could still settle later with a live body/socket that was never drained/closed;withTimeoutAndCleanup(src/utils/timeout.ts) pluscontainerFetchWithTimeout/wsConnectWithTimeout(src/utils/container-rpc.ts) cancel the late body / close the late socket, applied to the public proxy, diagnostics, and thegateway-apiprobes. (3) Live gateway logs: a running gateway’s SDK log stream never settles, so reading it opened an RPC stream that was later disconnected; a runningGatewayProcess.getLogs()now tails/tmp/gateway-stdout.log, and/debug/processes?logs=true//api/admin/logs?id=refuse a live stream for a still-running process. (4) WS teardown: both peer close handlers recorded the same session metric twice, and a client tab disconnect was logged aterror; a singlesettleSession()now records the metric once and client disconnects log atwarnwith thesessionId. Tests:src/utils/container-rpc.test.ts,src/utils/timeout.test.ts,src/logging.test.ts,src/index.test.ts,src/gateway/process.test.ts,src/routes/api/misc.test.ts,src/routes/debug.test.ts; local observability canarytest/e2e/ad_observability_canary.txt+test/e2e/fixture/observability-query; corpustest/business-rules/operational-modes.txt; docsdocs/TESTING.md. -
Two local-E2E self-skip fixes.
provision_and_dispatch’s first scenario (%require) echoed aSKIP:line andexit 1in local mode, so it failed instead of skipping — and because it is%require, that failure skipped every later scenario in the file. It now emits the samePASS: … (skipped - local E2E mode)line andexit 0the other local skips in the file use.p_otel_billing_queueassertedemitted >= 1, but a dev-environment/staging-platform instance is non-billable by design (resolveInstanceDimensions,src/routes/otel.ts): the receiver accepts + ignores approved spans (200) and returnsbillable: falsewithout emitting. The local worker (DEV_MODE=true, no routableWORKER_URL) is always non-billable, so the test now skips onbillable == false(like its existing missing-secret/401skip); the emit path stays asserted by the cloud profile andsrc/routes/otel-traces.test.ts. Corpus:test/business-rules/operational-modes.txt,test/business-rules/dev-expense.txt; docsdocs/TESTING.md. -
Local E2E readiness gate accepts a reachable-but-starting Worker.
fixture/server/start-localwaited for/api/healthwithcurl -sf -m 5(60×2s), but a cold boot legitimately answers 503 while the container/gateway starts (the non-waking-health semantics) and the non-passive probe can block on container provisioning — so the gate misread a live-but-starting Worker as dead. Worse, the non-passive probe on every 2s poll re-triggeredrestoreIfNeeded/ensureGateway, launching overlapping R2 restores that abort each other withOperationInterruptedError: ... runtime changed. The gate now probes the fast, non-wakingGET /api/health?passive=1and accepts any HTTP status (200/503/…) via the sharede2e_worker_reachablehelper (test/lib/e2e-helpers.sh), failing only on a connection failure (status000); the window is overridable withE2E_STARTUP_TIMEOUT_SECS(default 120s) and the “local dev server died” check still fails fast. The real gateway wait (startup.ok+readiness.ok) stays in the_setup.txtreadiness step. Tests:src/e2e-helpers.test.ts; corpustest/business-rules/operational-modes.txt; docsdocs/TESTING.md. -
Restore failures are logged as structured
persistence.restore.failedevents withbackupId,errorName, anderrorMessage; manual route failures also include the Cloudflare Ray ID asstorage.restore.failed. This keeps the backup identity searchable and correlates async failures to the originating request. Tests:src/persistence.test.ts,src/routes/api/storage.test.ts. -
Backup retention is enforced on catalog reads, and restores survive a failed pre-restore
umount. Livefouria-backup-tarjetero-devheld 7 archives undermaxVersions: 3: every snapshot was killed aftersandbox.createBackup()(260 MB) but before its commit, so a later read adopted all 7 as orphans, and every scheduled snapshot after that failed at the legacy stage before its own prune — the cap was only ever applied on the happy snapshot path.reconcileOrphanedBackupsnow runscleanupOldBackupsafter merging, so the invariant holds on every read (getBackupHistoryincluded) and no write/delete happens when the catalog is already within the cap. Restores failed in a loop on[persistence] Restore of … failed: … runShell … restoreIfNeeded: the best-effortumount(|| trueguards the shell, not the exec RPC) threw a cold-start/platform interruption and aborted the restore beforerestoreBackup, keepingrestore-neededand failing the startup barrier. The umount is now caught + logged inrestoreIfNeededandrestoreBackupById; an empty catalog read no longer deletesrestore-needed(a transient reconcilelistfailure silently dropped an operator restore) — the marker clears only after a successful restore. Tests:src/persistence.test.ts(over-cap prune, adopted-orphan prune, no-delete within cap, marker retention, umount-failure restore); corpustest/business-rules/operational-modes.txt; docsdocs/WORKER-DEVELOPMENT.md,docs/OPERATOR-GUIDE.md. -
Async admin backup/restore + non-blocking Backups page. A manual snapshot/restore can wait up to the 120s R2 lease and then transfer a large archive; holding the admin HTTP request open surfaced as a bare
500after ~132s (restore) with the error dropped.POST /api/admin/storage/sync?async=1andPOST /api/admin/storage/restore/:id?async=1now acknowledge202 { accepted: true }and run inexecutionCtx.waitUntil(default stays synchronous for the E2E checkpoint).BackupsPagepaints every section independently and reads the fastGET /api/admin/config/rawinstead of the slow structured/api/admin/config(which awaitsensureGateway+ up to two 30s shell waits) — so no single request blocks the page.BackupHistoryPanelfires the async endpoints, pushes a “started” bell notification, then pollsGET /storage/backups(snapshot) //api/health(restore) and pushes success/failure viaNotificationProvider(survives navigation). Restore failures are reported as structuredstorage.restore.failedevents. Tests:src/routes/api/storage.test.ts,src/client/components/__tests__/BackupHistoryPanel.test.tsx,src/client/pages/__tests__/BackupsPage.test.tsx; corpustest/business-rules/operational-modes.txt; docsdocs/WORKER-DEVELOPMENT.md. -
Startup-stream and dashboard polling back-off.
/api/startup/wsnow drops the 2-exec startup-progress read and backs off toSTARTUP_WS_READY_POLL_INTERVAL_MS(30s) once the gateway is ready — the 5s cadence plus a per-pushlistProcesses/exec probe burst is a large share of the Sandbox DO control-channel churn (Establishing RPC connection) during a boot.useGatewayHealthnow stops its HTTP fallback polling when a/api/health/wssnapshot arrives (it previously kept double-polling after the socket recovered).transformErrorMessagemaps the gateway’s1008 device token mismatchclose to a “re-open/_admin/to re-approve this browser” message instead of a reconnect loop. Tests:src/client/hooks/useGatewayHealth.test.tsx; corpustest/business-rules/operational-modes.txt. -
Backup local-bucket mode is decoupled from
DEV_MODE.isLocalBucketDevMode()keyedcreateBackup({ localBucket })onprocess.env.DEV_MODE, and withnodejs_compat_populate_process_envthe cloud E2E’sDEV_MODE=truevar (set only to bypass Access on a bareworkers.devhost) made every E2E snapshot land in the Sandbox local binding bucket — invisible to the Worker’sBACKUP_BUCKETHEAD, so restore loggedBackup <id> missing from R2and booted without tenant state. It is nowisLocalBucketBackupMode(env): an explicitBACKUP_LOCAL_BUCKET=trueopt-in that falls back toDEV_MODEonly when unset; the E2E deploy pinsBACKUP_LOCAL_BUCKET=false.resolveRestoreEntrythreadsenvthrough. Tests:src/persistence.test.ts(explicit override wins); corpustest/business-rules/operational-modes.txt; docsdocs/ENV-VARIABLES.md,docs/OPERATOR-GUIDE.md,docs/WORKER-DEVELOPMENT.md,.dev.vars.example. -
A dropped port probe no longer spawns a duplicate gateway. When the DO↔container control channel flapped (
Network connection lost.on everyexec),ensureGatewayImpl’s prelaunchisGatewayPortOpenResilientthrow was treated as “gateway absent”, so it re-ranstart-openclaw.sh starton each request even thoughlistProcessesmatched the gatewayrunning— piling lock contention and RPC load onto the unhealthy channel. A transport-classified probe failure is now “unknown”: the Worker logsPort probe transport drop …and returns without spawning, retrying on the next request. Test:src/gateway/process.test.ts(ensureGateway transport-drop resilience); corpustest/business-rules/operational-modes.txt. -
E2E
ac_gateway_startup_concurrencyaccepts the 503 cold-boot health response. The concurrent non-passive/api/healthfan-out usedcurl -f, which fails on any non-2xx, but/api/healthanswers 503 whilestarting(JSON snapshot present) since the non-waking-health change — so a healthy cold boot always reportedprobes_ok=0. The probes now accept200/503with a shaped{startup, readiness}body; the ready-wait and single-owner assertions are unchanged. Docs/corpus updated. -
Health is not a routing gate; ROOT operators can always reach a tenant. Tenant containers are loaded lazily by the User Worker, so a failed health probe is not an outage — yet the dispatch route gate treated
clients.status/instances.status = erroras non-operational (and lerma’sHealthMonitor/HealthPollerwrote thaterror), which locked operators out of a reachable tenant (observed ontenant-tarjetero-dev).workers/fouria-dispatchtenantRouteGatenow never blocks on a health-derivederror(production client or dev instance), keepsfrozen/stoppedgated only for non-ROOT callers (a ROOT operator is exempt so they can revive a stopped tenant), and runs after authentication so the exemption can be applied.apps/lermaHealthMonitorandTenantProvisioner.verify_healthare now observability-only on failure: they record the down transition for the instance uptime widget but never change instance/client status (a successful probe still marks the instancerunningand recovers a client left inerrorby a provisioning failure). The now-meaninglessHEALTH_MONITOR_FAILURE_THRESHOLDenv var is removed. Tests:workers/fouria-dispatch/test/integration/resolution.test.ts,apps/lerma/test/lerma/provisioning/health_monitor_test.exs,tenant_provisioner_pipeline_test.exs; corpustest/business-rules/operational-modes.txt; docsworkers/fouria-dispatch/README.md,docs/CLOUD-ARCHITECTURE.md,docs/ENV-VARIABLES.md. -
Sandbox→R2 bridges resolve the tenant from the per-tenant Worker env, not the fixed sandbox id. The
leads.r2.internalanddocs.r2.internalbridges (src/sandbox-leads-bridge.ts,src/sandbox-docs-bridge.ts) scoped every path to the caller’s<slug>/usingctx.containerId, but production provisions the sandbox under the fixed idopenclaw(getSandbox(..., 'openclaw')insrc/index.ts,src/tasks/index.ts,src/activity/scheduler.ts), so listings resolved toopenclaw/and always returned empty. AddedresolveTenantSlug/tenantSlugFromEnvinsrc/sandbox-tenant.ts: an explicitcustomer-<slug>container id still wins (legacy multi-tenant), otherwise the slug is derived from the per-tenant Worker env —CUSTOMER_SLUG,WORKER_NAMEviaderiveCustomerSlug, or the always-provisionedWORKER_URLhostname (stripstenant-/staging-prefixes and the-devsuffix, sotenant-tarjetero-dev/https://tarjetero-dev.fouria.io→tarjetero). Tests:src/sandbox-tenant.test.ts,src/sandbox-leads-bridge.test.ts,src/sandbox-docs-bridge.test.ts; corpustest/business-rules/tenant-isolation.txt; docsdocs/WORKER-DEVELOPMENT.md. -
Dev-agent personas source the client proposal via the
fouria-leadsskill. The Fouria Builder and Q&A personas (packages/moltlazy/workspace/fouria-builder.ts+fouria-qa.ts) now read the skill first and fetch the proposal through the sandboxleads.r2.internalbridge (globalpaso4-leadsR2 bucket) instead of the legacy/home/openclaw/clawd/client-docs/path; the Worker’s dev-only docs materialization (materializeClientDocsIfDev) is no longer the personas’ input path. Docs updated alongside (packages/moltlazy/src/agents/README.md,packages/moltlazy/AGENTS.md,docs/DEVELOPMENT.md). -
Crash-consistent backup cataloging (issue #344). The scheduled
backupcontrol action runs the full snapshot orchestration inside one tenant request and could exceed the Worker CPU budget aftersandbox.createBackup()had uploadedbackups/<id>/…but before the handle was written — leaving an orphaned snapshot, a missingbackup-handle.json(and emptyconfigs/), a livebackup-lock, and a dispatch[cron] Failed to trigger backup.createSnapshotUnlockednow commits the catalog entry immediately after the archive is created (after encryption, before the config-version capture and retention prune), andreconcileOrphanedBackupsrebuilds entries frombackups/*/meta.jsonon read (getBackupHistory,restoreIfNeeded,scheduleBackupIfNeeded) so pre-existing orphans self-heal and stay restorable. The Dispatch Worker gives thebackupaction a dedicated CPU budget (BACKUP_CONTROL_CPU_MS) and logs the underlying error on a thrown control fetch. Tests:src/persistence.test.ts(commit ordering, config-save failure, reconciliation),workers/fouria-dispatch/test/unit/control.test.ts,cron-fanout.test.ts; corpustest/business-rules/operational-modes.txt; docsdocs/OPERATOR-GUIDE.md,docs/WORKER-DEVELOPMENT.md. -
Transport-resilient backup stage (issue #344 follow-up). The pre-snapshot legacy-stage shell now waits via
runShellResilient(→waitForExitResilient): a transient Sandbox DO↔container transport drop mid-stage reattaches to the process by id and resumes within the stage deadline, instead of ending the/controlbackup request (observed onstaging-client-dev: the stage ran ~5 min then the request ended with aNetwork connection lost.drop and noCreating backup...). The DispatchbackupCPU budget (BACKUP_CONTROL_CPU_MS) is raised 300s → 600s to match the 10-minute stage bound. Tests:src/persistence.test.ts(transport-drop retry); corpustest/business-rules/operational-modes.txt; docsdocs/OPERATOR-GUIDE.md,docs/WORKER-DEVELOPMENT.md. -
Legacy-stage backup robustness + diagnosability (issue #344 follow-up). The scheduled
backupstage shell (stageLegacyRootsForBackup→runLegacyStageShell) copies/root/openclaw/.openclaw+/root/clawdwithcp -aLand runs anode:sqliteonline backup; on a large tenant it exceeded the old 300 s bound and killed the backup (tenant-tarjetero, observed at exactly 300 s).LEGACY_STAGE_TIMEOUT_MSis now env-overridable (default600000/ 10 min), the idempotent stage is retried once on a transient timeout/transport failure, and stage failures logexit/timedOut/stderr plus a structuredsandbox.process.failedWorker event withprocessIdand a stable operation label. The process id joins Worker Logs to the Containers dataset without exposing the raw command/output. Tests:src/persistence.test.ts(retry, no-retry, timeout override, error logging); corpustest/business-rules/operational-modes.txt; docsdocs/OPERATOR-GUIDE.md,docs/WORKER-DEVELOPMENT.md,docs/ENV-VARIABLES.md. -
Sleeping tenants + non-waking health.
/api/healthnow answers 503 unless the gateway is ready, and a passive probe (?passive=1) short-circuits on the non-wakingisRuntimeActive()read and reportsstatus: "sleeping"(HTTP 200, healthy-idle) for an idle container instead of running the exec RPCs that auto-start it — so Lerma’sHealthMonitor(which now probes?passive=1) never wakes a tenant. Non-billable tenants (staging 15m, dev 2h) sleep per a derived idle timeout (the Sandbox SDK 1.0setInactivityTimeout()platform cap; there is noSANDBOX_SLEEP_AFTERbinding; the value comes from the instance hostname viasrc/dev-mode.ts), and theActivitySchedulerDO alarm wakes/heals ahead of the next automation and runs an interval-gated R2 checkpoint after the wake. Billable production (platform === 'production' && environment === 'production') does not auto-sleep; the container stays awake and is stopped only by an explicit lerma freeze (POST /control→sandbox.destroy()), so a warmed gateway never incurs a cold-start/downtime window after an idle period. Runtime cutover is gated bySANDBOX_RUNTIME(legacydefault |v1);SandboxV1is still scaffold-only, soresolveSandboxRuntime()reportsv1aspendingand the legacy class keeps serving (src/sandbox-runtime.ts). Tests:src/routes/public.test.ts,src/gateway/health-probes.test.ts,src/activity/alarm.test.ts,src/sandbox-runtime.test.ts,src/sandbox-do.test.ts,src/build-sandbox-options.test.ts; corpustest/business-rules/operational-modes.txt; lermahealth_monitor_test.exs,health_poller_test.exs,tenant_provisioner_pipeline_test.exs. See docs/WORKER-DEVELOPMENT.md. -
Single-owner gateway launch (
start-openclaw.sh start): the production cold-start mode acquires the container-wide startup lock before config setup and holds fd 9 through the Gatewayexec, so one owner covers setup and launch. The lock is inherited by the gateway process and released exactly when it exits, so only one owner can run — the OpenClaw 2026.9.8 state-DB single-owner migration otherwise caused concurrent launches (from different Worker isolates) to lose the migration, exit before binding the port, and leave the boot forever unhealthy (state.sqlite is undergoing offline maintenance). Losers poll the local/health(bounded) and exit 0 without a duplicate launch, or take over if the owner dies and releases the lock; the log-mirrortail -fcloses fd 9 (9>&-) so it cannot hold the lock. Thelaunchmode remains a standalone smoke/manual entry point with its dedicated/tmp/openclaw-gateway-launch.lock;fullroutes through the same lock inbackgroundmode. Regression: the serialtest/e2e/ac_gateway_startup_concurrency.txt(phaseD_startup) asserts exactly one owner and the absence of both the migration-input-change error and the SQLite offline-maintenance error. See docs/WORKER-DEVELOPMENT.md. -
Fixed the develop cold-boot stall introduced by #318 (
startup.ok=false / readiness.ok=falsefor the whole E2E readiness budget; 68 processes,start-openclaw.sh setupexited 0,curl /healthalways exit 7, noopenclaw gateway runever present). Root cause: #318 had split the boot so the Worker launched the gateway afterstart-openclaw.sh setupexited (supervised exec) — this was the former behavior; current production collapses setup and launch into the atomicstart-openclaw.sh startdescribed above (startup lock acquired before setup, held through the Gatewayexec)./api/healthis bounded to a 5s snapshot, so the request returned long before the ~24s setup finished; when the response was sent, workerd cancelled the unregisteredensureGatewaycontinuation, so the container-side setup completed but the Worker never resumed to launch the gateway. Fix:/api/health(and/api/warmup) now register the in-flight boot withc.executionCtx.waitUntil(ensureGateway(sandbox, env, { waitForReady: true }))so the setup → gateway-launch sequence survives past the response;waitForReady: truereturns the shared start promise unwrapped and dedupes onto the one the snapshot already started. Also hardened the RPC-first health probe (getRpcHealthSnapshot,src/gateway/health-probes.ts) to run only once a gateway process is detected —sandbox.containerFetchauto-starts the container and blocks on the closed gateway port during a cold boot, piling port-waits on the Sandbox DO control channel.buildHealthSnapshot(src/routes/public.ts) passes its already-fetchedfindGatewayProcessresult as{ proc }. Bisected: production E2E boot green atad9ad4a5(#317), red at9ae5f66c(#318). Seeapps/fouria/docs/WORKER-DEVELOPMENT.md§Admin RPC. -
Startup resilience to Sandbox DO code updates: a Worker/DO redeploy (e.g. a Lerma redeploy of a tenant script) resets the DO↔container control session while the container keeps running, so the supervised
waitForPorton the atomicstart-openclaw.sh startprocess rejected withNetwork connection lost.and the cold start was aborted — killing a gateway that would have completed. The atomicstartpath has no separate setup-process wait (one supervised process spans config setup through the gateway’s whole life), so only the port wait is transport-resilient:ensureGatewaywaits withwaitForPortResilient(src/gateway/sandbox-exec.ts), which reattaches to the process by id viagetProcess(id)and resumes the wait within the sharedSTARTUP_TIMEOUT_MSdeadline; only a non-transport error, an un-reattachable process, or an exhausted deadline fails. The genericwaitForExitResilienthelper remains insandbox-exec.tsfor the legacy/boot-smoke supervised waits. The loading page’sGET /api/startup/wsstream also treats a transport-drop snapshot as transient: it re-sends the last known state (or a syntheticstartingview) and backs off exponentially (STARTUP_WS_POLL_INTERVAL_MS→STARTUP_WS_MAX_BACKOFF_MS, 30s) instead of re-probing and re-logging the same error every 5s. -
Backups run only on running instances; health is observability-only.
GET /api/healthno longer schedules snapshots (passive or active) — the lermaHealthMonitoris a pure probe and MUST NOT alter the R2 catalog. The Dispatch Worker cron (workers/fouria-dispatch, 15 min prod / hourly staging) fans abackupcontrol action (POST /control,X-FourIA-Control: backup) out to every non-frozen tenant; the tenant assertssandbox.isRuntimeActive()(a non-waking bridge read) and skips a sleeping container rather than waking it (an idle instance has no workload or filesystem changes, and waking would restart billing), then runs the interval/lock-gatedscheduleBackupIfNeeded.POST /debug/reprovisionnow checkpoints only a live runtime, bounded byREPROVISION_CHECKPOINT_TIMEOUT_MS, verifies restorability by HEADing the newest catalog entries’meta.json, and fails closed (409) unless a restorable backup exists;{"force":true}overrides for an already-unrecoverable container. Root cause of the empty buckets: the old health-triggered path scheduled a snapshot regardless ofreadiness.ok, so a booting/stopped container could startsandbox.createBackup(), hang, and leave a stalebackup-lock(observed infouria-backup-tarjetero); dev instances the HealthMonitor does not monitor had no trigger at all. Tests:src/routes/public.test.ts,src/routes/debug.test.ts,src/index.test.ts,workers/fouria-dispatch/test/unit/control.test.ts; corpustest/business-rules/operational-modes.txt; docsdocs/OPERATOR-GUIDE.md,apps/fouria/docs/WORKER-DEVELOPMENT.md,docs/CLOUD-ARCHITECTURE.md,workers/fouria-dispatch/README.md. E2E:test/e2e-backup/zz_reprovision_guard.txt+test/e2e-backup/zzz_backup_cold_restore.txt+test/e2e-backup/zzz_backup_semantics.txt(separatetest/e2e-backup/corpus, excluded from the defaulttest/e2erun; run locally withbun run test:e2e:backup; registered inphaseE_restore, reprovision guard first), the dispatch fan-out unit testworkers/fouria-dispatch/test/unit/cron-fanout.test.ts, deterministic E2E backup env intest/e2e/fixture/server/deploy, andPOST /_control/:slug/backupon the dispatch worker; design intest/e2e/BACKUP-E2E-DESIGN.md. Dispatch note:vitest.config.tsnow hasunit(node) +integration(workerd) projects so the fan-out logic is testable without Miniflare. -
PR E2E scope: the
e2ejob now setsE2E_BOOT_ONLY=1for pull requests, sorun-parallel.shprovisions the worker and only asserts the gateway reachesstartup.ok + readiness.ok, then tears down without running any cctr group.developand a manualworkflow_dispatch(newfull_testsinput, default checked) run the full corpus. Seeapps/fouria/test/e2e-parallel/run-parallel.shanddocs/TESTING.md. -
Paso4-exclusive instances + agent: lerma can now provision internal
scope: "paso4"instances (nullableclient_id, no billing plan,@paso4.io-only Access), and moltlazy ships an opt-inpaso4-devagent (PASO4_AGENTS, gated byMOLTLAZY_PASO4_AGENTS=trueand named inMOLTLAZY_AGENTS) that picks up GitHub issues via the bundledgithubskill (ghCLI, installed in the image) authenticated byGITHUB_TOKEN. The Dispatch Worker resolves paso4 tenants frominstances(noclientsrow) and rejects non-@paso4.ioemails. See apps/lerma/README.md anddocs/ENV-VARIABLES.md. -
Setup serialization (
start-openclaw.sh):start,setup, andfulltake an exclusiveflockon/tmp/openclaw-setup.lock(~900s wait;OPENCLAW_SETUP_LOCK_TIMEOUToverride) right after the gateway fast-path, then re-check the fast-path and exit 0 if a concurrent invocation brought the gateway up while they waited. In production,startacquires this lock before the setup body and holds it through the Gatewayexec, so one owner serializes setup and launch. The Worker’sgatewayStartPromisesguard is per-isolate, so an E2E cold-start burst lands several runs in different isolates at once; without the lock they ran the slow setup body concurrently, contended on OpenClaw’s native config/state locks, and stalled cold start for ~18 min (run 36692775633 —[SandboxRecovery]never fired because the admission-failure streak is also per-isolate).fullreleases fd 9 before backgrounding the gateway so the child does not inherit the lock;post-startis outside it. Degrades to the previous unserialized behaviour whenflockis absent. -
Native supervised gateway launch (
src/gateway/process.ts): the Worker’s production cold start is one atomic supervisedsandbox.execofstart-openclaw.sh start, which acquires the container-wide startup lock before the setup body (R2 restore → onboard → plugin installs →moltlazy patch) and holds it through the gatewayexec—/bin/bash -lc 'touch /tmp/gateway-stdout.log; pgrep … || ( tail -f … & ); exec openclaw gateway run … >> /tmp/gateway-stdout.log 2>&1'— instead of the script backgrounding it with&.execreplaces the bash wrapper so the supervised process is the gateway (listProcesses/getProcesstrack it for the life of the container;kill()targets it directly); the guardedtailstarts the container-stdout log mirror as a sibling (safe because the Worker neverwaitForExits on the gateway handle), and the redirect preserves the ops-readable log file. After the port is ready the Worker dispatchesstart-openclaw.sh post-start(dev automation + boot-log reset, best-effort) detached (void, not awaited) on every cold start — the gateway is already serving, so post-start must not extend the shared start promise (dev instances runopenclaw automations list/createthere, 30–60s, and awaiting it left every request loggingGateway start already in progress … waiting...while/healthalready returned 200); because it is dispatched unconditionally, a/api/health-triggered boot (waitForReady:false) still creates the dev automation even though its caller returned early.start-openclaw.shgainssetup/post-startmodes; the defaultfullmode keeps the legacy background launch for manual runs andtest:gateway:boot. A sharedSTARTUP_TIMEOUT_MSdeadline bounds setup + port wait;launchWithRecoverycentralizes the stale/orphan/transport retry policy shared by both phases. Thesetup/post-startphases must stay child-free: the SDK’swaitForExitonly settles once the supervised terminal exits, which never happens while the script leaves a live background child (atail -flog mirror or a backgroundcloudflaredthere stalled every cold start for the fullSTARTUP_TIMEOUT_MS, re-running setup until the ~11-minute mark). The mirror is started by the native launch; cloudflared is detached withsetsid. Never calloutput()/runShellon the live gateway handle (it never exits): read diagnostics from the/tmp/gateway-stdout.log//tmp/startup.logtails instead (readLogTail). -
All three ClawHub plugins (
cloudflare-ai-gatewayprovider,diagnostics-otel,cloudflare-unified-billing) are pre-installed into the container image at Docker build time (unpinned → latest ClawHub release; capability consent recorded in the image). This moves the per-boot network installs off the latency-critical startup path:start-openclaw.shdetects them via the extension dir +plugins listand only falls back to a ClawHub install when that state is missing (e.g. a warm R2 restore replaced it). The Dockerfile does not copy the Paso4 plugin source — it downloads the published ClawHub artifact — so the image stays source-free. The plugin’s source and tests now live in Paso4/openclaw-plugins (removed from this repo); see docs/OPENCLAW-PLUGINS.md. Runtime enablement is written by a single workingconfig patchthat also enablesadmin-http-rpcand disablesdiagnostics-otelunless its endpoint is set (the patch must not carryplugins.bundledDiscovery— removed in 2026.9.x, it fails validation and drops the whole patch — norplugins.allow, which disables every bundled provider not listed). Memory search is disabled via the top-levelmemory.search.enabled=false(the oldagents.defaults.memorySearchkey is gone in 2026.9.x). Measured local cold boot: 35s vs 56s before -
v0.3.x $include Configuration: immutable per-section $include files (session/tools/logging); gateway/agents inline; gateway token via env SecretRef
-
Gateway restart split:
POST /api/admin/gateway/restartis the soft reload — sendsSIGUSR2to the liveopenclaw gateway runprocess (in-process, no connection drop, no port rebind) viasoftReloadGatewayand falls back tosignalRestoreNeededonly when no live gateway is found;POST /debug/restart-openclaw(DEBUG_ROUTES=true, ROOT) is the hard restart —killGateway+resetStartupCache+signalRestoreNeeded+start-openclaw.shreprovision (in-place, nostartGatewayDirect) so existing client tokens keep working.SIGUSR1is not a restart signal (it opens Node’s inspector on127.0.0.1:9229); the E2En_gateway_soft_reload.txtasserts the inspector stays closed after a soft reload. The reload targets the server, not theopenclaw gateway runlauncher:buildSoftReloadScriptsignals the exactcomm(pgrep -x 'openclaw-gatewa', Linux 15-char truncation, plus a defensiveopenclaw-gateway), because signalling the launcher kills it and orphans the server without restarting it (readiness never dips while the endpoint still reportsreloaded: true). The E2E therefore requires both a stable/api/status.processIdand an increasedhttp server listeningcount, sinceprocessIdalone survives a broken reload. Issue #463:OPENCLAW_NO_RESPAWNstays — removing it cannot enable process-level restarts inside a container (see the decision record indocs/plans/sandbox-memory-usage.md) -
OPENCLAW_NO_RESPAWN=1is set in the Dockerfile as explicit intent only: inside a container OpenClaw already resolves restart mode to in-process (isContainerEnvironment()→disabled/unmanaged), so removing the flag cannot enable process-level restarts or reclaim memory (#463);softReloadGatewayonly signals live processes and skips zombies -
Sandbox SDK 1.0 preview (
@cloudflare/sandbox@0.13.0-next.776.1+cloudflare/sandbox:0.13.0-next.776.1image):startProcess/execStream/getLogs()andSANDBOX_TRANSPORTare gone. All command execution goes throughsandbox.exec(argv)(which resolves on launch, not completion);src/gateway/sandbox-exec.tscentralizes the shapes (startShellfor a handle,runShellfor buffered output,outputOf,collectLogs).listProcesses()now returnsProcessStatus[](.state, argv.command) and a live handle is fetched viagetProcess(id);findExistingGatewayProcessreturns aGatewayProcesswrapper.waitForProcessdelegates towaitForExit({timeout})and kills on timeout.SandboxOptions.labelswas removed upstream. Deploy cutover requireswrangler deploy --containers-rollout=immediate(stable/1.0 control protocols are incompatible) -
Stale sandbox recovery (
src/gateway/sandbox-recovery.ts): when the platform terminates the container behind a live Sandbox DO, every RPC fails withsandbox.exec error … HTTP error! status: 500—isStaleContainerErrorclassifies these andrecoverStaleSandboxdestroys the dead instance + resets startup/persistence caches so the next RPC auto-provisions a fresh container; wired intofindExistingGatewayProcess,checkGatewayHealthand theensureGatewaystartShellretry; cooldown-guarded (60s), never fires for deployOperationInterruptedErroror transient provisioning errors. It also matches the control-protocol mismatch ('utils.getRuntimeMetadata' is not a function,activateControlSession,RuntimeControlProtocolError) seen during a deploy cutover while the still-running container predates the new SDK’s control handshake — the SDK logs asandbox.exec error …line but re-throws the bareTypeError, so the message patterns are matched directly and the stale instance is reprovisioned instead of failing until the platform rollout replaces it -
Dev-only Fouria agents + client docs: moltlazy ships no production premade agents (vanilla
researcher/coderremoved) and injects Fouria Builder (fouria-builder) + Fouria Q&A agent (fouria-qa) only when the instance is a dev environment (OPENCLAW_DEV_MODE=trueon the container, derived by the Worker from the instance’s-devhostname — seesrc/dev-mode.ts); per-tenant client documentation lives in R2 undercustomers/<slug>/docs/and is materialized by the Worker into/home/openclaw/clawd/client-docs/before gateway start (src/services/fouria-docs/,materializeClientDocsIfDev), with a dev-only admin API at/api/admin/fouria-docs(404 outside dev); the Dockerfile installs thecctrbinary, platform docs are fetched from the docs site (https://fouria.io, fallbackhttps://fouria-docs.pages.dev, starting at/llms.txt), andstart-openclaw.shkeepssecurity.installPolicy allowin dev so the Builder can self-install plugins -
Orphaned-instance recovery (
src/gateway/sandbox-recovery.ts): a container destroyed out-of-band (Cloudflare Containers dashboard “Destroy”, failed force-stop) leaves the DO alive pointing at a dead placement and surfaces the SDK’s retryable admission errors (no container instance available,max container instances exceeded) instead of the 500 signature.isContainerUnavailableError+trackProvisioningFailureescalate to a recovery only once the streak reachesORPHANED_INSTANCE_MIN_FAILURESand spansORPHANED_INSTANCE_GRACE_MS(120s, > the SDK’s ≈110s provisioning budget, so slow-but-healthy cold starts are never cancelled);maybeRecoverSandboxunifies both failure classes and clears the streak whenever a sandbox RPC succeeds. Operator lever:POST /debug/reprovision(DEBUG_ROUTES, ROOT) destroys the instance + clears caches + resets the streak without touching the gateway, so it works when the container is entirely gone -
Activity-driven wake (
src/activity/): automations run inside the Gateway, so it must be running for schedules to fire. A per-tenantActivitySchedulerDurable Object (bindingACTIVITY, SQLite-backed, separate from the SDKSandboxDO whose alarm drivessleepAfter) mirrors the tenant’s schedule (openclaw automations list --json, refreshed on warm/api/healthwhenreadiness.ok, throttled to 5 min) and arms one alarmCRON_WAKE_AHEAD_MINUTES(default 10) before the next enabled job. On fire it wakes + heals viaensureGateway(recovering an orphaned instance) then re-arms; no enabled jobs → alarm cleared → container sleeps. Healing/wakeup is independent of user traffic, gated on job activity, and free when idle (src/activity/alarm.tsstate machine is unit-tested; the DO is a thin adapter). Requires theACTIVITYbinding inwrangler.jsonc+ lerma (tenant_provisioner.exresource_bindings,dispatch_namespace.ex@do_sqlite_classes) -
Container image build profiles: the Dockerfile takes a
BUILD_PROFILEbuild arg (defaultproduction→ slim, onlyxz-utils;dev→ adds the optionalpandoc+weasyprintPDF/document stack, ~760 MB, used by the dev-only Fouria Builderfouria-leadsskill). The profile is selected per wrangler environment viaimage_vars(equivalent todocker build --build-arg):env.staging(dev/test) setsdev;env.productionand the default/local config keep the slim default. The release CD builds the slim default, and the E2Ebasic(PR) andproduction(develop) profiles both provision the slim production image on the samestandard-2instance (differ by env only) -
Sandbox → leads R2 bridge (
src/sandbox-leads-bridge.ts,leads.r2.internal): thefouria-leadsskill now reaches the globalpaso4-leadsbucket through the SandboxoutboundByHostbridge (plain HTTP resolved in the Workers runtime with theLEADS_BUCKETbinding — no R2 creds in the container, no public route) instead of the publicwss://<worker>/api/leads/wsproxy. The proxy is intercepted by Cloudflare Access on tenant hostnames (a per-tenant Access app shadows the*.fouria.io/<path>apps — confirmed via the login redirect’skid) and is retained only for the host-side E2E fixture. Related: lerma’sHealthMonitornow probes withredirect: false, so a shadowed/api/health302 (Access login page) is surfaced instead of being followed and misreported as healthy -
Client docs hardening + operations (dev-only): tenant-isolated the
leads.r2.internalbridge (src/sandbox-leads-bridge.tsresolves the caller slug via the sharedsrc/sandbox-tenant.tsresolver and treats every agent path as relative to<slug>/, so a sandboxed agent can only reach its own proposal; the sharedbuild-pdf.shstays read-only). Added the Admin Client Docs page (/_admin/docs, ADMIN, dev-only) + the extendedsrc/routes/api/fouria-docs.tsmanaging docs (customers/<slug>/docs/) and leads directly from the Worker (no CLI) with markdown rendering (marked+DOMPurify), native-browser PDF opening via/raw, and multipart uploads; HTML/SVG raw downloads are forced toattachment+ sandbox CSP. Added Worker-side git-like versioning (DocsGitStore: content-addressed blobs/commits/refs,GET /commits,POST /rollback/:shaas a non-destructive revert) and backups (DocsBackupStore:POST /backup,GET /backups,POST /backups/:id/restore, retention 10). Seedocs/WORKER-DEVELOPMENT.md§“Client Docs (dev-only admin surface)” -
ECONNRESETfrom the sandbox R2 bridges fixed: the tenant’sLEADS_BUCKETwas provisioned as a plain-text var (the bucket name) instead of anr2_bucketbinding, soenv.LEADS_BUCKETwas a string,bucket.list()threw inside theoutboundByHosthandler, and the escaping exception surfaced to the container as an opaqueECONNRESET. Two fixes: (1) the bridges (leads/docs/releases) now validate the binding withisR2Bucket(src/r2-binding.ts) and wrap their bodies in try/catch, returning a structured JSON503/500instead of resetting the socket; (2) lerma (instances.exinstance_bindings/2, configLEADS_R2_BUCKETdefaultpaso4-leads) now provisionsLEADS_BUCKETas anr2_bucketbinding on every tenant worker (redeploy existing tenants to replace the stray var) -
Admin RPC-first read path (
src/gateway/admin-rpc.ts): the hot dashboard reads/writes now go through OpenClaw’s bundledadmin-http-rpcplugin (POST /api/v1/admin/rpc) oversandbox.containerFetch, reusing the typedmoltlazy/sdkHTTPRpcClient(which gained an injectablefetcher). This removes the ~10–15s per-call CLI spawn/WS-reconnect onGET /api/admin/devices,POST .../approve,GET /api/admin/models, andGET /api/admin/channels/probe(measured locally: ~1.3–2.5s CLI → ~30–40ms RPC), each of which keeps its CLI path as a fallback (createAdminRpcClientreturns null / throws → existing parser runs).start-openclaw.shenables the plugin with a dedicatedplugins.entries["admin-http-rpc"].enabled=truepatch — NOTplugins.allow, which disables every bundled provider not listed (and the nearby batched plugin patch is rejected upstream for the removedplugins.bundledDiscoverykey).src/index.tsshadows/api/v1/admin/rpcwith a 404 before the public catch-all proxy so the operator surface stays container-local. The home-grownmoltlazy/sdkWebSocket client is intentionally not used for admin RPC -
Gateway WebSocket RPC (
src/gateway/gateway-rpc.ts): methods outside theadmin-http-rpcallowlist (notablyaudit.activity.list) are reached with OpenClaw’s official@openclaw/gateway-client/browserreference client +@openclaw/gateway-protocoltypes, bridged oversandbox.wsConnectand authenticated with the shared gateway token as anoperator. The wire-v4 handshake (connect.challenge→connect→hello-ok) is owned by the official client; the connect frame’sclient.versioncomes from__OPENCLAW_VERSION__, injected byvite.config.tsfromcatalog.openclaw. RPC-first with CLI fallback:GET /api/admin/audit-logs(audit.activity.list) and the agent board (agents.create/agents.delete/agents.update, model viaconfig.patch).GET /api/admin/integrationsis deliberately WebSocket-free (the gateway drops these sockets): channels/agents over the admin HTTP RPC, plugin enabled-state fromconfig.get, toggles/config viaconfig.patch, secrets reload via the CLI.theme, agent bindings,secrets.apply, andchannels list --allhave no RPC method and stay on the CLI. The two packages are pinned incatalog.@openclaw/gateway-client/catalog.@openclaw/gateway-protocoland must trackcatalog.openclaw -
Realtime device-pairing notifications (
src/gateway/gateway-rpc.tsopenGatewayEventStream,src/routes/api/devices.tsGET /ws): the dashboardNotificationBellnow subscribes toGET /api/admin/devices/ws, which pushes an initial{ pending, paired }snapshot then relays Gatewaydevice.pair.*events (reconnecting via the official gateway client oversandbox.wsConnect). Admins approve/reject pending devices inline from the bell viaPOST /api/admin/devices/:requestId/approve/.../reject(RPC-first; the reject route was added). Also fixedsecrets applyinvocations (integrations/dreaming): current OpenClaw removed--yes(only--from,--dry-run,--allow-exec,--json) -
CLI→RPC sweep (finishing the RPC-first migration):
configset/unset useconfig.patch;dreamingconfig writes useconfig.patch(sharedpatchConfigViaRpcinsrc/routes/api/shared.ts);secrets.reloaduses the WSsecrets.reload; andresolveAgentFlag(used bymodels/knowledge) resolves the roster viaagents.listinstead ofopenclaw agents list. CLI fallbacks are retained everywhere. Remaining CLI-only paths have no Gateway RPC method:secrets apply/audit,openclaw wiki *, agentbind/unbind/bindings, andopenclaw --version