# Agent Guidelines

Guidelines for AI agents working on this codebase.

## Project Overview

This is a Cloudflare Worker that runs [OpenClaw](https://github.com/openclaw/openclaw) (formerly Moltbot/Clawdbot) in a Cloudflare Sandbox container. It provides:

- Proxying to the OpenClaw gateway (web UI + WebSocket)
- Admin UI at `/_admin/` for device management
- API endpoints at `/api/*` for device pairing
- Debug endpoints at `/debug/*` for troubleshooting

**Note:** The CLI tool and npm package are now named `openclaw`. Config files use `.openclaw/openclaw.json`. Legacy `.clawdbot` paths are supported for backward compatibility during transition.

### Paso4 Extensions (`moltlazy/`)

The `moltlazy/` module is fully owned by the **Paso4 developers** and provides two systems:

1. **Config Module** (`config/` + `moltlazy/index.ts`) — filesystem patcher used by `start-openclaw.sh` at container startup to write `openclaw.json` before the gateway launches.
2. **Features Module** (`sdk/`) — RPC-based config applied to a **running** gateway via the Admin HTTP API. Supported feature packs: Knowledge Graph, CF Unified Billing.

## Project Structure

```
src/
├── index.ts          # Main Hono app, route mounting
├── types.ts          # TypeScript type definitions (AUTOGENERATED by `bun run types`, do not apply changes directly here, instead re-run the command after implementing the rest of the features)
├── config.ts         # Constants (ports, timeouts, paths)
├── auth/             # Cloudflare Access authentication
│   ├── jwt.ts        # JWT verification
│   ├── jwks.ts       # JWKS fetching and caching
│   └── middleware.ts # Hono middleware for auth
├── gateway/          # OpenClaw gateway management
│   ├── process.ts    # Process lifecycle (find, start)
│   ├── env.ts        # Environment variable building
│   ├── r2.ts         # R2 bucket mounting
│   ├── sync.ts       # R2 backup sync logic
│   └── utils.ts      # Shared utilities (waitForProcess)
├── routes/           # API route handlers
│   ├── api.ts        # /api/* endpoints (devices, gateway)
│   ├── admin.ts      # /_admin/* static file serving
│   └── debug.ts      # /debug/* endpoints
└── client/           # React admin UI (Vite)
    ├── App.tsx
    ├── api.ts        # API client
    └── pages/
moltlazy/             # Paso4 proprietary configuration utilities
├── cli.ts            # CLI entry point for patching config
├── index.ts          # Main logic for config patching
├── types.ts          # Type definitions for OpenClaw config
├── config/
│   └── include.ts     # per-section $include injection (session/tools/logging)
├── config/
├── config/           # Configuration templates
├── agents/           # Agent configuration templates
│   ├── defaults.ts   # Default agent settings
│   └── premade/      # Pre-configured agent definitions
├── models/           # Provider-specific model configurations
├── sdk/              # Features Module — RPC-based config for running gateway
│   ├── client.ts     # Typed RPC client over Admin HTTP API
│   ├── orchestrator.ts # Applies supported feature packs sequentially
│   └── feature-packs/
│       ├── knowledge-graph.ts         # Obsidian memory backend, wiki, QMD
│       └── plugin-cf-unified-billing.ts # CF AI Gateway unified billing
└── tests/            # Tests for configuration utilities
```

## Key Patterns

### Environment Variables

- `DEV_MODE` - Skips CF Access auth AND bypasses device pairing (maps to `OPENCLAW_DEV_MODE` for container)
- `DEBUG_ROUTES` - Enables `/debug/*` routes (disabled by default)
- See `src/types.ts` for full `MoltbotEnv` interface
- See [`docs/schemas/secrets-manifest.schema.json`](./docs/schemas/secrets-manifest.schema.json) for the secrets manifest

### CLI Commands

On OpenClaw 2026.8.2 the CLI **no longer accepts** `--url ws://localhost:18789`
(and most local commands reject `--token` too). Commands run inside the container
(the gateway host) either operate on local state or auto-connect to the local
Gateway — auth resolves from the `OPENCLAW_GATEWAY_TOKEN` env SecretRef, so no
connection flags are passed. Commands that require an owner on a multi-agent
roster take `--agent <id>`. Never hardcode the id: resolve it at runtime via
`openclaw agents list --json` (prefer `main` when present, else the first
configured agent — e.g. the dev-only `fouria-builder`/`fouria-qa` premades when
running with `DEV_MODE=true`; see
`resolveAgentFlag()` in `src/routes/api/shared.ts` and `resolveCliAgentId()`
in `moltlazy/sdk/cli.ts`):

```typescript
sandbox.startProcess('openclaw devices list --json');
// multi-agent commands need an explicit, dynamically-resolved owner:
sandbox.startProcess(`openclaw models list --json${await resolveAgentFlag(sandbox)}`);
```

Gateway-connecting commands (`devices`, `cron`, `secrets reload`) may still accept
`--url`/`--token` when given together, but the 2026.8.2 guidance is to drop `--url`
and let the CLI use the configured local target.

CLI commands take 10-15 seconds due to WebSocket connection overhead. Use `waitForProcess()` helper in `src/routes/api.ts`.

### Success Detection

The CLI outputs "Approved" (capital A). Use case-insensitive checks:

```typescript
stdout.toLowerCase().includes('approved');
```

## Commands

```bash
bun run lint          # Run linting (oxlint)
bun run test          # Run tests (vitest)
bun run test:watch    # Run tests in watch mode
bun run test:gateway:boot     # Local gateway boot smoke test (Docker)
bun run test:gateway:boot:ai  # Same, with real AI Gateway onboard path (.dev.vars.e2e)
bun run build         # Build client (vite)
bun run build:all     # Build client + moltlazy + plugin
bun run build:docker  # Build all + install deps (pre-Docker build)
bun run deploy        # Build all and deploy to production
bun run deploy:dev    # Build all and deploy to dev
bun run dev           # Vite dev server
bun run start         # wrangler dev (local worker)
bun run typecheck     # TypeScript check
```

## Contribution Guidelines

### Validation Requirements

Before submitting any contribution:

1. **Run lint and tests** - All builds must pass:

   ```bash
   bun run lint
   bun run test
   ```

2. **Environment variable changes** - If your contribution adds, updates, or deletes an environment variable, you MUST update it in all three locations:
   - `README.md` - Add to the secrets reference table
   - `src/gateway/env.ts` - Add to `buildEnvVars()` function if passed to container
   - `.dev.vars.example` - Add to the mapping comments and example

### Testing

Tests use Vitest. Test files are colocated with source files (`*.test.ts`).

Current test coverage:

- `auth/jwt.test.ts` - JWT decoding and validation
- `auth/jwks.test.ts` - JWKS fetching and caching
- `auth/middleware.test.ts` - Auth middleware behavior
- `gateway/env.test.ts` - Environment variable building
- `gateway/process.test.ts` - Process finding logic
- `gateway/r2.test.ts` - R2 mounting logic
- `gateway/sync.test.ts` - R2 backup sync logic
- `moltlazy/tests/index.test.ts` - per-section $include injection and config generation

When adding new functionality, add corresponding tests.

## Code Style

- Use TypeScript strict mode
- Prefer explicit types over inference for function signatures
- Keep route handlers thin - extract logic to separate modules
- Use Hono's context methods (`c.json()`, `c.html()`) for responses

## Documentation

- `README.md` - User-facing documentation (setup, configuration, usage)
- `AGENTS.md` - This file, for AI agents

Development documentation goes in AGENTS.md, not README.md.

---

## Architecture

```
Browser
   │
   ▼
┌─────────────────────────────────────┐
│     Cloudflare Worker (index.ts)    │
│  - Starts OpenClaw in sandbox       │
│  - Proxies HTTP/WebSocket requests  │
│  - Passes secrets as env vars       │
└──────────────┬──────────────────────┘
               │
               ▼
┌─────────────────────────────────────┐
│     Cloudflare Sandbox Container    │
│  ┌───────────────────────────────┐  │
│  │     OpenClaw Gateway          │  │
│  │  - Control UI on port 18789   │  │
│  │  - WebSocket RPC protocol     │  │
│  │  - Agent runtime              │  │
│  └───────────────────────────────┘  │
└─────────────────────────────────────┘
```

### Key Files

| File                | Purpose                                                               |
| ------------------- | --------------------------------------------------------------------- |
| `src/index.ts`      | Worker that manages sandbox lifecycle and proxies requests            |
| `src/metrics.ts`    | Cost and usage metrics instrumentation for Analytics Engine           |
| `Dockerfile`        | Container image based on `cloudflare/sandbox` with Node 22 + OpenClaw |
| `start-openclaw.sh` | Startup script: R2 restore → onboard → config patch → launch gateway  |
| `wrangler.jsonc`    | Cloudflare Worker + Container configuration                           |

## Cost Observability

The Worker instruments cost metrics using Cloudflare Analytics Engine for per-client cost tracking:

| Event Type         | Dimensions                            | Metrics                        |
| ------------------ | ------------------------------------- | ------------------------------ |
| `ai-query`         | model, provider, customer, actionType | tokensIn, tokensOut, costUsd   |
| `container-active` | customer, instanceType, action        | heartbeat count                |
| `r2-operation`     | customer, opType, opClass             | bytesTransferred, durationMs   |
| `cron-trigger`     | customer, triggerReason               | wokeContainer flag             |
| `ws-session`       | customer, sessionId                   | durationSeconds, messagesCount |

### Analytics Engine Configuration

The `ANALYTICS` binding writes to dataset `fourai_cost_metrics`. Query via GraphQL:

```graphql
query CostMetrics($accountTag: String!) {
  viewer {
    accounts(filter: { accountTag: $accountTag }) {
      analyticsEngineDatasets(filter: { name: "fourai_cost_metrics" }) {
        data(
          query: "SELECT blob1 as event_type, blob2 as customer, blob3 as model, SUM(double1) as tokens_in, SUM(double2) as tokens_out FROM fourai_cost_metrics WHERE timestamp > NOW() - INTERVAL 1 DAY GROUP BY blob1, blob2, blob3"
        ) {
          result
        }
      }
    }
  }
}
```

### Cost API Endpoint

GET `/api/admin/costs` returns pricing constants and query examples for cost dashboards.

## Local Development

```bash
npm install
cp .dev.vars.example .dev.vars
# Edit .dev.vars with your CLOUDFLARE_AI_GATEWAY_API_KEY + CLOUDFLARE_API_TOKEN + CF_AI_GATEWAY_ACCOUNT_ID + CF_AI_GATEWAY_GATEWAY_ID
npm run start
```

### Environment Variables

For local development, create `.dev.vars`:

```bash
CLOUDFLARE_AI_GATEWAY_API_KEY=your-provider-api-key
CLOUDFLARE_API_TOKEN=your-ai-scoped-api-token  # REST transport; use the same AI-scoped credential locally
CF_AI_GATEWAY_ACCOUNT_ID=your-account-id
CF_AI_GATEWAY_GATEWAY_ID=your-gateway-id
DEV_MODE=true           # Skips CF Access auth + device pairing
DEBUG_ROUTES=true       # Enables /debug/* routes
```

### WebSocket Limitations

Local development with `wrangler dev` has issues proxying WebSocket connections through the sandbox. HTTP requests work but WebSocket connections may fail. Deploy to Cloudflare for full functionality.

## Docker Image Caching

The Dockerfile includes a cache bust comment. When changing `start-openclaw.sh`, bump the version:

```dockerfile
# Build cache bust: 2026-02-06-v28-openclaw-upgrade
```

## Gateway Configuration

OpenClaw configuration is built at container startup:

1. R2 backup is restored if available (with migration from legacy `.clawdbot` paths)
2. If no config exists, `openclaw onboard --non-interactive` creates one based on env vars
3. Feature flags come from the per-customer `MOLTLAZY_FEATURE_FLAGS` binding (managed via the FourIA API)
4. `start-openclaw.sh` runs `moltlazy patch` which generates per-section include files (`moltlazy-session/tools/logging.json`) and injects them as per-section `$include` directives (no root `$include`); `gateway.*`/`agents.*` are written inline and the gateway token is an env SecretRef
5. Gateway starts with `openclaw gateway --allow-unconfigured --bind lan`

### Feature Flag Configuration

Feature flags are managed via the **FourIA API** and applied as a **per-customer binding** (`MOLTLAZY_FEATURE_FLAGS`, a JSON string). The Worker reads the binding and passes it to the container via `buildEnvVars()`. The legacy `CONFIG_BUCKET` / `config-cache.ts` (R2 + ETag) mechanism was removed.

**Feature Flags Example:**

```json
{
  "knowledgeGraph": true,
  "unifiedBilling": false
}
```

The container consumes `MOLTLAZY_FEATURE_FLAGS` at startup (`moltlazy patch` generates the per-section include files and injects them as per-section `$include` into `openclaw.json`). There is no worker-side cache — the binding is read directly.

### AI Provider

Cloudflare AI Gateway is the only provisioned AI provider path. The startup script configures
auth when all of these env vars are set:

1. **Cloudflare AI Gateway** (native): `CLOUDFLARE_AI_GATEWAY_API_KEY` + `CF_AI_GATEWAY_ACCOUNT_ID` + `CF_AI_GATEWAY_GATEWAY_ID`

### Container Environment Variables

These are the env vars passed TO the container (internal names):

| Variable                        | Config Path                   | Notes                                                                       |
| ------------------------------- | ----------------------------- | --------------------------------------------------------------------------- |
| `CLOUDFLARE_AI_GATEWAY_API_KEY` | (env var)                     | Native AI Gateway key                                                       |
| `CLOUDFLARE_API_TOKEN`          | (env var)                     | AI Gateway REST API Bearer token; tenant binding is derived from the AI key |
| `CF_AI_GATEWAY_ACCOUNT_ID`      | (env var)                     | Account ID for AI Gateway                                                   |
| `CF_AI_GATEWAY_GATEWAY_ID`      | (env var)                     | Gateway ID for AI Gateway                                                   |
| `OPENCLAW_GATEWAY_TOKEN`        | `--token` flag                | Mapped from `MOLTBOT_GATEWAY_TOKEN`                                         |
| `OPENCLAW_DEV_MODE`             | `controlUi.allowInsecureAuth` | Mapped from `DEV_MODE`                                                      |
| `TELEGRAM_BOT_TOKEN`            | `channels.telegram.botToken`  | **@deprecated** — E2E only. Use integrations dashboard in production.       |
| `DISCORD_BOT_TOKEN`             | `channels.discord.token`      | **@deprecated** — E2E only. Use integrations dashboard in production.       |
| `SLACK_BOT_TOKEN`               | `channels.slack.botToken`     | **@deprecated** — E2E only. Use integrations dashboard in production.       |
| `SLACK_APP_TOKEN`               | `channels.slack.appToken`     | **@deprecated** — E2E only. Use integrations dashboard in production.       |

## OpenClaw Config Schema

OpenClaw has strict config validation. Common gotchas:

- `agents.defaults.model` must be `{ "primary": "model/name" }` not a string
- `gateway.mode` must be `"local"` for headless operation
- No `webchat` channel - the Control UI is served automatically
- `gateway.bind` is not a config option - use `--bind` CLI flag

See [OpenClaw docs](https://docs.openclaw.ai/) for full schema.

## Common Tasks

### Adding a New API Endpoint

1. Add route handler in `src/routes/api.ts`
2. Add types if needed in `src/types.ts`
3. Update client API in `src/client/api.ts` if frontend needs it
4. Add tests

### Adding a New Environment Variable

1. Add to `MoltbotEnv` interface in `src/types.ts`
2. If passed to container, add to `buildEnvVars()` in `src/gateway/env.ts`
3. Update `.dev.vars.example`
4. Document in README.md secrets table

### Debugging

```bash
# View live logs
npx wrangler tail

# Check secrets
npx wrangler secret list
```

Enable debug routes with `DEBUG_ROUTES=true` and check `/debug/processes`.

## R2 Backup Encryption (F4E1)

When `BACKUP_ENCRYPTION_KEY` (base64 32-byte secret) is set, the Worker post-encrypts every
snapshot after `sandbox.createBackup()`: `backups/<id>/data.sqsh` is replaced by
`data.sqsh.enc` (chunked AES-256-GCM, per-object HKDF key) and `meta.json` is flagged
`encrypted: true` (kept readable — the SDK needs it). Restores decrypt transiently and delete
the plaintext in a `finally` block. Config versions under `configs/` are encrypted with
`customMetadata.encrypted='true'`. Legacy plaintext backups remain restorable.
Set `BACKUP_ENCRYPTION_REQUIRED=true` to fail closed. Crypto lives in `src/crypto/`.
See `docs/plans/r2-encryption-at-rest.md`.

## R2 Storage Notes

Persistence uses the Sandbox SDK `createBackup()` / `restoreBackup()` API with
the `BACKUP_BUCKET` binding. The snapshot root is `/home/openclaw`; it is not an
s3fs mount and there is no `/data/moltbot` runtime path.

Never delete tenant backup objects outside the retention or decommission
procedure. Hard restarts must checkpoint before killing the gateway.

- **Process status**: The sandbox API's `proc.status` may not update immediately after a process completes. Instead of checking `proc.status === 'completed'`, verify success by checking for expected output (e.g., timestamp file exists after sync).

- **R2 prefix migration**: Backups are now stored under `openclaw/` prefix in R2 (was `clawdbot/`). The startup script handles restoring from both old and new prefixes with automatic migration.

## Cloudflare Tunnel Setup (Manual Flow)

This section documents how to expose an internal service (e.g. the Obsidian sync port 18790) via a Cloudflare Tunnel with a public hostname. Follow this end-to-end when setting up a new tunnel route.

### 1. Register and Add a Domain

You need a domain zone in your Cloudflare account before any DNS record can be created.

1. Register a domain (e.g. via Cloudflare Registrar or a third-party registrar like Spaceship/Namecheap).
2. In the [Cloudflare Dashboard](https://dash.cloudflare.com/), select the **Paso4** account → **Websites** → **Add a site**.
3. Enter the domain name and choose the **Free** plan.
4. Cloudflare will assign two nameservers, e.g.:
   - `albert.ns.cloudflare.com`
   - `priscilla.ns.cloudflare.com`
5. At your registrar, replace the existing nameservers with the Cloudflare ones.
6. Wait for DNS propagation (typically 5–30 minutes). The zone status will change from `pending` to `active`.

**Current domain:** `paso4.io` (zone ID: `6ce23f544f57a1909578032abcda18dd`, account: Paso4)

### 2. Create the Cloudflare Tunnel

If the tunnel does not yet exist, create it with `cloudflared`:

```bash
cloudflared tunnel create obsidian-sync
```

This prints a tunnel UUID and writes credentials to `~/.cloudflared/<UUID>.json`.

**Current tunnel:** `obsidian-sync` — UUID `90792969-7077-40a8-ae10-28e716290d9d`

To get the tunnel token (needed for `CLOUDFLARE_TUNNEL_TOKEN`):

```bash
cloudflared tunnel token 90792969-7077-40a8-ae10-28e716290d9d
```

Set it as a Worker secret:

```bash
npx wrangler secret put CLOUDFLARE_TUNNEL_TOKEN
```

### 3. Create the DNS CNAME Record

A CNAME record must point from your public subdomain to `<UUID>.cfargotunnel.com`.

**Via Cloudflare Dashboard:**

1. Go to **paso4.io** → **DNS** → **Records** → **Add record**.
2. Fill in:

   | Field        | Value                                                   |
   | ------------ | ------------------------------------------------------- |
   | Type         | `CNAME`                                                 |
   | Name         | `obsidian-sync`                                         |
   | Target       | `90792969-7077-40a8-ae10-28e716290d9d.cfargotunnel.com` |
   | Proxy status | **Proxied** (orange cloud — required)                   |
   | TTL          | Auto                                                    |

3. Save. Public URL becomes `https://obsidian-sync.paso4.io`.

**Via MCP API (requires DNS:Edit on the API token):**

```js
cloudflare.request({
  method: 'POST',
  path: `/zones/6ce23f544f57a1909578032abcda18dd/dns_records`,
  body: {
    type: 'CNAME',
    name: 'obsidian-sync',
    content: '90792969-7077-40a8-ae10-28e716290d9d.cfargotunnel.com',
    proxied: true,
    comment: 'Cloudflare Tunnel route for obsidian-sync',
  },
});
```

To grant DNS:Edit to the MCP token: `https://dash.cloudflare.com/profile/api-tokens` → edit the token → add **Zone → DNS → Edit**.

### 4. Configure the Tunnel Ingress

Create or update `~/.cloudflared/config.yml` on the host running `cloudflared`:

```yaml
tunnel: 90792969-7077-40a8-ae10-28e716290d9d
credentials-file: /root/.cloudflared/90792969-7077-40a8-ae10-28e716290d9d.json

ingress:
  - hostname: obsidian-sync.paso4.io
    service: http://localhost:18790
  - service: http_status:404
```

Then run:

```bash
cloudflared tunnel run obsidian-sync
```

### 5. Verify

```bash
# Check tunnel is connected
cloudflared tunnel info obsidian-sync

# Test the public endpoint
curl -I https://obsidian-sync.paso4.io
```

### Gotchas

- The `cfargotunnel.com` CNAME only routes traffic for DNS records in the **same Cloudflare account** as the tunnel. Cross-account routing is blocked.
- The zone must be **active** (nameservers delegated) for public DNS to resolve. A `pending` zone means nameserver propagation is not complete yet.
- If the tunnel stops, the DNS record is not deleted — visitors will see a Cloudflare `1016` error.
- The `CLOUDFLARE_TUNNEL_TOKEN` secret in the Worker is used by `start-openclaw.sh` to run `cloudflared` inside the container, exposing port 18790 to the tunnel.

---

FourIA offers Private Cloud only (Business On Premise is not offered at this time). Pricing is credit-based: Credits are the exclusive billable unit, consumed by completed messages and automations. Credits do NOT include AI costs — AI expenses are managed separately via BYOL or AI Gateway billing. Reader users are unlimited and free. Builder seats (users who can create and edit agents) are plan-limited. The v1.0-beta is delivered through the Design Partner Program with a single engaged partner. Partners pay a fixed credit fee negotiated with sales. See docs/VISION.md for the full pricing definition.

## Active Technologies

- OpenTofu 1.6+ (HCL) + `cloudflare` provider (Terraform/OpenTofu) (001-opentofu-deploy)
- Cloudflare R2 (for FourIA data), OpenTofu Local/Remote State (for infrastructure tracking) (001-opentofu-deploy)

## Recent Changes

- v0.3.x $include Configuration: immutable per-section $include files (session/tools/logging); gateway/agents inline; gateway token via env SecretRef
- Gateway restart split: `POST /api/admin/gateway/restart` is the **soft reload** — sends `SIGUSR1` to the live `openclaw gateway run` process (in-process, no connection drop, no port rebind) via `softReloadGateway` and falls back to `signalRestoreNeeded` only when no live gateway is found; `POST /debug/restart-openclaw` (DEBUG_ROUTES=true, ROOT) is the **hard restart** — `killGateway` + `resetStartupCache` + `signalRestoreNeeded` + `start-openclaw.sh` reprovision (in-place, no `startGatewayDirect`) so existing client tokens keep working
- `OPENCLAW_NO_RESPAWN=1` is set in the Dockerfile so the gateway performs in-process restarts (no orphan/respawn loops); `softReloadGateway` only signals live processes and skips zombies
- Stale sandbox recovery (`src/gateway/sandbox-recovery.ts`): when the platform terminates the container behind a live Sandbox DO, every RPC fails with `sandbox.exec error … HTTP error! status: 500` — `isStaleContainerError` classifies these and `recoverStaleSandbox` destroys the dead instance + resets startup/persistence caches so the next RPC auto-provisions a fresh container; wired into `findExistingGatewayProcess`, `checkGatewayHealth` and the `ensureGateway` `startProcess` retry; cooldown-guarded (60s), never fires for deploy `OperationInterruptedError` or transient provisioning errors
- Dev-only Fouria agents + client docs: moltlazy ships no production premade agents (vanilla `researcher`/`coder` removed) and injects **Fouria Builder** (`fouria-builder`) + **Fouria Q&A agent** (`fouria-qa`) only when `OPENCLAW_DEV_MODE=true`; per-tenant client documentation lives in R2 under `customers/<slug>/docs/` and is materialized by the Worker into `/home/openclaw/clawd/client-docs/` before gateway start (`src/services/fouria-docs/`, `materializeClientDocsIfDev`), with a dev-only admin API at `/api/admin/fouria-docs` (404 outside dev); the Dockerfile installs the `cctr` binary, platform docs are fetched from the docs site (`https://fouria.io`, fallback `https://fouria-docs.pages.dev`, starting at `/llms.txt`), and `start-openclaw.sh` keeps `security.installPolicy allow` in dev so the Builder can self-install plugins
- Orphaned-instance recovery (`src/gateway/sandbox-recovery.ts`): a container destroyed out-of-band (Cloudflare Containers dashboard "Destroy", failed force-stop) leaves the DO alive pointing at a dead placement and surfaces the SDK's _retryable_ admission errors (`no container instance available`, `max container instances exceeded`) instead of the 500 signature. `isContainerUnavailableError` + `trackProvisioningFailure` escalate to a recovery only once the streak reaches `ORPHANED_INSTANCE_MIN_FAILURES` **and** spans `ORPHANED_INSTANCE_GRACE_MS` (120s, > the SDK's ≈110s provisioning budget, so slow-but-healthy cold starts are never cancelled); `maybeRecoverSandbox` unifies both failure classes and clears the streak whenever a sandbox RPC succeeds. Operator lever: `POST /debug/reprovision` (DEBUG_ROUTES, ROOT) destroys the instance + clears caches + resets the streak **without** touching the gateway, so it works when the container is entirely gone
- Activity-driven wake (`src/activity/`): automations run inside the Gateway, so it must be running for schedules to fire. A per-tenant `ActivityScheduler` Durable Object (binding `ACTIVITY`, SQLite-backed, **separate** from the SDK `Sandbox` DO whose alarm drives `sleepAfter`) mirrors the tenant's schedule (`openclaw automations list --json`, refreshed on warm `/api/health` when `readiness.ok`, throttled to 5 min) and arms one alarm `CRON_WAKE_AHEAD_MINUTES` (default 10) before the next enabled job. On fire it wakes + heals via `ensureGateway` (recovering an orphaned instance) then re-arms; no enabled jobs → alarm cleared → container sleeps. Healing/wakeup is independent of user traffic, gated on job activity, and free when idle (`src/activity/alarm.ts` state machine is unit-tested; the DO is a thin adapter). Requires the `ACTIVITY` binding in `wrangler.jsonc` + lerma (`tenant_provisioner.ex` `resource_bindings`, `dispatch_namespace.ex` `@do_sqlite_classes`)
