# Sandbox memory growth: worker-thread heaps + oversized sessions

> Status: **in progress — diagnosis captured from a dev instance; investigation and
> remediation tracked in milestone 11 (v2026.10 - Perfomance enhacements).**
> This plan bounds the OpenClaw gateway's resident memory (RSS) growth inside the
> Cloudflare Sandbox container, from the Worker/container layer we own, so a dev
> instance does not drift toward the container/sandbox memory limit without an
> automatic recovery.
>
> Landed so far: bounded transcript compaction (2.1), the soft-reload fixes
> (`SIGUSR1` → `SIGUSR2`, and the signal now reaches the gateway server instead of
> the `openclaw gateway run` launcher), the `OPENCLAW_NO_RESPAWN` keep decision
> ([#463](https://github.com/0xCAB0/fouria/issues/463) — see the decision record),
> **RSS/thread observability (Phase 1)** and **the dev-gated watchdog (2.4)** —
> both in `apps/fouria/src/gateway/memory.ts`, surfaced on `/api/health` and the
> health stream.
> Remaining: re-measuring the drift on a dev instance with the watchdog enabled,
> and the worker-pool cap investigation (2.2).

## Background

A dev instance (`fouria` / OpenClaw gateway in a Cloudflare Sandbox container)
was observed at **RSS ~1.85 GiB (peak 2.36 GiB), swap 0**, with OpenClaw's own
memory diagnostic already firing:

```
subsystem=gateway/diagnostics/memory  reason=rss_threshold  rss=1.84 GiB
heap=339.9 MiB  threshold=1.5 GiB  thresholdRatio=122.7%
```

A V8 heap snapshot of the **main isolate** (`/tmp/gw310-main.heapsnapshot`,
290 MB) showed the main isolate heap is **not** the driver:

| Metric (main isolate)             | Value                    |
| --------------------------------- | ------------------------ |
| `heapUsed`                        | ~340 MB / limit 2.35 GiB |
| total `self_size`                 | 306 MiB                  |
| top retained: `JSArrayBufferData` | 35.6 MiB                 |
| empty strings `""`                | 25 MiB (186k)            |
| `(object properties)`             | 17.7 MiB                 |
| `system/BytecodeArray`            | 9.4 MiB                  |

No large JS retainer, no JS leak in the main isolate. The log file is tiny
(316 KB → 1.5 MB); `--verbose` only raises log **frequency**, not memory.

## Evidence

Measured from `/proc/<pid>` on the live gateway process:

| Observation                         | Value                                                        |
| ----------------------------------- | ------------------------------------------------------------ |
| `[anon]`                            | 1,665 MB (Pss_Anon 1.82 GiB)                                 |
| glibc `[heap]`                      | 99 MB                                                        |
| node binary (RSS)                   | 66 MB                                                        |
| anonymous mappings of ~256 KB       | 4,885 ≈ **1.2 GiB**                                          |
| total threads                       | **31** (18 `WorkerThread` + 4 `V8Worker` + 4 `libuv-worker`) |
| OpenClaw `worker startup state` log | `workerCount=8`                                              |

The ~4,885 anonymous mappings of ~256 KB match V8's **heap page granularity**
(256 KB), i.e. per-isolate heap pages. The main isolate only accounts for
~412 MB of committed heap, so the remaining ~1.2 GiB lives in the **heaps of the
in-process `worker_thread`s** — each `worker_thread` is its own V8 isolate with
its own heap, invisible to the main isolate's `getHeapStatistics()`.

### Corroborating signals

- `workerCount=8` at startup vs **18** `WorkerThread` observed → thread/heap
  accumulation (workers not being reclaimed over the instance's life).
- SQLite contention: `sessions.history.read` / `canonical-validation` taking
  **9–10 s**.
- Session **`fouria-builder:main`** with **407,708 tokens** in compaction
  preflight — a huge persisted transcript.

### Side observation (soft-reload vs. inspector) — resolved

To take the snapshot, `SIGUSR1` was sent to the gateway PID; it **opened the Node
inspector on `127.0.0.1:9229`** (localhost-only; closes on gateway restart). This
confirmed a real defect: OpenClaw's documented restart signal is **`SIGUSR2`**
(`SIGUSR1` only starts the inspector), while our `softReloadGateway`
(`apps/fouria/src/gateway/process.ts`) sent `SIGUSR1` expecting an in-process
reload — so the soft reload was a **silent no-op that left a debug socket open**.

**Fixed.** `buildSoftReloadScript()` now sends `kill -USR2`, and
`test/e2e/n_gateway_soft_reload.txt` asserts the inspector stays closed
(`inspector-closed`) after a soft reload. `SIGUSR2` is authorized by default:
`isRestartEnabled(config)` is `commands.restart !== false` (default on) and
`server-start` calls `setGatewayRestartPolicy({ allowExternal: … })`, so an
external `SIGUSR2` schedules the in-process restart rather than logging
`SIGUSR2 restart ignored (not authorized; commands.restart=false)`.

**Second, independent defect (found while measuring the fix): the signal
targeted the launcher, not the server.** `buildSoftReloadScript()` signalled
`pgrep -f "[o]penclaw gateway run"`, which matches the **launcher** process
(`node /usr/local/bin/openclaw gateway run …`) that supervises the real server.
`kill -USR2 <launcher>` makes the launcher exit **without** restarting the
server: the server is orphaned (PPID 1), readiness never dips, no new
`http server listening` line appears — yet the endpoint still reports
`reloaded: true`, so a broken reload looked successful. Measured on a live local
worker (OpenClaw 2026.10.5): launcher-targeted reload `ack_ms=30`, readiness
`123/123` at `200 true`, launcher gone; a direct `kill -USR2 <server>` instead
restarted the server (`http server listening` +1), dipped readiness at +120 ms
(503) and recovered at ~3.3 s to listening / ~12–14 s to full readiness.

`buildSoftReloadScript()` now targets the server's exact `comm`
(`pgrep -x 'openclaw-gatewa'`, Linux's 15-char truncation, plus a defensive
`openclaw-gateway` for kernels that keep longer names), deduped with `sort -u`,
skipping zombies. Because `/api/status.processId` is the Worker-tracked
supervisor (which survives an in-process reload), the E2E
`n_gateway_soft_reload.txt` now asserts **both** the same `processId` **and** an
increased `http server listening` count — `processId` stability alone passed in
the broken case.

## Root cause (working hypothesis)

Two coupled causes, both reachable from our layer:

1. **Oversized session transcripts.** `fouria-builder:main` at 407k tokens keeps
   a large persisted transcript (JSONL + SQLite history), so the `worker_thread`s
   that serve it hold large heaps; OpenClaw's compaction may stay
   provider-side/server-side while the **persisted** transcript keeps growing.
   This also explains the 9–10 s SQLite `sessions.history.read`.
2. **In-process restarts + worker-thread accumulation.** Inside a container the
   gateway never tears the process down on a restart, so the per-worker V8 heaps
   are never reclaimed — memory only returns to baseline on a **hard** restart
   (`killGateway` + reprovision). `workerCount=8` → 18 `WorkerThread` is the
   accumulation signature.

> **Not yet proven:** the report could not snapshot the worker isolates
> (`linux-node`-style per-worker inspector needed). The heap location is inferred
> from the 256 KB mapping analysis. Phase 1 of the plan confirms it before
> committing to the fixes.

## What our layer controls

| Lever                      | Location                                                                           | Effect on the problem                                                                             |
| -------------------------- | ---------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- |
| Session compaction config  | `packages/moltlazy/src/agents/defaults.ts` (+ `types.ts`)                          | **Root cause #1** — bound the persisted transcript                                                |
| `OPENCLAW_NO_RESPAWN=1`    | `apps/fouria/Dockerfile`                                                           | **Explicit intent only** — see the #463 decision                                                  |
| `--verbose`                | `start-openclaw.sh` `launch_gateway_process`; `process.ts` `buildGatewayRunScript` | Log-noise only (coupled to boot-log legibility)                                                   |
| Gateway worker-thread pool | (upstream OpenClaw config — to confirm)                                            | **Root cause #2** — 18 vs 8 workers                                                               |
| Health probe (RSS/threads) | `apps/fouria/src/gateway/memory.ts`, `routes/public.ts`, `metrics.ts`              | **Fixed** — throttled `/proc` sample on the health snapshot + a `gateway-resources` AE row        |
| Soft-reload `SIGUSR2`      | `apps/fouria/src/gateway/process.ts` `buildSoftReloadScript`                       | **Fixed** — was `SIGUSR1` (opened the inspector) and signalled the launcher instead of the server |

## Plan

### Phase 0 — immediate containment (no code)

Hard-restart the dev instance (`POST /debug/restart-openclaw` → `killGateway` +
reprovision) to drop RSS to baseline and tear down the accumulated worker
threads. **Containment only** — confirms recoverability, buys time.

### Phase 1 — confirm the diagnosis (low risk, high value)

> **Landed.** `apps/fouria/src/gateway/memory.ts` reads `VmRSS` + `Threads` from
> `/proc/<pid>/status` for the gateway server in ONE shell exec, throttled to
> `GATEWAY_RESOURCE_SAMPLE_INTERVAL_MS` (60s) so the frequent health poll does not
> double its Sandbox RPC volume. The sample is attached to the health snapshot as
> `gateway: { pid, rssBytes, threads }` (only while the gateway is `running`, so a
> sleeping tenant is never woken) and written as a `gateway-resources` Analytics
> Engine row (`double1` = RSS bytes, `double2` = threads). This removes the need
> to SSH into a container to answer "was it memory?".

The worker-heap growth was inferred, not proven. Before coding against it:

1. Add a cheap read of the gateway process `VmRSS` (`/proc/<pid>/status`) and its
   thread count (`/proc/<pid>/task` or `ps -p <pid> -o rss,nlwp`) to the health
   snapshot / a new metric (`src/metrics.ts`, Analytics Engine), so the drift is
   visible over time.
2. Distinguish a **monotonic leak** (threads/RSS unbounded) from a **transient
   peak** (one large session that later compacts).
3. ~~Verify `SIGUSR1` semantics on the pinned OpenClaw version~~ **Done.** `SIGUSR1`
   opens the inspector and does not reload; the soft-reload signal is now
   `SIGUSR2` (see "Side observation" above).

### Phase 2 — root-cause fixes

**2.1 Bound the session transcript (highest impact).** Add compaction config to
moltlazy so the persisted transcript cannot grow unbounded even when the model
context is managed provider-side (OpenClaw key:
`agents.defaults.compaction.maxActiveTranscriptBytes`, plus `keepRecentTokens`):

```jsonc
agents: { defaults: { compaction: {
  maxActiveTranscriptBytes: <threshold>,
  keepRecentTokens: 20000,
} } }
```

> OpenClaw 2026.10.5 retired `compaction.enabled` (embedded auto-compaction is no
> longer toggleable) and `compaction.truncateAfterCompaction`; `openclaw config
> validate` rejects both keys. Byte-triggered compaction now opts in via
> `maxActiveTranscriptBytes` alone.

- Extend `MoltLazyAgentDefaultsConfig` (`packages/moltlazy/types.ts`) to `Pick`
  `'compaction'`; add `compaction` to `AGENT_DEFAULTS`.
- TDD in `packages/moltlazy/tests/agents.test.ts`; verify exact keys/values
  against `docs.openclaw.ai` before finalizing.
- This also relieves the SQLite `sessions.history.read` cost (symptom).

**2.2 Contain worker-thread accumulation.** Re-evaluate
`OPENCLAW_NO_RESPAWN=1`: **decision (issue #463) — keep it.** Removing the flag
does not enable process-level restarts inside the Cloudflare Sandbox container.
`resolveGatewayRestartDecision()` (`dist/cli/gateway-lifecycle.runtime.js`, checked
on 2026.10.4 and 2026.10.5) returns `mode: "disabled"` both for
`OPENCLAW_NO_RESPAWN` (`reason: "no-respawn"`) **and**, with no supervisor marker,
for any container (`isContainerEnvironment()` → `reason: "unmanaged"`, detail
`container: use in-process restart to keep PID 1 alive`). A restart therefore
never re-spawns the process with or without the flag, so it cannot reclaim the
worker-thread heaps. The flag stays as explicit intent; the Dockerfile comment and
the operator docs were corrected to stop claiming it is what makes reloads
in-process. The remaining lever is the Phase 2.4 watchdog (a hard
`killGateway` + reprovision is the only thing that actually returns RSS to
baseline). Also confirm whether OpenClaw exposes the worker pool size
(`workerCount`) and cap it via moltlazy if configurable.

**2.3 Reconsider `--verbose` (not "remove blindly").** It does not fix RAM; it
only lowers log rate/IO. Removing it risks the boot-log legibility provided by
`start_boot_log_stream`, which tails the gateway's `--verbose` stdout. Options:
drop to non-verbose **after** readiness, or gate it behind a flag.

**2.4 Memory watchdog (materializes "memory isn't reclaimed automatically").**
With the Phase 1 metric: when gateway `VmRSS` exceeds a threshold, replace the
gateway. Gate to **dev** first (non-billable, low risk); production only after
validation.

> **Landed** as `maybeRestartGatewayForMemory` (`src/gateway/memory.ts`), wired
> into the health route/stream. Design decisions that differ from the sketch
> above:
>
> - **Threshold is the container limit, not OpenClaw's 1.5 GiB warning.**
>   `GATEWAY_RSS_RESTART_BYTES` is 3 GiB of the 4 GiB container. An observed
>   healthy dev gateway sits at ~2.1 GiB (peak ~2.6 GiB), so a warning-level
>   threshold would restart it in a loop — and it would make a permanently
>   firing `rss_threshold` diagnostic look like an action item.
> - **No soft-reload step.** The measurement above shows an in-process `SIGUSR2`
>   restart does **not** reclaim RSS (1.14 → 1.38 → 1.18 GiB), so a reload first
>   would only add a failed attempt before the real fix.
> - **`killGateway` + reprovision, `skipRestore`.** The filesystem is untouched
>   by a kill, so re-entering the R2 restore barrier would deadlock against the
>   backup lock for no benefit.
> - **Cooldown** (`GATEWAY_MEMORY_WATCHDOG_COOLDOWN_MS`, 30 min) is claimed
>   _before_ acting, so concurrent health polls cannot start two replacements.
> - The watchdog runs **detached** from the health response (a replacement takes
>   minutes and must never delay or fail a health probe).

### Phase 3 — SQLite (only if 2.1 is insufficient)

`sessions.history.read` / `canonical-validation` slowness is downstream of the
oversized transcript. 2.1 should relieve it. WAL/checkpoint tuning is
OpenClaw-internal and only pursued if 2.1 does not suffice.

## Remaining follow-up

Tracked as a milestone 11 issue (see below). Items:

1. Phase 1 observability: RSS + thread count in the health snapshot / Analytics
   Engine; confirm monotonic leak vs. transient peak. **Done** — `gateway:
   { pid, rssBytes, threads }` on `/api/health` + the health stream, and a
   `gateway-resources` AE row, sampled at most once a minute.
2. ~~Verify `SIGUSR1` reload semantics~~ **Done** — `SIGUSR1` opens the inspector;
   the soft reload now uses `SIGUSR2`, targets the gateway server's exact `comm`
   (`openclaw-gatewa`) rather than the launcher, and the E2E asserts a new
   `http server listening` line plus `inspector-closed`.
3. Phase 2.1 compaction config in moltlazy (with TDD + docs verification).
   **Done** (2026.10.5 schema: `maxActiveTranscriptBytes` + `keepRecentTokens`;
   `enabled` retired).
4. Phase 2.2 `OPENCLAW_NO_RESPAWN` decision + worker-pool cap investigation.
   **Decided: keep** (see below); worker-pool cap still open.
5. Phase 2.3 `--verbose` scoping (preserve boot-log legibility).
6. Phase 2.4 memory watchdog (dev-gated). **Done** — `maybeRestartGatewayForMemory`,
   dev-gated, 3 GiB threshold, 30-minute cooldown.
7. Re-measure RSS/thread drift after 2.1–2.4 on a dev instance.

## Decision record — `OPENCLAW_NO_RESPAWN` ([#463](https://github.com/0xCAB0/fouria/issues/463))

**Decision: keep `OPENCLAW_NO_RESPAWN=1`.**

The issue proposed removing the flag so the gateway could perform process-level
restarts and reclaim memory. That premise does not hold on the pinned OpenClaw.
`resolveGatewayRestartDecision()` returns `mode: "disabled"` when
`OPENCLAW_NO_RESPAWN` is truthy (`reason: "no-respawn"`) **and** — absent a
supervisor marker — whenever `isContainerEnvironment()` is true
(`reason: "unmanaged"`, detail `container: use in-process restart to keep PID 1
alive`). Verified on both `2026.10.4` and `2026.10.5`.

Consequences:

- Removing the flag changes **nothing** inside the Cloudflare Sandbox container:
  restarts stay in-process, the PID is preserved, and the worker-thread V8 heaps
  are not reclaimed.
- The flag is therefore **explicit intent**, not a memory lever. Its Dockerfile
  comment and the `OPERATOR-GUIDE` note were corrected (they claimed the flag was
  what made reloads in-process).
- The only mechanism that returns RSS to baseline is the **hard** restart
  (`POST /debug/restart-openclaw` → `killGateway` + reprovision). Automating it
  is Phase 2.4, the remaining lever for #463/#458.

Separately, this investigation found and fixed two soft-reload defects: the Worker
sent `SIGUSR1` (which starts Node's inspector on `127.0.0.1:9229` and does **not**
reload) instead of the documented `SIGUSR2`, and it signalled the
`openclaw gateway run` **launcher** rather than the gateway **server** — killing
the launcher and orphaning the server (PPID 1, no restart) while still reporting
`reloaded: true`. Guards: `buildSoftReloadScript` unit tests (a "never sends
SIGUSR1" regression and the exact `comm` target), and the
`n_gateway_soft_reload.txt` E2E scenario, which requires a stable
`/api/status.processId` **and** an increased `http server listening` count across
the reload, plus `inspector-closed` via `/debug/cli`.

Measurement of a real in-process restart (local worker, OpenClaw 2026.10.5):
~3.3 s to listening, ~12–14 s to full readiness, server PID changes while the
supervisor `processId` stays stable; RSS 1,142,368 KiB → 1,376,380 KiB (peak,
+22 s) → 1,182,244 KiB (+49 s), threads 23 → 22. So the in-process restart
**does not reclaim memory** — reinforcing the keep decision and leaving the
hard-restart watchdog (Phase 2.4) as the only RSS lever.

## Acceptance criteria

- A dev instance's gateway RSS is observable (RSS + thread count) from the Worker
  without an ad-hoc `/proc` inspection. **Done** — `gateway: { pid, rssBytes,
  threads }` on `/api/health`, `/api/health-snapshot` and `/api/health-ws`, plus a
  `gateway-resources` Analytics Engine row.
- The `fouria-builder` session transcript is bounded by config; compaction fires
  locally before the persisted transcript grows unbounded. **Done** (2.1).
- RSS returns toward baseline automatically (watchdog) instead of only on a
  manual hard restart. **Done, dev-gated** — `maybeRestartGatewayForMemory`
  replaces the gateway at 3 GiB with a 30-minute cooldown.
- ~~The soft-reload path is confirmed to reload (not open an inspector), or fixed.~~
  **Done** — soft reload uses `SIGUSR2`, targets the gateway server (not the
  launcher), and the E2E requires a new `http server listening` line as well as a
  stable `processId`.

## Out of scope

- SandboxV1 migration (`docs/plans/sandbox-v1-scaffold.md`).
- R2 backup/restore paths.
- Upstream OpenClaw internals beyond what config knobs we can set from moltlazy
  (no patching OpenClaw source).
- Changing the container instance size / Cloudflare sandbox memory limits.
