Sandbox memory growth: worker-thread heaps + oversized sessions
On this page
Status: in progress — diagnosis captured from a dev instance; investigation and remediation tracked in milestone 11 (v2026.10 - Perfomance enhacements). This plan bounds the OpenClaw gateway’s resident memory (RSS) growth inside the Cloudflare Sandbox container, from the Worker/container layer we own, so a dev instance does not drift toward the container/sandbox memory limit without an automatic recovery.
Landed so far: bounded transcript compaction (2.1), the soft-reload fixes (
SIGUSR1→SIGUSR2, and the signal now reaches the gateway server instead of theopenclaw gateway runlauncher), theOPENCLAW_NO_RESPAWNkeep decision (#463 — see the decision record), RSS/thread observability (Phase 1) and the dev-gated watchdog (2.4) — both inapps/fouria/src/gateway/memory.ts, surfaced on/api/healthand the health stream. Remaining: re-measuring the drift on a dev instance with the watchdog enabled, and the worker-pool cap investigation (2.2).
Background
A dev instance (fouria / OpenClaw gateway in a Cloudflare Sandbox container)
was observed at RSS ~1.85 GiB (peak 2.36 GiB), swap 0, with OpenClaw’s own
memory diagnostic already firing:
subsystem=gateway/diagnostics/memory reason=rss_threshold rss=1.84 GiB
heap=339.9 MiB threshold=1.5 GiB thresholdRatio=122.7%
A V8 heap snapshot of the main isolate (/tmp/gw310-main.heapsnapshot,
290 MB) showed the main isolate heap is not the driver:
| Metric (main isolate) | Value |
|---|---|
heapUsed | ~340 MB / limit 2.35 GiB |
total self_size | 306 MiB |
top retained: JSArrayBufferData | 35.6 MiB |
empty strings "" | 25 MiB (186k) |
(object properties) | 17.7 MiB |
system/BytecodeArray | 9.4 MiB |
No large JS retainer, no JS leak in the main isolate. The log file is tiny
(316 KB → 1.5 MB); --verbose only raises log frequency, not memory.
Evidence
Measured from /proc/<pid> on the live gateway process:
| Observation | Value |
|---|---|
[anon] | 1,665 MB (Pss_Anon 1.82 GiB) |
glibc [heap] | 99 MB |
| node binary (RSS) | 66 MB |
| anonymous mappings of ~256 KB | 4,885 ≈ 1.2 GiB |
| total threads | 31 (18 WorkerThread + 4 V8Worker + 4 libuv-worker) |
OpenClaw worker startup state log | workerCount=8 |
The ~4,885 anonymous mappings of ~256 KB match V8’s heap page granularity
(256 KB), i.e. per-isolate heap pages. The main isolate only accounts for
~412 MB of committed heap, so the remaining ~1.2 GiB lives in the heaps of the
in-process worker_threads — each worker_thread is its own V8 isolate with
its own heap, invisible to the main isolate’s getHeapStatistics().
Corroborating signals
workerCount=8at startup vs 18WorkerThreadobserved → thread/heap accumulation (workers not being reclaimed over the instance’s life).- SQLite contention:
sessions.history.read/canonical-validationtaking 9–10 s. - Session
fouria-builder:mainwith 407,708 tokens in compaction preflight — a huge persisted transcript.
Side observation (soft-reload vs. inspector) — resolved
To take the snapshot, SIGUSR1 was sent to the gateway PID; it opened the Node
inspector on 127.0.0.1:9229 (localhost-only; closes on gateway restart). This
confirmed a real defect: OpenClaw’s documented restart signal is SIGUSR2
(SIGUSR1 only starts the inspector), while our softReloadGateway
(apps/fouria/src/gateway/process.ts) sent SIGUSR1 expecting an in-process
reload — so the soft reload was a silent no-op that left a debug socket open.
Fixed. buildSoftReloadScript() now sends kill -USR2, and
test/e2e/n_gateway_soft_reload.txt asserts the inspector stays closed
(inspector-closed) after a soft reload. SIGUSR2 is authorized by default:
isRestartEnabled(config) is commands.restart !== false (default on) and
server-start calls setGatewayRestartPolicy({ allowExternal: … }), so an
external SIGUSR2 schedules the in-process restart rather than logging
SIGUSR2 restart ignored (not authorized; commands.restart=false).
Second, independent defect (found while measuring the fix): the signal
targeted the launcher, not the server. buildSoftReloadScript() signalled
pgrep -f "[o]penclaw gateway run", which matches the launcher process
(node /usr/local/bin/openclaw gateway run …) that supervises the real server.
kill -USR2 <launcher> makes the launcher exit without restarting the
server: the server is orphaned (PPID 1), readiness never dips, no new
http server listening line appears — yet the endpoint still reports
reloaded: true, so a broken reload looked successful. Measured on a live local
worker (OpenClaw 2026.10.5): launcher-targeted reload ack_ms=30, readiness
123/123 at 200 true, launcher gone; a direct kill -USR2 <server> instead
restarted the server (http server listening +1), dipped readiness at +120 ms
(503) and recovered at ~3.3 s to listening / ~12–14 s to full readiness.
buildSoftReloadScript() now targets the server’s exact comm
(pgrep -x 'openclaw-gatewa', Linux’s 15-char truncation, plus a defensive
openclaw-gateway for kernels that keep longer names), deduped with sort -u,
skipping zombies. Because /api/status.processId is the Worker-tracked
supervisor (which survives an in-process reload), the E2E
n_gateway_soft_reload.txt now asserts both the same processId and an
increased http server listening count — processId stability alone passed in
the broken case.
Root cause (working hypothesis)
Two coupled causes, both reachable from our layer:
- Oversized session transcripts.
fouria-builder:mainat 407k tokens keeps a large persisted transcript (JSONL + SQLite history), so theworker_threads that serve it hold large heaps; OpenClaw’s compaction may stay provider-side/server-side while the persisted transcript keeps growing. This also explains the 9–10 s SQLitesessions.history.read. - In-process restarts + worker-thread accumulation. Inside a container the
gateway never tears the process down on a restart, so the per-worker V8 heaps
are never reclaimed — memory only returns to baseline on a hard restart
(
killGateway+ reprovision).workerCount=8→ 18WorkerThreadis the accumulation signature.
Not yet proven: the report could not snapshot the worker isolates (
linux-node-style per-worker inspector needed). The heap location is inferred from the 256 KB mapping analysis. Phase 1 of the plan confirms it before committing to the fixes.
What our layer controls
| Lever | Location | Effect on the problem |
|---|---|---|
| Session compaction config | packages/moltlazy/src/agents/defaults.ts (+ types.ts) | Root cause #1 — bound the persisted transcript |
OPENCLAW_NO_RESPAWN=1 | apps/fouria/Dockerfile | Explicit intent only — see the #463 decision |
--verbose | start-openclaw.sh launch_gateway_process; process.ts buildGatewayRunScript | Log-noise only (coupled to boot-log legibility) |
| Gateway worker-thread pool | (upstream OpenClaw config — to confirm) | Root cause #2 — 18 vs 8 workers |
| Health probe (RSS/threads) | apps/fouria/src/gateway/memory.ts, routes/public.ts, metrics.ts | Fixed — throttled /proc sample on the health snapshot + a gateway-resources AE row |
Soft-reload SIGUSR2 | apps/fouria/src/gateway/process.ts buildSoftReloadScript | Fixed — was SIGUSR1 (opened the inspector) and signalled the launcher instead of the server |
Plan
Phase 0 — immediate containment (no code)
Hard-restart the dev instance (POST /debug/restart-openclaw → killGateway +
reprovision) to drop RSS to baseline and tear down the accumulated worker
threads. Containment only — confirms recoverability, buys time.
Phase 1 — confirm the diagnosis (low risk, high value)
Landed.
apps/fouria/src/gateway/memory.tsreadsVmRSS+Threadsfrom/proc/<pid>/statusfor the gateway server in ONE shell exec, throttled toGATEWAY_RESOURCE_SAMPLE_INTERVAL_MS(60s) so the frequent health poll does not double its Sandbox RPC volume. The sample is attached to the health snapshot asgateway: { pid, rssBytes, threads }(only while the gateway isrunning, so a sleeping tenant is never woken) and written as agateway-resourcesAnalytics Engine row (double1= RSS bytes,double2= threads). This removes the need to SSH into a container to answer “was it memory?”.
The worker-heap growth was inferred, not proven. Before coding against it:
- Add a cheap read of the gateway process
VmRSS(/proc/<pid>/status) and its thread count (/proc/<pid>/taskorps -p <pid> -o rss,nlwp) to the health snapshot / a new metric (src/metrics.ts, Analytics Engine), so the drift is visible over time. - Distinguish a monotonic leak (threads/RSS unbounded) from a transient peak (one large session that later compacts).
VerifyDone.SIGUSR1semantics on the pinned OpenClaw versionSIGUSR1opens the inspector and does not reload; the soft-reload signal is nowSIGUSR2(see “Side observation” above).
Phase 2 — root-cause fixes
2.1 Bound the session transcript (highest impact). Add compaction config to
moltlazy so the persisted transcript cannot grow unbounded even when the model
context is managed provider-side (OpenClaw key:
agents.defaults.compaction.maxActiveTranscriptBytes, plus keepRecentTokens):
agents: { defaults: { compaction: {
maxActiveTranscriptBytes: <threshold>,
keepRecentTokens: 20000,
} } }
OpenClaw 2026.10.5 retired
compaction.enabled(embedded auto-compaction is no longer toggleable) andcompaction.truncateAfterCompaction;openclaw config validaterejects both keys. Byte-triggered compaction now opts in viamaxActiveTranscriptBytesalone.
- Extend
MoltLazyAgentDefaultsConfig(packages/moltlazy/types.ts) toPick'compaction'; addcompactiontoAGENT_DEFAULTS. - TDD in
packages/moltlazy/tests/agents.test.ts; verify exact keys/values againstdocs.openclaw.aibefore finalizing. - This also relieves the SQLite
sessions.history.readcost (symptom).
2.2 Contain worker-thread accumulation. Re-evaluate
OPENCLAW_NO_RESPAWN=1: decision (issue #463) — keep it. Removing the flag
does not enable process-level restarts inside the Cloudflare Sandbox container.
resolveGatewayRestartDecision() (dist/cli/gateway-lifecycle.runtime.js, checked
on 2026.10.4 and 2026.10.5) returns mode: "disabled" both for
OPENCLAW_NO_RESPAWN (reason: "no-respawn") and, with no supervisor marker,
for any container (isContainerEnvironment() → reason: "unmanaged", detail
container: use in-process restart to keep PID 1 alive). A restart therefore
never re-spawns the process with or without the flag, so it cannot reclaim the
worker-thread heaps. The flag stays as explicit intent; the Dockerfile comment and
the operator docs were corrected to stop claiming it is what makes reloads
in-process. The remaining lever is the Phase 2.4 watchdog (a hard
killGateway + reprovision is the only thing that actually returns RSS to
baseline). Also confirm whether OpenClaw exposes the worker pool size
(workerCount) and cap it via moltlazy if configurable.
2.3 Reconsider --verbose (not “remove blindly”). It does not fix RAM; it
only lowers log rate/IO. Removing it risks the boot-log legibility provided by
start_boot_log_stream, which tails the gateway’s --verbose stdout. Options:
drop to non-verbose after readiness, or gate it behind a flag.
2.4 Memory watchdog (materializes “memory isn’t reclaimed automatically”).
With the Phase 1 metric: when gateway VmRSS exceeds a threshold, replace the
gateway. Gate to dev first (non-billable, low risk); production only after
validation.
Landed as
maybeRestartGatewayForMemory(src/gateway/memory.ts), wired into the health route/stream. Design decisions that differ from the sketch above:
- Threshold is the container limit, not OpenClaw’s 1.5 GiB warning.
GATEWAY_RSS_RESTART_BYTESis 3 GiB of the 4 GiB container. An observed healthy dev gateway sits at ~2.1 GiB (peak ~2.6 GiB), so a warning-level threshold would restart it in a loop — and it would make a permanently firingrss_thresholddiagnostic look like an action item.- No soft-reload step. The measurement above shows an in-process
SIGUSR2restart does not reclaim RSS (1.14 → 1.38 → 1.18 GiB), so a reload first would only add a failed attempt before the real fix.killGateway+ reprovision,skipRestore. The filesystem is untouched by a kill, so re-entering the R2 restore barrier would deadlock against the backup lock for no benefit.- Cooldown (
GATEWAY_MEMORY_WATCHDOG_COOLDOWN_MS, 30 min) is claimed before acting, so concurrent health polls cannot start two replacements.- The watchdog runs detached from the health response (a replacement takes minutes and must never delay or fail a health probe).
Phase 3 — SQLite (only if 2.1 is insufficient)
sessions.history.read / canonical-validation slowness is downstream of the
oversized transcript. 2.1 should relieve it. WAL/checkpoint tuning is
OpenClaw-internal and only pursued if 2.1 does not suffice.
Remaining follow-up
Tracked as a milestone 11 issue (see below). Items:
- Phase 1 observability: RSS + thread count in the health snapshot / Analytics
Engine; confirm monotonic leak vs. transient peak. Done —
gateway: { pid, rssBytes, threads }on/api/health+ the health stream, and agateway-resourcesAE row, sampled at most once a minute. VerifyDone —SIGUSR1reload semanticsSIGUSR1opens the inspector; the soft reload now usesSIGUSR2, targets the gateway server’s exactcomm(openclaw-gatewa) rather than the launcher, and the E2E asserts a newhttp server listeningline plusinspector-closed.- Phase 2.1 compaction config in moltlazy (with TDD + docs verification).
Done (2026.10.5 schema:
maxActiveTranscriptBytes+keepRecentTokens;enabledretired). - Phase 2.2
OPENCLAW_NO_RESPAWNdecision + worker-pool cap investigation. Decided: keep (see below); worker-pool cap still open. - Phase 2.3
--verbosescoping (preserve boot-log legibility). - Phase 2.4 memory watchdog (dev-gated). Done —
maybeRestartGatewayForMemory, dev-gated, 3 GiB threshold, 30-minute cooldown. - Re-measure RSS/thread drift after 2.1–2.4 on a dev instance.
Decision record — OPENCLAW_NO_RESPAWN (#463)
Decision: keep OPENCLAW_NO_RESPAWN=1.
The issue proposed removing the flag so the gateway could perform process-level
restarts and reclaim memory. That premise does not hold on the pinned OpenClaw.
resolveGatewayRestartDecision() returns mode: "disabled" when
OPENCLAW_NO_RESPAWN is truthy (reason: "no-respawn") and — absent a
supervisor marker — whenever isContainerEnvironment() is true
(reason: "unmanaged", detail container: use in-process restart to keep PID 1 alive). Verified on both 2026.10.4 and 2026.10.5.
Consequences:
- Removing the flag changes nothing inside the Cloudflare Sandbox container: restarts stay in-process, the PID is preserved, and the worker-thread V8 heaps are not reclaimed.
- The flag is therefore explicit intent, not a memory lever. Its Dockerfile
comment and the
OPERATOR-GUIDEnote were corrected (they claimed the flag was what made reloads in-process). - The only mechanism that returns RSS to baseline is the hard restart
(
POST /debug/restart-openclaw→killGateway+ reprovision). Automating it is Phase 2.4, the remaining lever for #463/#458.
Separately, this investigation found and fixed two soft-reload defects: the Worker
sent SIGUSR1 (which starts Node’s inspector on 127.0.0.1:9229 and does not
reload) instead of the documented SIGUSR2, and it signalled the
openclaw gateway run launcher rather than the gateway server — killing
the launcher and orphaning the server (PPID 1, no restart) while still reporting
reloaded: true. Guards: buildSoftReloadScript unit tests (a “never sends
SIGUSR1” regression and the exact comm target), and the
n_gateway_soft_reload.txt E2E scenario, which requires a stable
/api/status.processId and an increased http server listening count across
the reload, plus inspector-closed via /debug/cli.
Measurement of a real in-process restart (local worker, OpenClaw 2026.10.5):
~3.3 s to listening, ~12–14 s to full readiness, server PID changes while the
supervisor processId stays stable; RSS 1,142,368 KiB → 1,376,380 KiB (peak,
+22 s) → 1,182,244 KiB (+49 s), threads 23 → 22. So the in-process restart
does not reclaim memory — reinforcing the keep decision and leaving the
hard-restart watchdog (Phase 2.4) as the only RSS lever.
Acceptance criteria
- A dev instance’s gateway RSS is observable (RSS + thread count) from the Worker
without an ad-hoc
/procinspection. Done —gateway: { pid, rssBytes, threads }on/api/health,/api/health-snapshotand/api/health-ws, plus agateway-resourcesAnalytics Engine row. - The
fouria-buildersession transcript is bounded by config; compaction fires locally before the persisted transcript grows unbounded. Done (2.1). - RSS returns toward baseline automatically (watchdog) instead of only on a
manual hard restart. Done, dev-gated —
maybeRestartGatewayForMemoryreplaces the gateway at 3 GiB with a 30-minute cooldown. The soft-reload path is confirmed to reload (not open an inspector), or fixed.Done — soft reload usesSIGUSR2, targets the gateway server (not the launcher), and the E2E requires a newhttp server listeningline as well as a stableprocessId.
Out of scope
- SandboxV1 migration (
docs/plans/sandbox-v1-scaffold.md). - R2 backup/restore paths.
- Upstream OpenClaw internals beyond what config knobs we can set from moltlazy (no patching OpenClaw source).
- Changing the container instance size / Cloudflare sandbox memory limits.