Files
openclaw-openclaw/docs/help/debugging.md
Peter Steinberger f7d7148cf0 docs: rewrite published docs grounded in current source (#100142)
Source-grounded rewrite of 529 published docs pages with per-unit information-loss verification: 1,713 factual corrections cited to src/**, generated surfaces regenerated, frontmatter titles preserved for i18n, release notes pages untouched. All docs gates green.

Closes #100141
2026-07-05 00:32:47 -04:00

9.9 KiB

summary, read_when, title
summary read_when title
Debugging tools: watch mode, raw model streams, and tracing reasoning leakage
You need to inspect raw model output for reasoning leakage
You want to run the Gateway in watch mode while iterating
You need a repeatable debugging workflow
Debugging

Debugging helpers for streaming output, gateway iteration, and startup profiling.

Runtime debug overrides

/debug sets runtime-only config overrides (memory, not disk). Disabled by default; enable with commands.debug: true.

/debug show
/debug set messages.responsePrefix="[openclaw]"
/debug unset messages.responsePrefix
/debug reset

/debug reset clears all overrides and returns to the on-disk config.

Session trace output

/trace shows plugin-owned trace/debug lines for one session without enabling full verbose mode. Use it for plugin diagnostics such as Active Memory debug summaries; use /verbose for normal status/tool output.

/trace
/trace on
/trace off

Plugin lifecycle trace

Set OPENCLAW_PLUGIN_LIFECYCLE_TRACE=1 for a phase-by-phase breakdown of plugin metadata, discovery, registry, runtime mirror, config mutation, and refresh work. Writes to stderr, so JSON command output stays parseable.

OPENCLAW_PLUGIN_LIFECYCLE_TRACE=1 openclaw plugins install tokenjuice --force
[plugins:lifecycle] phase="config read" ms=6.83 status=ok command="install"
[plugins:lifecycle] phase="slot selection" ms=94.31 status=ok command="install" pluginId="tokenjuice"
[plugins:lifecycle] phase="registry refresh" ms=51.56 status=ok command="install" reason="source-changed"

Use this before reaching for a CPU profiler. From a source checkout, measure the built runtime with node dist/entry.js ... after pnpm build; pnpm openclaw ... also measures source-runner overhead.

CLI startup and command profiling

Checked-in startup benchmarks:

pnpm test:startup:bench:smoke
pnpm tsx scripts/bench-cli-startup.ts --preset real --case status --runs 3
pnpm tsx scripts/bench-cli-startup.ts --preset real --cpu-prof-dir .artifacts/cli-cpu

For one-off profiling through the normal source runner, set OPENCLAW_RUN_NODE_CPU_PROF_DIR:

OPENCLAW_RUN_NODE_CPU_PROF_DIR=.artifacts/cli-cpu pnpm openclaw status

The source runner adds Node CPU profile flags and writes a .cpuprofile for the command. Use this before adding temporary instrumentation to command code.

For startup stalls that look like synchronous filesystem or module-loader work, add Node's sync I/O trace flag through the source runner:

OPENCLAW_TRACE_SYNC_IO=1 pnpm openclaw gateway --force

pnpm gateway:watch leaves this flag disabled by default for the watched Gateway child; set OPENCLAW_TRACE_SYNC_IO=1 when you want sync I/O trace output in watch mode too.

Gateway watch mode

pnpm gateway:watch

By default this starts or restarts a tmux session named openclaw-gateway-watch-<profile> (for example openclaw-gateway-watch-main), with a port suffix such as openclaw-gateway-watch-dev-19001 added only when OPENCLAW_GATEWAY_PORT differs from the default port 18789. It auto-attaches from interactive terminals; non-interactive shells, CI, and agent exec calls stay detached and print attach instructions instead:

tmux attach -t openclaw-gateway-watch-main

The tmux pane runs the raw watcher:

node scripts/watch-node.mjs gateway --force

Foreground mode without tmux:

pnpm gateway:watch:raw
# or
OPENCLAW_GATEWAY_WATCH_TMUX=0 pnpm gateway:watch

Keep tmux management but disable auto-attach:

OPENCLAW_GATEWAY_WATCH_ATTACH=0 pnpm gateway:watch

Profile watched Gateway CPU time when debugging startup/runtime hotspots:

pnpm gateway:watch --benchmark

The watch wrapper consumes --benchmark before invoking the Gateway and writes one V8 .cpuprofile per Gateway child exit under .artifacts/gateway-watch-profiles/. Stop or restart the watched gateway to flush the current profile, then open it with Chrome DevTools or Speedscope:

npx speedscope .artifacts/gateway-watch-profiles/*.cpuprofile
  • --benchmark-dir <path>: write profiles somewhere else.
  • --benchmark-no-force: skip the default --force port cleanup and fail fast if the Gateway port is already in use.

Benchmark mode suppresses sync-I/O trace spam by default. Set OPENCLAW_TRACE_SYNC_IO=1 with --benchmark to get both CPU profiles and sync-I/O stack traces; in benchmark mode those trace blocks go to gateway-watch-output.log under the benchmark directory (filtered from the terminal pane), while normal Gateway logs stay visible.

The tmux wrapper carries common non-secret runtime selectors into the pane, including OPENCLAW_PROFILE, OPENCLAW_CONFIG_PATH, OPENCLAW_STATE_DIR, OPENCLAW_GATEWAY_PORT, and OPENCLAW_SKIP_CHANNELS. Put provider credentials in your normal profile/config, or use raw foreground mode for one-off ephemeral secrets.

If the watched Gateway exits during startup, the watcher runs openclaw doctor --fix --non-interactive once and restarts the Gateway child. Set OPENCLAW_GATEWAY_WATCH_AUTO_DOCTOR=0 to see the original startup failure without the dev-only repair pass.

The managed tmux pane defaults to colored Gateway logs; set FORCE_COLOR=0 when starting pnpm gateway:watch to disable ANSI output.

The watcher restarts on build-relevant files under src/, extension source files, extension package.json and openclaw.plugin.json metadata, tsconfig.json, package.json, and tsdown.config.ts. Extension metadata changes restart the gateway without forcing a rebuild; source and config changes still rebuild dist first.

Add gateway CLI flags after gateway:watch and they pass through on each restart. Re-running the same watch command respawns the named tmux pane; the raw watcher keeps a single-watcher lock so duplicate watcher parents are replaced instead of piling up.

Dev profile + dev gateway (--dev)

Two separate --dev flags:

  • Global --dev (profile): isolates state under ~/.openclaw-dev and defaults the gateway port to 19001 (derived ports shift with it).
  • gateway --dev: tells the Gateway to auto-create a default config + workspace when missing (and skip bootstrap).

Recommended flow (dev profile + dev bootstrap):

pnpm gateway:dev
OPENCLAW_PROFILE=dev openclaw tui

Without a global install, run the CLI via pnpm openclaw ....

What this does:

  1. Profile isolation (global --dev)

    • OPENCLAW_PROFILE=dev
    • OPENCLAW_STATE_DIR=~/.openclaw-dev
    • OPENCLAW_CONFIG_PATH=~/.openclaw-dev/openclaw.json
    • OPENCLAW_GATEWAY_PORT=19001 (browser/canvas ports shift accordingly)
  2. Dev bootstrap (gateway --dev)

    • Writes a minimal config if missing (gateway.mode=local, bind loopback).
    • Sets agents.defaults.workspace to the dev workspace and agents.defaults.skipBootstrap=true.
    • Seeds the workspace files if missing: AGENTS.md, SOUL.md, TOOLS.md, IDENTITY.md, USER.md.
    • Default identity: C3-PO (protocol droid).
    • pnpm gateway:dev also sets OPENCLAW_SKIP_CHANNELS=1 to skip channel providers.

Reset flow (fresh start):

pnpm gateway:dev:reset
`--dev` is a **global** profile flag and gets eaten by some runners. If you need to spell it out, use the env var form:
OPENCLAW_PROFILE=dev openclaw gateway --dev --reset

--reset wipes config, credentials, sessions, and the dev workspace (moved to trash, not deleted), then recreates the default dev setup.

If a non-dev gateway is already running (launchd or systemd), stop it first:
openclaw gateway stop

Raw stream logging

OpenClaw can log the raw assistant stream before any filtering/formatting. This is the best way to see whether reasoning is arriving as plain text deltas (or as separate thinking blocks).

Enable it via CLI:

pnpm gateway:watch --raw-stream

Optional path override:

pnpm gateway:watch --raw-stream --raw-stream-path ~/.openclaw/logs/raw-stream.jsonl

Equivalent env vars:

OPENCLAW_RAW_STREAM=1
OPENCLAW_RAW_STREAM_PATH=~/.openclaw/logs/raw-stream.jsonl

Default file: ~/.openclaw/logs/raw-stream.jsonl

Safety notes

  • Raw stream logs can include full prompts, tool output, and user data.
  • Keep logs local and delete them after debugging.
  • If you share logs, scrub secrets and PII first.

Debugging in VSCode

Source maps are required because the build hashes generated filenames. The included launch.json targets the Gateway service:

  1. Rebuild and Debug Gateway - deletes /dist and rebuilds with debugging enabled before starting the Gateway.
  2. Debug Gateway - debugs an existing build without touching /dist.

Setup

  1. Open Run and Debug (Activity Bar, or Ctrl+Shift+D).
  2. Select Rebuild and Debug Gateway and press Start Debugging.

To manage the build/debug cycle manually instead:

  1. Enable source maps in a terminal:
    • Linux/macOS: export OUTPUT_SOURCE_MAPS=1
    • Windows (PowerShell): $env:OUTPUT_SOURCE_MAPS="1"
    • Windows (CMD): set OUTPUT_SOURCE_MAPS=1
  2. Rebuild: pnpm clean:dist && pnpm build
  3. Select Debug Gateway and press Start Debugging.

Set breakpoints in src/ TypeScript files; the debugger maps them to compiled JavaScript via source maps.

Notes

  • Rebuild and Debug Gateway deletes /dist and runs a full pnpm build with source maps on every launch.
  • Debug Gateway can start/stop without affecting /dist, but you manage the build cycle in a separate terminal.
  • Edit launch.json args to debug other CLI subcommands.
  • To use the built CLI for other tasks (for example dashboard --no-open if your debug session spawns a new auth token), run it from another terminal: node ./openclaw.mjs or an alias like alias openclaw-build="node $(pwd)/openclaw.mjs".