openclaw-openclaw/docs/nodes/talk.md

summary, read_when, title

summary	read_when	title
Talk mode: continuous speech conversations across local STT/TTS and realtime voice	Implementing Talk mode on macOS/iOS/Android Changing voice/TTS/interrupt behavior	Talk mode

Talk mode covers five runtime shapes:

• Native macOS/iOS/Android Talk: local speech recognition, Gateway chat, and talk.speak TTS. Nodes advertise the talk capability and declare which talk.* commands they support.
• iOS Talk (realtime): client-owned WebRTC for OpenAI realtime configs that select webrtc transport or omit transport. Explicit gateway-relay, provider-websocket, and non-OpenAI realtime configs stay on the Gateway-owned relay; non-realtime configs use the native speech loop.
• Browser Talk: talk.client.create for client-owned webrtc/provider-websocket sessions, or talk.session.create for Gateway-owned gateway-relay sessions. managed-room is reserved for Gateway handoff and walkie-talkie rooms.
• Android Talk (realtime): opt in with talk.realtime.mode: "realtime" and talk.realtime.transport: "gateway-relay". Otherwise Android stays on native speech recognition, Gateway chat, and talk.speak.
• Transcription-only clients: talk.session.create({ mode: "transcription", transport: "gateway-relay", brain: "none" }), then talk.session.appendAudio, talk.session.cancelTurn, and talk.session.close for captions/dictation without an assistant voice response. One-shot uploaded voice notes still use the media understanding audio path.

Native Talk is a continuous loop: listen for speech, send the transcript to the model through the active session, wait for the response, then speak it via the configured Talk provider (talk.speak).

Client-owned realtime Talk forwards provider tool calls through talk.client.toolCall instead of calling chat.send directly. While a realtime consult is active, clients can call talk.client.steer or talk.session.steer to classify spoken input as status, steer, cancel, or followup. Accepted steering queues into the active embedded run; rejected steering returns a reason such as no_active_run, not_streaming, or compacting.

Transcription-only Talk emits the same Talk event envelope as realtime and STT/TTS sessions, but uses mode: "transcription" and brain: "none". All Talk sessions broadcast events on the talk.event channel; clients subscribe to it for partial/final transcript updates (transcript.delta/transcript.done) and other session telemetry.

Behavior (macOS)

• Always-on overlay while Talk mode is enabled.
• Listening → Thinking → Speaking phase transitions.
• On a short pause (silence window), the current transcript is sent.
• Replies are written to WebChat (same as typing).
• Interrupt on speech (default on): if the user talks while the assistant is speaking, playback stops and the interruption timestamp is noted for the next prompt.

Voice directives in replies

The assistant can prefix a reply with a single JSON line to control voice:

{ "voice": "<voice-id>", "once": true }

Rules:

• First non-empty line only; the JSON line is stripped before TTS playback.
• Unknown keys are ignored.
• once: true applies to the current reply only; without it, the voice becomes the new Talk mode default.

Supported keys: voice / voice_id / voiceId, model / model_id / modelId, speed, rate (WPM), stability, similarity, style, speakerBoost, seed, normalize, lang, output_format, latency_tier, once.

Config (~/.openclaw/openclaw.json)

{
  talk: {
    provider: "elevenlabs",
    providers: {
      elevenlabs: {
        voiceId: "elevenlabs_voice_id",
        modelId: "eleven_v3",
        outputFormat: "mp3_44100_128",
        apiKey: "elevenlabs_api_key",
      },
      mlx: {
        modelId: "mlx-community/Soprano-80M-bf16",
      },
      system: {},
    },
    speechLocale: "ru-RU",
    silenceTimeoutMs: 1500,
    interruptOnSpeech: true,
    realtime: {
      provider: "openai",
      providers: {
        openai: {
          apiKey: "openai_api_key",
          model: "gpt-realtime-2",
          speakerVoice: "cedar",
        },
      },
      instructions: "Speak warmly and keep answers brief.",
      mode: "realtime",
      transport: "webrtc",
      brain: "agent-consult",
    },
  },
}

Key	Default	Notes
provider	-	Active Talk TTS provider. Use elevenlabs, mlx, or system for macOS-local playback paths.
providers.<id>.voiceId	-	ElevenLabs falls back to ELEVENLABS_VOICE_ID / SAG_VOICE_ID, or the first available voice with an API key.
providers.elevenlabs.modelId	eleven_v3
providers.mlx.modelId	mlx-community/Soprano-80M-bf16
providers.elevenlabs.apiKey	-	Falls back to ELEVENLABS_API_KEY (or gateway shell profile if available).
speechLocale	device default	BCP 47 locale id for on-device Talk speech recognition on iOS/macOS.
silenceTimeoutMs	700 ms macOS/Android, 900 ms iOS	Pause window before Talk sends the transcript.
interruptOnSpeech	true
outputFormat	pcm_44100 macOS/iOS, pcm_24000 Android	Set mp3_* to force MP3 streaming.
consultThinkingLevel	unset	Thinking level override for the agent run behind realtime openclaw_agent_consult calls.
consultFastMode	unset	Fast-mode override for realtime openclaw_agent_consult calls.
realtime.provider	-	openai for WebRTC, google for provider WebSocket, or a bridge-only provider through Gateway relay.
realtime.providers.<id>	-	Provider-owned realtime config. Browsers receive only ephemeral/constrained session credentials, never a standard API key.
realtime.providers.openai.speakerVoice	alloy	Built-in OpenAI Realtime voice id (the older voice key still works but is deprecated). Current gpt-realtime-2 voices: alloy, ash, ballad, cedar, coral, echo, marin, sage, shimmer, verse; marin and cedar are recommended for best quality.
realtime.transport	-	webrtc: client-owned OpenAI WebRTC on iOS and in the browser. provider-websocket: browser-owned, stays on Gateway relay on iOS. gateway-relay: keeps provider audio on the Gateway; Android uses realtime only with this transport.
realtime.brain	-	agent-consult routes realtime tool calls through Gateway policy; direct-tools is legacy direct-tool compatibility; none is for transcription/external orchestration.
realtime.consultRouting	-	provider-direct preserves the provider's direct reply when it skips openclaw_agent_consult; force-agent-consult routes finalized user transcripts through OpenClaw instead.
realtime.instructions	-	Appends provider-facing system instructions to OpenClaw's built-in realtime prompt (voice style/tone); the default openclaw_agent_consult guidance stays.

talk.catalog exposes canonical provider ids and registry aliases, each provider's valid modes/transports/brain strategies/realtime audio formats/capability flags, and the runtime-selected readiness result. First-party Talk clients should read that catalog instead of maintaining provider aliases locally; treat an older Gateway that omits group readiness as unverified rather than definitively unconfigured. Streaming transcription providers are discovered through talk.catalog.transcription; the current Gateway relay uses the Voice Call streaming provider config until a dedicated Talk transcription config surface ships.

macOS UI

• Menu bar toggle: Talk
• Config tab: Talk Mode group (voice id + interrupt toggle)
• Overlay: Listening (cloud pulses with mic level) → Thinking (sinking animation) → Speaking (radiating rings). Click the cloud to stop speaking, click X to exit Talk mode.

Android UI

• Voice tab toggle: Talk
• Manual Mic and Talk are mutually exclusive capture modes.
• Manual Mic and realtime Talk prefer a connected Bluetooth Classic or BLE headset microphone; if it disconnects, the app requests another headset input or falls back to the default microphone, restoring the default preference once capture stops.
• Manual Mic stops when the app leaves the foreground or the user leaves the Voice tab.
• Talk Mode keeps running until toggled off or the node disconnects, using Android's microphone foreground-service type while active.
• Android supports pcm_16000, pcm_22050, pcm_24000, and pcm_44100 output formats for low-latency AudioTrack streaming.

Notes

• Requires Speech + Microphone permissions.
• Native Talk uses the active Gateway session and only falls back to history polling when response events are unavailable.
• The gateway resolves Talk playback through talk.speak using the active Talk provider. Android falls back to local system TTS only when that RPC is unavailable.
• macOS local MLX playback uses the bundled openclaw-mlx-tts helper when present, or an executable on PATH. Set OPENCLAW_MLX_TTS_BIN to point at a custom helper binary during development.
• Voice directive value ranges (ElevenLabs): stability, similarity, and style accept 0..1; speed accepts 0.5..2; latency_tier accepts 0..4.

Related

• Voice wake
• Audio and voice notes
• Media understanding