Files
openclaw-openclaw/docs/platforms/android.md
Peter Steinberger 91f188301d docs: explain remote Android screen mirroring (#100398)
* docs(android): document remote screen mirroring

* docs: refresh documentation map
2026-07-05 10:55:03 -07:00

16 KiB

summary, read_when, title
summary read_when title
Android app (node): connection runbook + Connect/Chat/Voice/Canvas command surface
Pairing or reconnecting the Android node
Debugging Android gateway discovery or auth
Mirroring or controlling an Android device from a remote Mac
Verifying chat history parity across clients
Android app
The official Android app is available on [Google Play](https://play.google.com/store/apps/details?id=ai.openclaw.app&hl=en_IN). It is a companion node and requires a running OpenClaw Gateway. Source: [apps/android](https://github.com/openclaw/openclaw/tree/main/apps/android) ([build instructions](https://github.com/openclaw/openclaw/blob/main/apps/android/README.md)).

Support snapshot

System control (launchd/systemd) lives on the Gateway host — see Gateway.

Mirror and control Android from a remote Mac

scrcpy mirrors an Android screen in a macOS window and forwards keyboard and pointer input through Android Debug Bridge (ADB). This is an operator-side workflow, separate from the OpenClaw node connection. It is useful when the Android device and the Mac are in different locations but share a private Tailscale network.

Before you begin

  • Install Tailscale on the Android device and the Mac, and connect both to the same tailnet.

  • On Android, enable Developer options and USB debugging. Android 16 places Wireless debugging under Settings > System > Developer options. See Android developer options.

  • Install scrcpy and ADB on the Mac:

    brew install scrcpy
    brew install --cask android-platform-tools
    
  • Keep the Android device available for the first connection. Android must approve each Mac's ADB key before that Mac can control the device.

Enable ADB over TCP

For the initial setup, connect the Android device by USB to a trusted computer and approve its debugging prompt. Then run:

adb devices
adb tcpip 5555

You can now disconnect USB. If port 5555 stops listening after a device reboot or debugging reset, repeat this local setup step. Android 11 and later can also establish the initial trust with Wireless debugging > Pair device with pairing code and adb pair.

Allow only the controller Mac

Tailnets with restrictive grants must explicitly allow the controller Mac to reach TCP port 5555 on the Android device. Add a narrow rule to the tailnet policy, replacing the example addresses with the two devices' stable Tailscale IPs:

{
  grants: [
    {
      src: ["<remote-mac-tailnet-ip>"],
      dst: ["<android-tailnet-ip>"],
      ip: ["tcp:5555"],
    },
  ],
}

See Tailscale grants for host aliases and other selectors. Do not grant this port to the public internet or expose it with Funnel: an authorized ADB client has broad control of the device.

Connect and start mirroring

On the remote Mac:

adb connect <android-tailnet-ip>:5555
adb devices
scrcpy --serial <android-tailnet-ip>:5555

The first adb connect from this Mac shows an authorization dialog on Android. Unlock the device, confirm the key fingerprint, and select Always allow from this computer only when the Mac is trusted. A successful adb devices entry ends in device; unauthorized means the on-device prompt has not been approved.

Once the scrcpy window opens, use it directly or target it with a macOS screen-automation tool such as Peekaboo. scrcpy carries the display and input; Tailscale provides only the private network path.

Troubleshooting

  • Connection timed out: verify the tailnet grant for TCP 5555. A successful tailscale ping proves peer reachability, not that policy permits this TCP port. Test with nc -vz <android-tailnet-ip> 5555 from the Mac.
  • unauthorized: unlock Android and approve the remote Mac's ADB key, or remove the stale workstation under Wireless debugging > Paired devices and pair it again.
  • Connection refused: reconnect locally and run adb tcpip 5555 again.
  • More than one device listed: keep the explicit --serial <android-tailnet-ip>:5555 argument.

When finished, close scrcpy and disconnect ADB:

adb disconnect <android-tailnet-ip>:5555

Connection runbook

Android node app ⇄ (mDNS/NSD + WebSocket) ⇄ Gateway

Android connects directly to the Gateway WebSocket and uses device pairing (role: node).

For Tailscale or public hosts, Android requires a secure endpoint:

  • Preferred: Tailscale Serve / Funnel with https://<magicdns> / wss://<magicdns>
  • Also supported: any other wss:// Gateway URL with a real TLS endpoint
  • Cleartext ws:// remains supported on private LAN addresses / .local hosts, plus localhost, 127.0.0.1, and the Android emulator bridge (10.0.2.2)

Prerequisites

  • Gateway running on another machine (or reachable via SSH).
  • Android device/emulator can reach the gateway WebSocket:
    • Same LAN with mDNS/NSD, or
    • Same Tailscale tailnet using Wide-Area Bonjour / unicast DNS-SD (see below), or
    • Manual gateway host/port (fallback)
  • Tailnet/public mobile pairing does not use raw tailnet IP ws:// endpoints. Use Tailscale Serve or another wss:// URL instead.
  • The openclaw CLI available on the gateway machine (or via SSH), to approve pairing requests.

1. Start the Gateway

openclaw gateway --port 18789 --verbose

Confirm in logs you see something like:

  • listening on ws://0.0.0.0:18789

For remote Android access over Tailscale, prefer Serve/Funnel instead of a raw tailnet bind:

openclaw gateway --tailscale serve

This gives Android a secure wss:// / https:// endpoint. A plain gateway.bind: "tailnet" setup is not enough for first-time remote Android pairing unless you also terminate TLS separately.

2. Verify discovery (optional)

From the gateway machine:

dns-sd -B _openclaw-gw._tcp local.

More debugging notes: Bonjour.

If you also configured a wide-area discovery domain, compare against:

openclaw gateway discover --json

That shows local. plus the configured wide-area domain in one pass, using the resolved service endpoint instead of TXT-only hints.

Cross-network discovery via unicast DNS-SD

Android NSD/mDNS discovery does not cross networks. If the Android node and the gateway are on different networks but connected via Tailscale, use Wide-Area Bonjour / unicast DNS-SD instead. Discovery alone is not sufficient for tailnet/public Android pairing — the discovered route still needs a secure endpoint (wss:// or Tailscale Serve):

  1. Set up a DNS-SD zone (example openclaw.internal.) on the gateway host and publish _openclaw-gw._tcp records.
  2. Configure Tailscale split DNS for your chosen domain pointing at that DNS server.

Details and example CoreDNS config: Bonjour.

3. Connect from Android

In the Android app:

  • The app keeps its gateway connection alive via a foreground service (persistent notification).
  • Open the Connect tab.
  • Use Setup Code or Manual mode.
  • If discovery is blocked, use manual host/port in Advanced controls. For private LAN hosts, ws:// still works. For Tailscale/public hosts, turn on TLS and use a wss:// / Tailscale Serve endpoint.

After the first successful pairing, Android auto-reconnects on launch: the manual endpoint (if enabled), otherwise the last discovered gateway (best-effort).

Presence alive beacons

After the authenticated node session connects, and when the app moves to the background while the foreground service is still connected, Android calls node.event with event: "node.presence.alive". The gateway records this as lastSeenAtMs/lastSeenReason on the paired node/device metadata only after the authenticated node device identity is known.

The app counts the beacon as successfully recorded only when the gateway response includes handled: true. Older gateways may acknowledge node.event with { "ok": true }; that response is compatible but does not count as a durable last-seen update.

4. Approve pairing (CLI)

On the gateway machine:

openclaw devices list
openclaw devices approve <requestId>
openclaw devices reject <requestId>

Pairing details: Pairing.

Optional: if the Android node always connects from a tightly controlled subnet, you can opt in to first-time node auto-approval with explicit CIDRs or exact IPs:

{
  gateway: {
    nodes: {
      pairing: {
        autoApproveCidrs: ["192.168.1.0/24"],
      },
    },
  },
}

This is disabled by default. It applies only to fresh role: node pairing with no requested scopes. Operator/browser pairing and any role, scope, metadata, or public-key change still require manual approval.

5. Verify the node is connected

openclaw nodes status
openclaw gateway call node.list --params "{}"

6. Chat + history

The Android Chat tab supports session selection (default main, plus other existing sessions):

  • History: chat.history (display-normalized — inline directive tags, plain-text tool-call XML payloads (<tool_call>, <function_call>, <tool_calls>, <function_calls>, and truncated variants), and leaked ASCII/full-width model control tokens are stripped; silent-token assistant rows such as exact NO_REPLY / no_reply are omitted; oversized rows can be replaced with placeholders)
  • Send: chat.send
  • Push updates (best-effort): chat.subscribe -> event:"chat"

7. Canvas + camera

To have the node show real HTML/CSS/JS that the agent can edit on disk, point the node at the Gateway canvas host.

Nodes load canvas from the Gateway HTTP server (same port as `gateway.port`, default `18789`).
  1. Create ~/.openclaw/workspace/canvas/index.html on the gateway host.
  2. Navigate the node to it (LAN):
openclaw nodes invoke --node "<Android Node>" --command canvas.navigate --params '{"url":"http://<gateway-hostname>.local:18789/__openclaw__/canvas/"}'

Tailnet (optional): if both devices are on Tailscale, use a MagicDNS name or tailnet IP instead of .local, e.g. http://<gateway-magicdns>:18789/__openclaw__/canvas/.

This server injects a live-reload client into HTML and reloads on file changes. The Gateway also serves /__openclaw__/a2ui/, but the Android app treats remote A2UI pages as render-only. Action-capable A2UI commands use the bundled app-owned A2UI page.

Canvas commands (foreground only):

  • canvas.eval, canvas.snapshot, canvas.navigate (use {"url":""} or {"url":"/"} to return to the default scaffold). canvas.snapshot returns { format, base64 } (default format="jpeg").
  • A2UI: canvas.a2ui.push, canvas.a2ui.reset (canvas.a2ui.pushJSONL legacy alias). These use the bundled app-owned A2UI page for action-capable rendering.

Camera commands (foreground only; permission-gated): camera.snap (jpg), camera.clip (mp4). See Camera node for parameters and CLI helpers.

8. Voice + expanded Android command surface

  • Voice tab: Android has two explicit capture modes. Mic is a manual Voice-tab session that sends each pause as a chat turn and stops when the app leaves the foreground or the user leaves the Voice tab. Talk is continuous Talk Mode and keeps listening until toggled off or the node disconnects.
  • Talk Mode promotes the existing foreground service from connectedDevice to connectedDevice|microphone before capture starts, then demotes it when Talk Mode stops. The node service declares FOREGROUND_SERVICE_CONNECTED_DEVICE with CHANGE_NETWORK_STATE; Android 14+ also requires the FOREGROUND_SERVICE_MICROPHONE declaration, the RECORD_AUDIO runtime grant, and the microphone service type at runtime.
  • By default, Android Talk uses native speech recognition, Gateway chat, and talk.speak through the configured gateway Talk provider. Local system TTS is used only when talk.speak is unavailable.
  • Android Talk uses realtime Gateway relay only when talk.realtime.mode is realtime and talk.realtime.transport is gateway-relay.
  • Voice wake is implemented in source (VoiceWakeMode) but the shipping app runtime always forces it to off on connect — there is no user-facing toggle today.
  • Additional Android command families (availability depends on device, permissions, and user settings):
    • device.status, device.info, device.permissions, device.health
    • device.apps only when Settings > Phone Capabilities > Installed Apps is enabled; it lists launcher-visible apps by default (pass includeNonLaunchable for the full list).
    • notifications.list, notifications.actions (see Notification forwarding below)
    • photos.latest
    • contacts.search, contacts.add
    • calendar.events, calendar.add
    • callLog.search
    • sms.search
    • motion.activity, motion.pedometer

Assistant entrypoints

Android supports launching OpenClaw from the system assistant trigger (Google Assistant). Holding the home button (or another ACTION_ASSIST trigger) opens the app; saying "Hey Google, ask OpenClaw <prompt>" matches the app's declared App Actions query pattern and hands the prompt into the chat composer without auto-sending it.

This uses Android App Actions (shortcuts.xml capability) declared in the app manifest. No gateway-side configuration is needed — the assistant intent is handled entirely by the Android app.

App Actions availability depends on the device, Google Play Services version, and whether the user has set OpenClaw as the default assistant app.

Notification forwarding

Android can forward device notifications to the gateway as node.event items. This is configured on the device, in the app's Settings sheet — not in gateway/openclaw.json config.

Setting Description
Forward Notification Events Master toggle. Off by default; requires Notification Listener Access to be granted first.
Package Filter Allowlist (only listed package IDs forwarded) or Blocklist (default: all packages except listed IDs). OpenClaw's own package is always excluded in Blocklist mode to prevent forwarding loops.
Quiet Hours Local HH:mm start/end window that suppresses forwarding. Disabled by default; defaults to 22:00-07:00 once enabled.
Max Events / Minute Per-device rate limit on forwarded notifications. Default 20.
Route Session Key Optional. Pins forwarded notification events into a specific session instead of the device's default notification route.
Notification forwarding requires the Android Notification Listener permission. The app prompts for this during setup.