--- slug: "remote-browser-service" source_type: "clawhub" source_url: "https://clawhub.ai/skills/remote-browser" repo: "" source_file: "description" --- --- name: remote-browser-service description: > Control a remote Chrome browser via HTTP API (Kubernetes or Docker backend). Use for web automation, form filling, navigation, and page inspection on sites the user owns or has permission to access. Exposes the accessibility tree, text extraction, Chrome screenshots, VNC-native screenshots, DOM actions, and VNC actions β€” optimized for AI agents. Requires an active browser session (created via HTTP or WebSocket). metadata: openclaw: emoji: "🌐" requires: env: - name: AC_API_KEY secret: true optional: true description: "Bearer token or API key for auth (user_id derived from token)" --- # Remote Browser Service Browser control for AI agents via HTTP API. Supports both DOM-oriented automation and remote-desktop/VNC control when you need the actual framebuffer. ## Index - [Setup](#setup) - [Core Workflow](#core-workflow) - [API Reference](#api-reference) - [Screenshot](#screenshot) - [VNC interface](#vnc-interface) - [VNC screenshot](#vnc-screenshot) - [Act on elements](#act-on-elements) - [VNC action](#vnc-action) - [HTML snapshot](#html-snapshot) - [Token Cost Guide](#token-cost-guide) - [Limitations & fallbacks](#limitations--fallbacks) - [Environment Variables](#environment-variables) - [Tips](#tips) ## Setup Ensure you have an active session: 1. **Create session** β€” `POST /api/sessions` (HTTP, no WebSocket), or open WebSocket to `/ws/{session_id}` (DevTools CDP), or run from UI. Optional `url` in body (HTTP) or query (WS) to navigate immediately. 2. **Or restore** β€” Use stored session from `GET /api/stored-sessions` 3. **Auth** β€” Pass `Authorization: Bearer ` or `X-API-Key`, or `?access_token=` Base URL: `https://rb.all-completed.com` (or `RBS_BASE_URL`). Replace `{session_id}` in examples. User ID is derived from the token. ## Core Workflow 1. **Navigate** to a URL 2. **Snapshot** the accessibility tree (get refs) β€” `GET .../json` 3. **Act** on refs or selectors (click, type, fill, press) 4. **Snapshot** again to see results For visual or OS-level flows, use the VNC path instead: 1. **Open VNC interface** β€” `GET /users/{user_id}/vnc/{session_id}` when you want a live noVNC view 2. **Capture VNC framebuffer** β€” `GET .../vnc/screenshot` 3. **Send VNC input** β€” `POST .../vnc/action` with coordinates or keys 4. **Capture again** to verify pixel-level results Refs (`e0`, `e1`, …) from `/json` can be used with `/action` via `selector` (use `ref` as selector for `e5` β†’ `"e5"` maps to role/name; for now use CSS `selector`). Supported actions by mode: | Mode | Kind | Example | |--------------------|----------|-------------------------------------------------------------| | DOM (`/action`) | `click` | `{"kind":"click","selector":"button.submit"}` | | DOM (`/action`) | `tap` | `{"kind":"tap","selector":"button.submit"}` | | DOM (`/action`) | `type` | `{"kind":"type","selector":"#email","text":"user@example.com"}` | | DOM (`/action`) | `fill` | `{"kind":"fill","selector":"#email","text":"user@example.com"}` | | DOM (`/action`) | `press` | `{"kind":"press","key":"Enter"}` | | DOM (`/action`) | `focus` | `{"kind":"focus","selector":"input[name=search]"}` | | DOM (`/action`) | `hover` | `{"kind":"hover","selector":"button.submit"}` | | DOM (`/action`) | `select` | `{"kind":"select","selector":"select","value":"option-1"}` | | DOM (`/action`) | `scroll` | `{"kind":"scroll","scrollY":800}` | | DOM (`/action`) | `submit` | `{"kind":"submit","selector":"#nav-search-form"}` (or a field within the form) | | Secrets (`/request-fill`) | β€” | `{"selector":"#pass","label":"Password","field":"password"}` β€” user fills it in the Keeper app; value never seen by the agent | | VNC (`/vnc/action`) | `move` | `{"kind":"move","x":320,"y":240}` | | VNC (`/vnc/action`) | `click` | `{"kind":"click","x":320,"y":240,"button":"left","repeat":1}` | | VNC (`/vnc/action`) | `type` | `{"kind":"type","text":"hello world"}` | | VNC (`/vnc/action`) | `press` | `{"kind":"press","keys":["Ctrl","l"]}` | | VNC (`/vnc/action`) | `scroll` | `{"kind":"scroll","x":320,"y":240,"direction":"down","repeat":3}` | ## API Reference ### Create session (HTTP) ```bash curl -X POST "https://rb.all-completed.com/api/sessions" \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{}' # Optional: {"session_id": "my-session", "url": "https://example.com"} # Fork from stored session: {"session_id": "my-fork", "from": "original-session"} # Ephemeral (start from metadata/fork but don't save): {"ephemeral": true} ``` **Session lifecycle β€” closing a connection does NOT close the session.** Persistent sessions live until a **5-minute idle timeout** or an **explicit terminate**, not when your CDP/VNC connection drops: - **CDP (`/ws`)**: while connected the pod is pinned (`active_ws_connections > 0`, never reaped). On disconnect it is **kept alive** and the 5-min idle clock restarts β€” so a reconnecting client (e.g. Playwright `connectOverCDP`, which connects/disconnects a lot) doesn't churn the pod. Only `ephemeral` sessions are deleted immediately on last disconnect. - **VNC (`/vnc/ws`)**: same as CDP β€” while a viewer streams the session is pinned (`active_ws_connections > 0`), so it isn't reaped mid-view; on disconnect the pod is kept and the idle clock restarts. - **Activity that restarts the 5-min clock**: opening a CDP/VNC connection, `GET .../status`, `GET .../{session_id}`, and `POST .../ping`. - **To close immediately** instead of waiting out the idle timeout: `DELETE /api/sessions/{session_id}` (or `terminate_session`). On teardown the profile is saved. Maximum 1 concurrent session per user. If creation returns 429 or WebSocket closes with a limit error: **wait a bit** (previous session may still be shutting down) **and/or close the previous session** via `DELETE /api/sessions/{session_id}` before retrying. ### List sessions ```bash curl "https://rb.all-completed.com/api/sessions" \ -H "Authorization: Bearer " ``` ### Session status ```bash curl "https://rb.all-completed.com/api/sessions/{session_id}/status" \ -H "Authorization: Bearer " ``` Returns live session state plus current page metadata: ```json { "session_id": "session-123", "created_at": "2026-02-12T10:00:00", "active_ws_connections": 1, "status": "ready", "last_error": null, "current_url": "https://example.com/page", "page_title": "Example Domain", "last_status_code": 200 } ``` HTTP status codes: - `200` - Session found; manager status returned, with live page metadata when available - `404` - Session not found for the authenticated user - `503` - Service not initialized `last_status_code` is the browser's last navigation response code when Chrome exposes it through Navigation Timing. If it is not available yet, the field is `null`. ### List stored sessions ```bash curl "https://rb.all-completed.com/api/stored-sessions" \ -H "Authorization: Bearer " ``` Returns `{sessions: [...], count}`. Connect via WebSocket to `/ws/{session_id}` to resume. ### Navigate ```bash curl -X POST "https://rb.all-completed.com/api/sessions/{session_id}/navigate" \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{"url": "https://example.com"}' # With timeout (seconds) curl -X POST "https://rb.all-completed.com/api/sessions/{session_id}/navigate" \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{"url": "https://example.com", "timeout": 60}' ``` ### Set location ```bash # Override geolocation for the page (e.g. for location-aware sites) curl -X POST "https://rb.all-completed.com/api/sessions/{session_id}/location" \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{"latitude": 37.7749, "longitude": -122.4194}' # With accuracy (meters) curl -X POST "https://rb.all-completed.com/api/sessions/{session_id}/location" \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{"latitude": 51.5074, "longitude": -0.1278, "accuracy": 50}' ``` ### Image (download by selector) ```bash # Capture a single element (e.g. image) by CSS selector curl "https://rb.all-completed.com/api/sessions/{session_id}/image?selector=img.hero" \ -H "Authorization: Bearer " \ -o image.jpg # With quality, raw binary (selector=#banner for id) curl "https://rb.all-completed.com/api/sessions/{session_id}/image?selector=img&quality=90&raw=true" \ -H "Authorization: Bearer " \ -o element.jpg ``` Use `selector` (CSS) or `ref` (from snapshot). Returns JPEG of the element's bounding box. ### Snapshot (accessibility tree) ```bash # Full tree curl "https://rb.all-completed.com/api/sessions/{session_id}/json" \ -H "Authorization: Bearer " # Interactive elements only (buttons, links, inputs) β€” much smaller curl "https://rb.all-completed.com/api/sessions/{session_id}/json?filter=interactive" \ -H "Authorization: Bearer " # Limit depth curl "https://rb.all-completed.com/api/sessions/{session_id}/json?depth=5" \ -H "Authorization: Bearer " ``` Returns `{nodes: [{ref, role, name, depth, value?, disabled?, focused?, nodeId?}], count}`. ### Extract text ```bash # Readability mode (default) β€” strips nav/footer/ads curl "https://rb.all-completed.com/api/sessions/{session_id}/text" \ -H "Authorization: Bearer " # Raw innerText curl "https://rb.all-completed.com/api/sessions/{session_id}/text?mode=raw" \ -H "Authorization: Bearer " ``` Returns `{url, title, text}`. Cheapest option (~800 tokens for most pages). ### Screenshot ```bash # JSON with base64 curl "https://rb.all-completed.com/api/sessions/{session_id}/screenshot" \ -H "Authorization: Bearer " # Raw JPEG bytes curl "https://rb.all-completed.com/api/sessions/{session_id}/screenshot?raw=true" \ -H "Authorization: Bearer " \ -o screenshot.jpg # With quality (1-100) curl "https://rb.all-completed.com/api/sessions/{session_id}/screenshot?quality=50&raw=true" \ -H "Authorization: Bearer " \ -o screenshot.jpg # Region capture (offset x,y and width,height in CSS pixels) curl "https://rb.all-completed.com/api/sessions/{session_id}/screenshot?x=0&y=0&width=800&height=600&raw=true" \ -H "Authorization: Bearer " \ -o region.jpg ``` Use this when Chrome DevTools rendering is enough. If you need browser chrome, OS dialogs, permission prompts, or the exact remote desktop pixels, use `/vnc/screenshot` instead. ### VNC interface ```bash # Built-in noVNC client page for a session open "https://rb.all-completed.com/users/{user_id}/vnc/{session_id}" # Under the hood the page connects to the VNC websocket proxy # /users/{user_id}/vnc/ws/{session_id} ``` Use the VNC interface when you need a live remote-desktop view of the session instead of DOM snapshots. ### VNC screenshot ```bash # Raw PNG bytes from the VNC framebuffer curl "https://rb.all-completed.com/api/sessions/{session_id}/vnc/screenshot?raw=true" \ -H "Authorization: Bearer " \ -o screen.png # Cropped framebuffer region curl "https://rb.all-completed.com/api/sessions/{session_id}/vnc/screenshot?x=0&y=0&width=800&height=600&raw=true" \ -H "Authorization: Bearer " \ -o region.png ``` Unlike `/screenshot`, this captures the VNC framebuffer directly. Use it for browser chrome, native permission prompts, OS-level dialogs, or anything only visible in the remote desktop. ### Page size ```bash # Get page content dimensions (use with screenshot clip) curl "https://rb.all-completed.com/api/sessions/{session_id}/page-size" \ -H "Authorization: Bearer " ``` Returns `{width, height}` in CSS pixels. ### Act on elements ```bash # Click by selector curl -X POST "https://rb.all-completed.com/api/sessions/{session_id}/action" \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{"kind": "click", "selector": "button.submit"}' # Click by coordinates (viewport x,y) curl -X POST "https://rb.all-completed.com/api/sessions/{session_id}/action" \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{"kind": "click", "x": 100, "y": 200}' # Type into element (focus + insertText) curl -X POST "https://rb.all-completed.com/api/sessions/{session_id}/action" \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{"kind": "type", "selector": "#email", "text": "user@example.com"}' # Fill (set value directly) curl -X POST "https://rb.all-completed.com/api/sessions/{session_id}/action" \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{"kind": "fill", "selector": "#email", "text": "user@example.com"}' # Press a key curl -X POST "https://rb.all-completed.com/api/sessions/{session_id}/action" \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{"kind": "press", "key": "Enter"}' # Press Enter in a specific input: -d '{"kind": "press", "key": "Enter", "selector": "input#search"}' # Focus, hover, select, scroll curl -X POST "https://rb.all-completed.com/api/sessions/{session_id}/action" \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{"kind": "focus", "selector": "input[name=search]"}' curl -X POST "https://rb.all-completed.com/api/sessions/{session_id}/action" \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{"kind": "scroll", "scrollY": 800}' ``` **Action kinds:** `click`, `type`, `fill`, `press`, `focus`, `hover`, `select`, `scroll`. Use `selector` (CSS) or `ref` (from snapshot). For `click` you can use `x` and `y` (viewport coordinates) instead of selector. For `fill`, the server focuses the field, `select()` only if the field already has text (then `Input.insertText` replaces), otherwise focuses and inserts like `type`β€”controlled inputs (e.g. React) update reliably. For `press` use `key` (e.g. `Enter`, `Tab`, `Escape`, `Space`, `ArrowUp`); optional `selector` focuses element first. ### VNC action ```bash # Move mouse curl -X POST "https://rb.all-completed.com/api/sessions/{session_id}/vnc/action" \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{"kind":"move","x":320,"y":240}' # Click at framebuffer coordinates curl -X POST "https://rb.all-completed.com/api/sessions/{session_id}/vnc/action" \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{"kind":"click","x":320,"y":240,"button":"left","repeat":1}' # Press keys directly over VNC curl -X POST "https://rb.all-completed.com/api/sessions/{session_id}/vnc/action" \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{"kind":"press","keys":["Ctrl","l"]}' ``` **VNC action kinds:** `move`, `click`, `type`, `press`, `scroll`. These actions are framebuffer-oriented and do not use DOM selectors. Prefer them when DOM automation cannot see or control the target UI. ### HTML snapshot ```bash # Full DOM with inlined CSS (opens in browser) curl "https://rb.all-completed.com/api/sessions/{session_id}/html" \ -H "Authorization: Bearer " ``` ## Token Cost Guide | Method | Typical tokens | When to use | |--------|----------------|-------------| | `/status` | ~50 | Just the current URL / title / HTTP status | | `/text` (readability) | ~800 | Reading page content | | `/text?mode=raw` | ~2K–8K | Content readability strips (hidden labels, etc.) | | `/json?filter=interactive` | ~3,600 | Finding buttons/links/inputs to act on (+ refs) | | `/json` (full a11y tree) | ~10,500 | Full structure / element relationships | | `/html?obfuscate=true` (simple markup) | ~10K–40K | Need exact CSS selectors / markup the a11y tree lacks | | `/html` (full markup) | very large | Raw DOM + inlined CSS; rarely needed, may exceed the 5 MiB cap | | `/image?selector=` | ~1K (vision) | Capture a single element / download an image | | `/screenshot` (clipped, low `quality`) | ~1K–2K (vision) | Visual check of one region | | `/screenshot` (full page) | ~2K+ (vision) | Whole-page layout / visual verification | | `/vnc/screenshot` | ~2K+ (vision) | Non-DOM/native UI, canvas, or when DOM tools fail | **Decision order β€” use the *cheapest* tool that answers your question, escalate only if it doesn't:** 1. **`/status`** β€” only need where you are (URL/title/status). 2. **`/text`** β€” reading/extracting content. (`?mode=raw` if readability hides what you need.) 3. **`/json?filter=interactive`** β€” locating things to click/type; returns refs to act on. 4. **`/json`** (full) β€” need structure/relationships the filtered tree omits. 5. **`/html?obfuscate=true`** β€” need a precise selector/markup not surfaced by the a11y tree. Prefer obfuscated (compact) over full. 6. **`/html`** (full) β€” last resort for raw markup/CSS; large. 7. **`/screenshot` (clipped + low quality)** β€” *only* for visual confirmation or non-DOM layout. Always clip (`x,y,width,height`) and drop `quality`; never grab a full high-quality page when a region will do. 8. **`/vnc/screenshot`** β€” only for native/canvas/non-DOM surfaces, or when DOM extraction genuinely fails. **Rule of thumb:** text/markup ≫ screenshots for token cost. A clipped JPEG is still an image; a `/text` call is a few hundred tokens. Act on **selectors/refs** from steps 2–6 rather than re-screenshotting to "look again," and verify state changes with the cheapest read, not a fresh full screenshot. ## Environment Variables | Var | Description | |-----|-------------| | `RBS_BASE_URL` | Base URL (e.g. https://rb.all-completed.com) | | `AC_API_KEY` | Bearer token or API key (user_id derived from token) | ## Tips - **Session required** β€” Ensure a session exists before calling navigate/json/text/action. Create via `POST /api/sessions` (HTTP), WebSocket, or restore from stored sessions. - **Check live URL** β€” Use `GET /api/sessions/{session_id}/status` when you need the current page URL/title or last response status without fetching full page text. - **429 / session limit** β€” If create fails with 429 or WebSocket closes (limit exceeded): wait a few seconds and/or terminate the existing session with `DELETE /api/sessions/{session_id}` first, then retry. - **Refs from snapshot** β€” Use `selector` with the `ref` string (e.g. `"e5"`) when the action API supports refβ†’DOM resolution; otherwise prefer CSS selectors. - **Readability vs raw** β€” `/text` (default) strips nav/footer/ads; `?mode=raw` returns full `innerText`. - **Interactive filter** β€” `?filter=interactive` on `/json` reduces nodes by ~75% for action tasks. - **VNC vs DOM** β€” Use `/action` for selectors/refs in the page DOM. Use `/vnc/action` and `/vnc/screenshot` for pixel-level automation and UI outside the DOM. - **Stored sessions** β€” Sessions persist to S3 and are restored on reopen. The **whole browser profile** is captured (cookies, localStorage, sessionStorage, IndexedDB, **Service Workers**, Cache Storage, metadata) and restored as one consistent unit, so apps that keep their login in IndexedDB/Service Workers (e.g. Telegram Web) come back **logged in**, not just at the login page. List with `GET /api/stored-sessions`, then reopen by creating a session with the same `session_id` (HTTP/WebSocket). If `url` is not provided on connect, the saved page URL is used for redirect. Use `GET/PUT /api/stored-sessions/{session_id}` to read or update metadata: `url` (redirect on reopen), `description` (free-text human label, max 500 chars, returned in the list endpoint's `descriptions` map), `width`/`height` (default resolution), `encrypt_with_api_key`. To move or edit individual persisted blobs without a live browser, use `GET/PUT /api/stored-sessions/{session_id}/cookies` (JSON array of cookie objects), `GET/PUT .../local-storage`, `GET/PUT .../session-storage` (both JSON objects with string keys and string values), `GET/PUT .../indexeddb` (IndexedDB snapshot object), and `GET/PUT .../cache-storage` (Cache Storage snapshot object). `DELETE /api/stored-sessions/{session_id}` wipes all persisted state for a session. ## Limitations & fallbacks Real-world heavy pages (Amazon, marketplaces, dashboards) hit these. Know the fallback for each: - **Prefer text/markup extraction over screenshots** β€” For reading page content, `/text`, `/json` (accessibility snapshot), and `/html` are *far* more efficient than `/screenshot` or `/vnc/screenshot`: they return compact, parseable structure instead of a large base64 image, so they cost a fraction of the tokens/bandwidth and give you selectors to act on. **Default to text/markup; use screenshots only for visual verification, pixel-level layout, or canvas/`