Webpilot

A browser tool. Real Chromium, DOM-first, WebSocket protocol.

Webpilot launches a real Chromium-based browser with a local extension runtime, exposes a WebSocket protocol, and lets a user, script, or LLM drive that browser through the same command surface.

What Webpilot does

• Starts and controls a real browser; no CDP, no detectable debugging port
• Exposes the live DOM directly: navigation, element discovery, querying, interaction, cookies
• Provides configurable cursor, click, typing, and scroll behavior
• Works from the CLI, raw WebSocket, Node, or an MCP adapter

What Webpilot does not do

• Decide what to do next
• Ship a tuned human profile
• Ship site strategy, retries, or route doctrine

Install

npm install -g h17-webpilot

Quick Start

1. Start

webpilot start

If no config exists, the first run will detect installed browsers, ask you to choose one, and generate ~/h17-webpilot/config.js.

Use webpilot start -d for an append-only session log (~/h17-webpilot/webpilot.log by default).

2. Use the tool

webpilot -c 'go example.com'
webpilot -c 'discover'
webpilot -c 'click h1'
webpilot -c 'wait h1'
webpilot -c 'html'
webpilot -c 'cookies load ./cookies.json'

Use the same loop every time: inspect, act, verify.

CLI

webpilot                        # interactive REPL
webpilot -c 'go example.com'   # single command
webpilot start                  # launch browser + WS server
webpilot start -d               # launch with session logging
webpilot stop                   # stop running server

Core Commands

Examples

# navigate
webpilot -c 'go https://example.com'

# discover interactive elements
webpilot -c 'discover'

# query by CSS selector
webpilot -c 'q "button.submit"'

# click an element
webpilot -c 'click h3'

# type into an input
webpilot -c 'type "input[name=q]" hello world'

# press a key
webpilot -c 'key Enter'

# scroll down 300px
webpilot -c 'sd 300'

# read page HTML
webpilot -c 'html'

# save a screenshot
webpilot -c 'ss'

# dump cookies
webpilot -c 'cookies'

Raw Mode

For direct access to capability groups, use the raw action syntax or pass raw JSON.

Action syntax

webpilot -c 'human.click {"selector": "button[type=submit]"}'

Raw JSON

webpilot -c '{"action": "dom.getHTML", "params": {}}'

WebSocket Protocol

Connect to the WebSocket server and send JSON messages to control the browser programmatically from any language.

// Connect to ws://localhost:7331

{
  "id": "1",
  "action": "tabs.navigate",
  "params": {
    "url": "https://example.com"
  }
}

Capability groups

Node API

The Node API is a wrapper over the same WebSocket protocol.

const { startWithPage } = require('h17-webpilot');
const { page } = await startWithPage();

Available methods

Full example

const { startWithPage } = require('h17-webpilot');

const { page } = await startWithPage();
await page.navigate('https://example.com');
await page.query('h1');
await page.click('h1');
await page.waitFor('body');

Config File

Config is loaded from ~/h17-webpilot/config.js (or config.json). Override with --config <path>.

Public config is split into two sections:

The public package exposes a lot of knobs on purpose. The user decides how much to tune. The package does not ship a strong profile.

module.exports = {
  framework: {
    debug: {
      cursor: true,
      sessionLogPath: '~/h17-webpilot/webpilot.log',
    },
  },
  human: {
    calibrated: false,
    profileName: 'public-default',
    cursor: {
      spreadRatio: 0.16,
      jitterRatio: 0,
      stutterChance: 0,
      driftThresholdPx: 0,
      overshootRatio: 0,
    },
    click: {
      thinkDelayMin: 35,
      thinkDelayMax: 90,
      maxShiftPx: 50,
    },
    type: {
      baseDelayMin: 8,
      baseDelayMax: 20,
      variance: 4,
      pauseChance: 0,
      pauseMin: 0,
      pauseMax: 0,
    },
  },
};

Human Behavior

The human config section controls how interactions behave:

• cursor: movement speed, overshoot, path jitter
• click: pre/post delays, shift tolerance
• typing: speed, variance, pause behavior
• scroll: speed, acceleration, drift
• avoid: element avoidance rules

These defaults do not represent a human profile: typing is very fast, overshoot is off, jitter is off, drift is off. They are there to show what is configurable. The package does not ship your final values.

Boot Config

Webpilot can load state on startup from config before the user or LLM sends any commands.

module.exports = {
  browser: "/Applications/Chromium.app/Contents/MacOS/Chromium",
  boot: {
    cookiesPath: './cookies.json',
    commands: [
      'go https://hugopalma.work',
      'cookies load ./cookies.json',
      { action: 'framework.getConfig', params: {} }
    ],
  },
};

Rules:

• boot.cookiesPath loads a cookie jar before commands run
• boot.commands accepts CLI-style strings
• boot.commands also accepts raw command objects: { action, params, tabId? }
• String commands support cookies load <file> in addition to normal shorthands

Tested Browsers

• Chromium
• Helium
• Google Chrome

Operating Loop

Every browser interaction follows the same three-phase cycle: Inspect, Act, Verify. Do not skip the inspect step unless you already have fresh page state from the immediately preceding command.

Quoting Rule

Always double-quote the entire -c argument when it contains spaces or special characters. Single-word commands can omit quotes. Never use single quotes. Never nest quotes inside the -c argument.

webpilot -c "type #input hello world"
webpilot -c "click #submit"
webpilot -c "go https://example.com"

# single-word commands can omit quotes
webpilot -c html
webpilot -c discover
webpilot -c ss

1. Inspect

Read page state before doing anything.

2. Act

Use the safest matching action.

webpilot -c "click #APjFqb"
webpilot -c "type #APjFqb hello world"

click accepts both handle IDs and CSS selectors. Always discover or q immediately before interacting so handles and selectors are fresh.

If the runtime refuses an action, respect the refusal and re-inspect the page.

3. Verify

After navigation or interaction, confirm the new state.

Safe Usage

1. Never guess selectors when html or discover can tell you the real ones.
2. Never assume a click worked. Verify it.
3. Never treat { "clicked": false } as something to brute-force through.
4. Never confuse DOM reading with interaction. Read first, then act.
5. Re-query stale handles instead of reusing them blindly.
6. Do not use eval; it hits CSP on most sites.

Strategy Notes

• Prefer html, discover, and q over eval when DOM inspection is enough.
• Use wait after page-changing actions.
• Use handle IDs for click, CSS selectors for type.
• Use screenshots when layout or visibility is the uncertainty, not HTML structure.
• If the task needs a preloaded authenticated session, load cookies first or use config boot commands.

Raw Protocol from LLM Context

If you need a protocol action that the shorthand CLI does not expose directly, send it through raw mode:

webpilot -c "dom.queryAllInfo {\"selector\": \"a[href]\"}"
webpilot -c "human.scroll {\"selector\": \".feed\", \"direction\": \"down\"}"
webpilot -c "framework.getConfig {}"

You can also send a full JSON message:

webpilot -c "{\"action\": \"tabs.navigate\", \"params\": {\"url\": \"https://example.com\"}}"

MCP Server

An MCP adapter is available for environments that support the Model Context Protocol (e.g. Claude Desktop, Cursor, Windsurf). The MCP server connects to the same WebSocket runtime at ws://localhost:7331.

Claude Desktop Config

Add the following to ~/Library/Application Support/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "webpilot": {
      "command": "npx",
      "args": ["webpilot-mcp"]
    }
  }
}

Connection

Webpilot exposes browser control over WebSocket. The protocol tells the browser what to do. It does not decide the task; the user or LLM chooses the sequence of actions.

WebSocket: ws://localhost:7331

The local server listens on this port. The extension connects on launch. Your client connects to the same server.

Runtime Config

The extension runtime exposes configuration controls:

Normal usage: Node loads ~/h17-webpilot/config.js (or path passed via --config), the server injects framework and human settings into commands, and clients can still call framework.setConfig and framework.getConfig directly.

The public package ships example defaults. They do not represent a human profile, and advanced cursor tuning or realistic timing still depends on user configuration.

Message Format

Request

{
  "id": "unique-string",
  "tabId": 123,
  "action": "action.name",
  "params": {}
}

• id: correlate responses to requests
• tabId: optional target tab
• action: protocol action name
• params: action-specific payload

Response

// success
{ "id": "same-id", "result": {} }

// error
{ "id": "same-id", "error": "error message" }

Keepalive

The server sends { "type": "ping" } every 20 seconds. The extension responds with { "type": "pong" }.

Handles

Many DOM commands return handleId values like el_42.

• Created by dom.querySelector, dom.querySelectorAll, dom.waitForSelector
• Used by dom.click, dom.boundingBox, human.click, and related commands
• Stored with WeakRef and cleaned up after TTL or GC
• If both handleId and selector are provided, handleId wins

Tabs Actions

Frames Actions

Cookies Actions

DOM Actions

dom.click uses the same safe interaction pipeline as human.click. Other dom.* commands are direct DOM/runtime operations.

Keyboard Names

Valid key names for dom.keyPress, dom.keyDown, dom.keyUp:

Meta, Control, Shift, Alt, Enter, Tab, Escape, Backspace, Delete, Space, ArrowUp, ArrowDown, ArrowLeft, ArrowRight, Home, End, PageUp, PageDown, single characters, or code forms like KeyA and Digit5.

Human Actions

Human commands add safety checks and use injected human.* config. Public config sections: human.cursor, human.click, human.type, human.scroll, human.avoid.

human.click

{
  "action": "human.click",
  "params": {
    "handleId": "el_42",
    "avoid": {
      "selectors": [".premium-upsell"],
      "classes": ["sponsored"],
      "ids": ["popup-cta"],
      "attributes": { "data-ad": "*" }
    }
  }
}

Returns { "clicked": true } on success, or { "clicked": false, "reason": "...", "detail": "..." } when blocked.

Built-in Safety Checks

1. Avoid rules
2. aria-hidden
3. Missing offsetParent
4. Honeypot class patterns
5. opacity: 0
6. visibility: hidden
7. Sub-pixel size
8. Missing bounding box
9. Scroll into view
10. Optional drift-away behavior from human.cursor
11. Bezier path using human.cursor
12. Think delay from human.click
13. Disappearance check
14. Shift check
15. mousedown → mouseup → click at actual cursor coordinates

human.type

{
  "action": "human.type",
  "params": {
    "text": "Hello world",
    "selector": "#search-input"
  }
}

Returns { "typed": true } or { "typed": false, "reason": "avoided" }. Typing cadence comes from human.type config. Public defaults are very fast: baseDelayMin: 8, baseDelayMax: 20, pauseChance: 0.

human.scroll

{
  "action": "human.scroll",
  "params": {
    "handleId": "el_7",
    "direction": "down"
  }
}

Accepts handleId, selector, or neither. Returns { "scrolled": true, "amount": 487 }.

human.clearInput

{
  "action": "human.clearInput",
  "params": { "selector": "#email-input" }
}

Returns { "cleared": true } or a click failure response. Behavior: safe click to focus, triple-click to select, backspace/delete sequence.

Avoid Rules

All human.* commands accept an avoid object. Per-request avoid merges with global human.avoid config.

{
  "avoid": {
    "selectors": [".cookie-banner", "#popup button"],
    "classes": ["sponsored", "ad-slot"],
    "ids": ["newsletter-signup"],
    "attributes": { "data-ad": "*", "data-tracking": "*" }
  }
}

Events

The server pushes events over the same WebSocket connection.

response

{
  "type": "event",
  "event": "response",
  "data": {
    "url": "https://...",
    "status": 200,
    "tabId": 123,
    "method": "GET"
  }
}

urlChanged

{
  "type": "event",
  "event": "urlChanged",
  "data": {
    "tabId": 123,
    "url": "https://..."
  }
}

cookiesChanged

{
  "type": "event",
  "event": "cookiesChanged",
  "data": {
    "cookies": [...],
    "count": 42
  }
}

CSP

Two execution contexts exist: the ISOLATED world (DOM-safe, CSP-safe) and the MAIN world (page globals, may be blocked by CSP).

Commands that work under CSP

• dom.querySelector
• dom.querySelectorAll
• dom.getHTML
• human.click
• human.type
• human.scroll
• dom.click

dom.evaluate and dom.elementEvaluate try MAIN first and fall back when possible.

Python Client Example

import asyncio, json, uuid, websockets

async def main():
    async with websockets.connect('ws://localhost:7331') as ws:
        async def send(action, params={}, tab_id=None):
            msg = {"id": str(uuid.uuid4()), "action": action, "params": params}
            if tab_id:
                msg["tabId"] = tab_id
            await ws.send(json.dumps(msg))
            resp = json.loads(await ws.recv())
            if "error" in resp:
                raise Exception(resp["error"])
            return resp["result"]

        tabs = await send("tabs.list")
        tab = tabs[0]["id"]
        await send("tabs.navigate", {"url": "https://example.com"}, tab)
        handle = await send("dom.querySelector", {"selector": "h1"}, tab)
        print(await send("human.click", {"handleId": handle}, tab))
        await send("human.type", {"text": "Hello", "selector": "#input"}, tab)

asyncio.run(main())

Limits

• Defaults are for demonstration and development, not for behavior parity
• The browser tool does not decide workflows
• The user or LLM still has to choose selectors, waits, retries, and verification steps
• dom.evaluate may hit CSP restrictions on some sites. DOM reading and interaction still work through the isolated content-script path.

License

Apache 2.0