Browser automation API with built-in anti-bot stealth. Send a URL and a sequence of declarative actions — fill a field, click, wait for an element — and get back cookies, the rendered page and, optionally, a screenshot, including sites behind Cloudflare-style JS challenges.

Status

Early MVP (v0), under active development. Single browser engine, no billing yet. Tasks and project wiki — Redmine.

Not implemented yet, though planned for v1:

API-key auth (X-API-Key is accepted but not checked) — #4
persisting request outcomes (success rate) — #17
cleanup of old browser profiles — #20

Architecture

Client → Caddy (TLS) → api (FastAPI) ──task──▶ RabbitMQ ──▶ worker (dramatiq, Chrome + Xvfb)
            ▲            │  POST /v2/tasks       common queue        browser slots, open windows
            │            │  POST /v1/solve       or the worker's     pass anti-bot → run actions
            │            ▼                       own queue                    │
            └─── GET /v2/tasks/{id} ◀── Redis: task status and result (TTL) ◀─┘

Engine: real Google Chrome (channel="chrome") driven via Patchright (undetected Playwright fork), headful under Xvfb. Browser session wrapper ported from scrapling. Firefox is not supported yet.
Two services: api (FastAPI, no browser) registers tasks and serves results; worker (dramatiq) runs them in Chrome. Tasks go through RabbitMQ, statuses and results live in Redis with a TTL.
API: POST /v2/tasks + GET /v2/tasks/{id} (asynchronous, poll by link), DELETE /v2/sessions/current; POST /v1/solve keeps the synchronous contract on top of the same queue; GET /health.
Concurrency: each worker has BROWSER_SLOTS Chrome windows. Tasks without a free slot wait in the queue; /v1/solve answers 503 with Retry-After when no slot is free.
Window holding: with browser_idle_timeout (minutes, 0–120) the session's Chrome stays open after the task; the next task with the same session_id is routed to that worker's own queue and runs in the open window, even without a url (on the current page). Idle time is a Redis key TTL; an open window occupies a slot.
Anti-bot: detect → solve → verify loop for Cloudflare Turnstile and reCAPTCHA; captchas inside pages (Turnstile, reCAPTCHA, hCaptcha) are solved via the client's 2captcha key.
Proxies: not managed by the service — the client may supply their own proxy per request. Browser locale and timezone are derived from the proxy's exit IP.
Sessions: session_id cookie → persistent Chrome profile on disk. A fingerprint (screen, DPR) is generated on the first request of a session; cookies passed in a request replace the profile's cookies.
Browser identity is not configurable: user agent and window size are the real browser's own — a spoofed UA mismatches userAgentData and Cloudflare blocks it. Clients take user_agent from the response.
Logging: stdout, level from LOG_LEVEL; client secrets (proxy credentials, 2captcha key, fill values, cookie values) are masked.
Deployment: Docker Compose — api, worker, rabbitmq, redis; TLS is terminated by Caddy on the host — see docs/Deployment.md.

Project layout

src/
  main.py              # api service: /v2/tasks, /v1/solve, /health, sessions
  tasks/               # shared by api and worker, no browser
    store.py           # Redis: tasks, results, open windows, worker heartbeats
    broker.py          # dramatiq broker (RabbitMQ), queues, messages
    sessions.py        # session_id cookie
  worker/
    actors.py          # dramatiq actors, worker lifecycle, heartbeat
    windows.py         # browser slots, windows held between tasks, task execution
  api/
    profile.py         # per-profile generated fingerprint
    schemas.py         # SolveRequest / TaskRequest / ActionRequest / responses + validation
    actions.py         # action registry and execution
    geoip.py           # proxy → locale / timezone
    logs.py            # request/response dumps without client secrets
  engine/              # Chrome session wrapper (ported from scrapling)
    stealthy.py        # AsyncStealthySession
    session.py         # browser / context launch options
    navigation.py, page_pool.py, validators.py, proxy_rotation.py, ad_domains.py, ...
  antibot/
    orchestrator.py    # detect → solve → verify loop
    registry.py, base.py
    known_providers/   # cloudflare.py, recaptcha.py
    humanize.py        # human-like mouse moves and typing
  captcha/             # 2captcha solvers: Turnstile, reCAPTCHA, hCaptcha
docs/                  # API, deployment, anti-bot notes
tests/
  unit/                # pytest without a browser: make test
  browser/             # end-to-end with real Chrome in Docker: make test-browser
  harness.py           # api + worker in one process (fakeredis, no RabbitMQ) for tests
  smoke.py             # check a running stack: make smoke
  loadtest.py          # concurrent load test + docker stats
  test_main.http       # manual requests (JetBrains HTTP client)
wheels/                # wheels for offline install on Ubuntu Noble
Dockerfile               # worker: Chrome + Xvfb
Dockerfile.api           # api: no browser
docker-compose.yml       # local / load testing
docker-compose.prod.yml  # template for releases (make release)
Makefile

API

Synchronous:

POST /v1/solve
X-API-Key: <key>

{
	"url": "https://freelancehunt.com/projects",
	"proxy": "http://user:pass@host:port",
	"captcha_type": "cloudflare",
	"actions": [
		{"type": "wait_for", "selector": "a[href*=\"/project/\"]", "timeout": 60}
	],
	"html": true
}

Response:

{
	"status": "ok",
	"user_agent": "Mozilla/5.0 (...) Chrome/155.0.0.0 Safari/537.36",
	"cookies": [{"name": "cf_clearance", "value": "..."}],
	"actions": [{"type": "wait_for", "ok": true, "duration_ms": 14}],
	"html": "<head>...</head><body>...</body>",
	"duration_ms": 6405
}

On failure status is "err" with a machine-readable code (no_slots, request_failed, challenge_failed, action_failed) and a human-readable error.

Asynchronous, keeping the browser window open between tasks:

POST /v2/tasks          {"url": "https://example.com/login", "actions": [...], "browser_idle_timeout": 10}
→ 202 {"task_id": "...", "status": "queued", "url": "/v2/tasks/..."}  + Set-Cookie: session_id
GET  /v2/tasks/{id}     → {"status": "queued|running|done|failed", "result": {... same as /v1/solve ...}}
POST /v2/tasks          {"actions": [...], "browser_idle_timeout": 10}   # same session: runs in the open window
DELETE /v2/sessions/current                                              # close the window, free the slot

Full reference — fields, action types, errors, sessions, v2: docs/API.md. How to use the API from your client (same UA and proxy as the browser, sessions, retries) — docs/Usage.md.

Roadmap

Not done yet:

Workers on several hosts (needs shared or per-host routing of browser profiles)
Real billing/usage plans beyond a flat API-key check
Broader action vocabulary
Second engine (Firefox, e.g. Camoufox) for targets where the Chrome fingerprint doesn't fit

Local development

make sync-offline        # venv in ~/.py_envs/robotex; another path: make VENV=... <target>
docker compose up -d rabbitmq redis                       # queue and storage
cd src && RABBITMQ_URL=... REDIS_URL=... uvicorn main:app --reload
cd src && dramatiq worker.actors --processes 1 --threads 4

The compose network is not published, so for running the services on the host publish 5672 and 6379 yourself, or just use make up. Chrome runs headful, so a display is required (locally — your desktop, in Docker — Xvfb).

Tests:

make sync-dev        # pytest into .venv
make test            # unit tests: api + worker + dramatiq, no browser, no RabbitMQ/Redis (~10 s)
make test-browser    # build the images and run end-to-end tests with real Chrome (needs network)
make smoke           # check the running stack from make up

Docker

make up      # build robotex-api and robotex-worker, start api, worker, rabbitmq, redis
make smoke   # health, /v1/solve, /v2/tasks, window holding
make logs
make down

The API listens on http://localhost:8000. Browser profiles (extra/user_data_dir) and the GeoIP cache live in the worker's extra named volume, so sessions survive container restarts.

Releases:

make build ver=v0.1.0      # docker images robotex-api:v0.1.0 and robotex-worker:v0.1.0
make release ver=v0.1.0    # images + deploy bundle in build/v0.1.0/ (compose, .env, images .tar.gz)

Details (Xvfb, tini, /dev/shm, env settings, load measurements) — docs/Deployment.md.

README.md Unescape Escape

Robotex