robotex/README.md

Robotex

Browser automation API with built-in anti-bot stealth. Send a URL and a sequence of declarative actions — fill a field, click, wait for an element — and get back cookies, the rendered page and, optionally, a screenshot, including sites behind Cloudflare-style JS challenges.

Status

Early MVP (v0), under active development. Single browser engine, no billing yet. Tasks and project wiki — Redmine.

Every request needs a client API key in X-API-Key (keys are issued with make key-create). Usage — requests, their duration, browser time per client — is recorded in Postgres (make stats).

Architecture

Client → Caddy (TLS) → api (FastAPI) ──task──▶ RabbitMQ ──▶ worker (dramatiq, Chrome + Xvfb)
            ▲            │  POST /v2/tasks       common queue        browser slots, open windows
            │            │  POST /v1/solve       or the worker's     pass anti-bot → run actions
            │            ▼                       own queue                    │
            └─── GET /v2/tasks/{id} ◀── Redis: task status and result (TTL) ◀─┘

• Engine: real Google Chrome (channel="chrome") driven via Patchright (undetected Playwright fork), headful under Xvfb. Browser session wrapper ported from scrapling. Firefox is not supported yet.
• Two services: api (FastAPI, no browser) registers tasks and serves results; worker (dramatiq) runs them in Chrome. Tasks go through RabbitMQ, statuses and results live in Redis with a TTL.
• API: POST /v2/tasks + GET /v2/tasks/{id} (asynchronous, poll by link), DELETE /v2/sessions/current; POST /v1/solve keeps the synchronous contract on top of the same queue; GET /health.
• Concurrency: each worker has BROWSER_SLOTS Chrome windows. Tasks without a free slot wait in the queue; /v1/solve answers 503 with Retry-After when no slot is free.
• Window holding: with browser_idle_timeout (minutes, 0–120) the session's Chrome stays open after the task; the next task with the same session_id is routed to that worker's own queue and runs in the open window, even without a url (on the current page). Idle time is a Redis key TTL; an open window occupies a slot.
• Anti-bot: detect → solve → verify loop for Cloudflare Turnstile and reCAPTCHA; captchas inside pages (Turnstile, reCAPTCHA, hCaptcha) are solved via the client's 2captcha key.
• Proxies: not managed by the service — the client may supply their own proxy per request. Browser locale and timezone are derived from the proxy's exit IP.
• Sessions: session_id cookie → persistent Chrome profile on disk, removed after PROFILE_TTL_DAYS (7) unused days. A fingerprint (screen, DPR, canvas noise seed) is generated on the first request of a session; cookies passed in a request replace the profile's cookies.
• Browser identity is not configurable: user agent and window size are the real browser's own — a spoofed UA mismatches userAgentData and Cloudflare blocks it. Clients take user_agent from the response.
• Clients and usage: Postgres — API keys (sha256 only), a journal of browser windows (opened, last used, closed) and of requests (queued, started, finished, outcome, task data without secrets). A task and a window are visible only to the client that created them; no keys — no access.
• Logging: stdout, level from LOG_LEVEL; client secrets (proxy credentials, 2captcha key, fill values, cookie values) are masked.
• Deployment: Docker Compose — api, worker, rabbitmq, redis, postgres; TLS is terminated by Caddy on the host — see docs/Deployment.md.

Project layout

src/
  main.py              # api service: /v2/tasks, /v1/solve, /health, sessions
  db/                  # Postgres, shared by api and worker
    models.py          # clients, windows, requests
    clients.py         # API keys: issue, check, revoke
    journal.py         # worker: windows and requests journal
    stats.py, cli.py   # usage report; python -m db.cli create|list|revoke|stats
  tasks/               # shared by api and worker, no browser
    store.py           # Redis: tasks, results, open windows, worker heartbeats
    broker.py          # dramatiq broker (RabbitMQ), queues, messages
    sessions.py        # session_id cookie
  worker/
    actors.py          # dramatiq actors, worker lifecycle, heartbeat
    windows.py         # browser slots, windows held between tasks, task execution
  api/
    profile.py         # per-profile generated fingerprint
    schemas.py         # SolveRequest / TaskRequest / ActionRequest / responses + validation
    actions.py         # action registry and execution
    geoip.py           # proxy → locale / timezone
    logs.py            # request/response dumps without client secrets
  engine/              # Chrome session wrapper (ported from scrapling)
    stealthy.py        # AsyncStealthySession
    session.py         # browser / context launch options
    navigation.py, page_pool.py, validators.py, proxy_rotation.py, ad_domains.py, ...
  antibot/
    orchestrator.py    # detect → solve → verify loop
    registry.py, base.py
    known_providers/   # cloudflare.py, recaptcha.py
    humanize.py        # human-like mouse moves and typing
  captcha/             # 2captcha solvers: Turnstile, reCAPTCHA, hCaptcha
client/                # Python client package robotex (pip install, make client-build)
docs/                  # API, deployment, anti-bot notes
tests/
  unit/                # pytest without a browser: make test
  browser/             # end-to-end with real Chrome in Docker: make test-browser
  harness.py           # api + worker in one process (fakeredis, no RabbitMQ) for tests
  smoke.py             # check a running stack: make smoke
  loadtest.py          # concurrent load test + docker stats
  test_main.http       # manual requests (JetBrains HTTP client)
wheels/                # wheels for offline install on Ubuntu Noble
Dockerfile               # worker: Chrome + Xvfb
Dockerfile.api           # api: no browser
docker-compose.yml       # local / load testing
docker-compose.prod.yml  # template for releases (make release)
Makefile

API

Synchronous:

POST /v1/solve
X-API-Key: <key>

{
	"url": "https://freelancehunt.com/projects",
	"proxy": "http://user:pass@host:port",
	"captcha_type": "cloudflare",
	"actions": [
		{"type": "wait_for", "selector": "a[href*=\"/project/\"]", "timeout": 60}
	],
	"html": true
}

Response:

{
	"status": "ok",
	"user_agent": "Mozilla/5.0 (...) Chrome/155.0.0.0 Safari/537.36",
	"cookies": [{"name": "cf_clearance", "value": "..."}],
	"actions": [{"type": "wait_for", "ok": true, "duration_ms": 14}],
	"html": "<head>...</head><body>...</body>",
	"duration_ms": 6405
}

On failure status is "err" with a machine-readable code (no_slots, request_failed, challenge_failed, action_failed) and a human-readable error.

Asynchronous, keeping the browser window open between tasks:

POST /v2/tasks          {"url": "https://example.com/login", "actions": [...], "browser_idle_timeout": 10}
→ 202 {"task_id": "...", "status": "queued", "url": "/v2/tasks/..."}  + Set-Cookie: session_id
GET  /v2/tasks/{id}     → {"status": "queued|running|done|failed", "result": {... same as /v1/solve ...}}
POST /v2/tasks          {"actions": [...], "browser_idle_timeout": 10}   # same session: runs in the open window
DELETE /v2/sessions/current                                              # close the window, free the slot

Full reference — fields, action types, errors, sessions, v2: docs/API.md. How to use the API from your client (same UA and proxy as the browser, sessions, retries) — docs/Usage.md.

Python client: client/README.md.

pip install "robotex @ git+https://git.sokolab.xyz/s0k0l/robotex.git@v0.1.0#subdirectory=client"

Roadmap

Not done yet:

• Workers on several hosts (needs shared or per-host routing of browser profiles)
• Real billing/usage plans beyond a flat API-key check
• Broader action vocabulary
• Second engine (Firefox, e.g. Camoufox) for targets where the Chrome fingerprint doesn't fit

Local development

make sync-offline              # venv in ~/.py_envs/robotex; another path: make VENV=... <target>
make dev                       # api + worker on the host, Chrome windows visible on your desktop; API on :8010
make dev-key name=local        # API key for it (stored in the same postgres volume as make up)

make dev stops the containerized api and worker (otherwise the Docker worker also takes tasks from the common queue and runs them invisibly), starts rabbitmq, redis and postgres with ports published on 127.0.0.1 (docker-compose.dev.yml: 5673, 6380, 5433), then runs uvicorn with --reload and the dramatiq worker in the foreground. Ctrl+C stops both; make down stops the infrastructure. Ports, slots: DEV_API_PORT, DEV_RABBITMQ_PORT, DEV_REDIS_PORT, DEV_POSTGRES_PORT, DEV_SLOTS (default 2). Chrome runs headful, so a display is required (locally — your desktop, in Docker — Xvfb).

Tests:

make sync-dev        # pytest into .venv
make test            # unit tests: api + worker + dramatiq, no browser, no RabbitMQ/Redis (~10 s)
make test-browser    # build the images and run end-to-end tests with real Chrome (needs network)
make smoke           # check the running stack from make up

Docker

make up      # build robotex-api and robotex-worker, start api, worker, rabbitmq, redis
make smoke   # health, /v1/solve, /v2/tasks, window holding
make logs
make down

The API listens on http://localhost:8000. Browser profiles (extra/user_data_dir) and the GeoIP cache live in the worker's extra named volume, so sessions survive container restarts.

Releases:

make build ver=v0.1.0      # docker images robotex-api:v0.1.0 and robotex-worker:v0.1.0
make release ver=v0.1.0    # push images to docker.sokolab.xyz + deploy bundle in build/v0.1.0/ (compose, .env)
make release ver=v0.1.0 out=file  # same, but images go to build/v0.1.0/robotex-v0.1.0.tar.gz instead of the registry

Details (Xvfb, tini, /dev/shm, env settings, load measurements) — docs/Deployment.md.