# Robotex Browser automation API with built-in anti-bot stealth. Send a URL and a sequence of declarative actions — fill a field, click, wait for an element — and get back cookies, the rendered page and, optionally, a screenshot, including sites behind Cloudflare-style JS challenges. ## Status Early MVP (v0), under active development. Single browser engine, no billing yet. Tasks and project wiki — [Redmine](https://redmine.sokolab.xyz/projects/robotex). Every request needs a client API key in `X-API-Key` (keys are issued with `make key-create`). Usage — requests, their duration, browser time per client — is recorded in Postgres (`make stats`). ## Architecture ``` Client → Caddy (TLS) → api (FastAPI) ──task──▶ RabbitMQ ──▶ worker (dramatiq, Chrome + Xvfb) ▲ │ POST /v2/tasks common queue browser slots, open windows │ │ POST /v1/solve or the worker's pass anti-bot → run actions │ ▼ own queue │ └─── GET /v2/tasks/{id} ◀── Redis: task status and result (TTL) ◀─┘ ``` - **Engine**: real Google Chrome (`channel="chrome"`) driven via [Patchright](https://github.com/Kaliiiiiiiiii-Vinyzu/patchright-python) (undetected Playwright fork), headful under Xvfb. Browser session wrapper ported from scrapling. Firefox is not supported yet. - **Two services**: `api` (FastAPI, no browser) registers tasks and serves results; `worker` (dramatiq) runs them in Chrome. Tasks go through RabbitMQ, statuses and results live in Redis with a TTL. - **API**: `POST /v2/tasks` + `GET /v2/tasks/{id}` (asynchronous, poll by link), `DELETE /v2/sessions/current`; `POST /v1/solve` keeps the synchronous contract on top of the same queue; `GET /health`. - **Concurrency**: each worker has `BROWSER_SLOTS` Chrome windows. Tasks without a free slot wait in the queue; `/v1/solve` answers `503` with `Retry-After` when no slot is free. - **Window holding**: with `browser_idle_timeout` (minutes, 0–120) the session's Chrome stays open after the task; the next task with the same `session_id` is routed to that worker's own queue and runs in the open window, even without a `url` (on the current page). Idle time is a Redis key TTL; an open window occupies a slot. - **Anti-bot**: detect → solve → verify loop for Cloudflare Turnstile and reCAPTCHA; captchas inside pages (Turnstile, reCAPTCHA, hCaptcha) are solved via the client's 2captcha key. - **Proxies**: not managed by the service — the client may supply their own proxy per request. Browser locale and timezone are derived from the proxy's exit IP. - **Sessions**: `session_id` cookie → persistent Chrome profile on disk, removed after `PROFILE_TTL_DAYS` (7) unused days. A fingerprint (screen, DPR, canvas noise seed) is generated on the first request of a session; cookies passed in a request replace the profile's cookies. - **Browser identity is not configurable**: user agent and window size are the real browser's own — a spoofed UA mismatches `userAgentData` and Cloudflare blocks it. Clients take `user_agent` from the response. - **Clients and usage**: Postgres — API keys (sha256 only), a journal of browser windows (opened, last used, closed) and of requests (queued, started, finished, outcome, task data without secrets). A task and a window are visible only to the client that created them; no keys — no access. - **Logging**: stdout, level from `LOG_LEVEL`; client secrets (proxy credentials, 2captcha key, `fill` values, cookie values) are masked. - **Deployment**: Docker Compose — `api`, `worker`, `rabbitmq`, `redis`, `postgres`; TLS is terminated by Caddy on the host — see [docs/Deployment.md](docs/Deployment.md). ### Project layout ``` src/ main.py # api service: /v2/tasks, /v1/solve, /health, sessions db/ # Postgres, shared by api and worker models.py # clients, windows, requests clients.py # API keys: issue, check, revoke journal.py # worker: windows and requests journal stats.py, cli.py # usage report; python -m db.cli create|list|revoke|stats tasks/ # shared by api and worker, no browser store.py # Redis: tasks, results, open windows, worker heartbeats broker.py # dramatiq broker (RabbitMQ), queues, messages sessions.py # session_id cookie worker/ actors.py # dramatiq actors, worker lifecycle, heartbeat windows.py # browser slots, windows held between tasks, task execution api/ profile.py # per-profile generated fingerprint schemas.py # SolveRequest / TaskRequest / ActionRequest / responses + validation actions.py # action registry and execution geoip.py # proxy → locale / timezone logs.py # request/response dumps without client secrets engine/ # Chrome session wrapper (ported from scrapling) stealthy.py # AsyncStealthySession session.py # browser / context launch options navigation.py, page_pool.py, validators.py, proxy_rotation.py, ad_domains.py, ... antibot/ orchestrator.py # detect → solve → verify loop registry.py, base.py known_providers/ # cloudflare.py, recaptcha.py humanize.py # human-like mouse moves and typing captcha/ # 2captcha solvers: Turnstile, reCAPTCHA, hCaptcha client/ # Python client package robotex (pip install, make client-build) docs/ # API, deployment, anti-bot notes tests/ unit/ # pytest without a browser: make test browser/ # end-to-end with real Chrome in Docker: make test-browser harness.py # api + worker in one process (fakeredis, no RabbitMQ) for tests smoke.py # check a running stack: make smoke loadtest.py # concurrent load test + docker stats test_main.http # manual requests (JetBrains HTTP client) wheels/ # wheels for offline install on Ubuntu Noble Dockerfile # worker: Chrome + Xvfb Dockerfile.api # api: no browser docker-compose.yml # local / load testing docker-compose.prod.yml # template for releases (make release) Makefile ``` ## API Synchronous: ``` POST /v1/solve X-API-Key: { "url": "https://freelancehunt.com/projects", "proxy": "http://user:pass@host:port", "captcha_type": "cloudflare", "actions": [ {"type": "wait_for", "selector": "a[href*=\"/project/\"]", "timeout": 60} ], "html": true } ``` Response: ```json { "status": "ok", "user_agent": "Mozilla/5.0 (...) Chrome/155.0.0.0 Safari/537.36", "cookies": [{"name": "cf_clearance", "value": "..."}], "actions": [{"type": "wait_for", "ok": true, "duration_ms": 14}], "html": "......", "duration_ms": 6405 } ``` On failure `status` is `"err"` with a machine-readable `code` (`no_slots`, `request_failed`, `challenge_failed`, `action_failed`) and a human-readable `error`. Asynchronous, keeping the browser window open between tasks: ``` POST /v2/tasks {"url": "https://example.com/login", "actions": [...], "browser_idle_timeout": 10} → 202 {"task_id": "...", "status": "queued", "url": "/v2/tasks/..."} + Set-Cookie: session_id GET /v2/tasks/{id} → {"status": "queued|running|done|failed", "result": {... same as /v1/solve ...}} POST /v2/tasks {"actions": [...], "browser_idle_timeout": 10} # same session: runs in the open window DELETE /v2/sessions/current # close the window, free the slot ``` Full reference — fields, action types, errors, sessions, v2: [docs/API.md](docs/API.md). How to use the API from your client (same UA and proxy as the browser, sessions, retries) — [docs/Usage.md](docs/Usage.md). Python client: [client/README.md](client/README.md). ```bash pip install "robotex @ git+https://git.sokolab.xyz/s0k0l/robotex.git@v0.1.0#subdirectory=client" ``` ## Roadmap Not done yet: - Workers on several hosts (needs shared or per-host routing of browser profiles) - Real billing/usage plans beyond a flat API-key check - Broader action vocabulary - Second engine (Firefox, e.g. Camoufox) for targets where the Chrome fingerprint doesn't fit ## Local development ```bash make sync-offline # venv in ~/.py_envs/robotex; another path: make VENV=... make dev # api + worker on the host, Chrome windows visible on your desktop; API on :8010 make dev-key name=local # API key for it (stored in the same postgres volume as make up) ``` `make dev` stops the containerized api and worker (otherwise the Docker worker also takes tasks from the common queue and runs them invisibly), starts rabbitmq, redis and postgres with ports published on 127.0.0.1 (`docker-compose.dev.yml`: 5673, 6380, 5433), then runs uvicorn with `--reload` and the dramatiq worker in the foreground. Ctrl+C stops both; `make down` stops the infrastructure. Ports, slots: `DEV_API_PORT`, `DEV_RABBITMQ_PORT`, `DEV_REDIS_PORT`, `DEV_POSTGRES_PORT`, `DEV_SLOTS` (default 2). Chrome runs headful, so a display is required (locally — your desktop, in Docker — Xvfb). Tests: ```bash make sync-dev # pytest into .venv make test # unit tests: api + worker + dramatiq, no browser, no RabbitMQ/Redis (~10 s) make test-browser # build the images and run end-to-end tests with real Chrome (needs network) make smoke # check the running stack from make up ``` ## Docker ```bash make up # build robotex-api and robotex-worker, start api, worker, rabbitmq, redis make smoke # health, /v1/solve, /v2/tasks, window holding make logs make down ``` The API listens on `http://localhost:8000`. Browser profiles (`extra/user_data_dir`) and the GeoIP cache live in the worker's `extra` named volume, so sessions survive container restarts. Releases: ```bash make build ver=v0.1.0 # docker images robotex-api:v0.1.0 and robotex-worker:v0.1.0 make release ver=v0.1.0 # push images to docker.sokolab.xyz + deploy bundle in build/v0.1.0/ (compose, .env) make release ver=v0.1.0 out=file # same, but images go to build/v0.1.0/robotex-v0.1.0.tar.gz instead of the registry ``` Details (Xvfb, tini, `/dev/shm`, env settings, load measurements) — [docs/Deployment.md](docs/Deployment.md).