robotex/README.md

# Robotex

Browser automation API with built-in anti-bot stealth. Send a URL and a sequence of declarative actions — fill a field,
click, wait for an element — and get back cookies, the rendered page and, optionally, a screenshot, including sites
behind Cloudflare-style JS challenges.

## Status

Early MVP (v0), under active development. Single browser engine, no billing yet. Tasks and project wiki —
[Redmine](https://redmine.sokolab.xyz/projects/robotex).

Every request needs a client API key in `X-API-Key` (keys are issued with `make key-create`). Usage — requests,
their duration, browser time per client — is recorded in Postgres (`make stats`).

## Architecture

```
Client → Caddy (TLS) → api (FastAPI) ──task──▶ RabbitMQ ──▶ worker (dramatiq, Chrome + Xvfb)
            ▲            │  POST /v2/tasks       common queue        browser slots, open windows
            │            │  POST /v1/solve       or the worker's     pass anti-bot → run actions
            │            ▼                       own queue                    │
            └─── GET /v2/tasks/{id} ◀── Redis: task status and result (TTL) ◀─┘
```

- **Engine**: real Google Chrome (`channel="chrome"`) driven via [Patchright](https://github.com/Kaliiiiiiiiii-Vinyzu/patchright-python)
  (undetected Playwright fork), headful under Xvfb. Browser session wrapper ported from scrapling. Firefox is not
  supported yet.
- **Two services**: `api` (FastAPI, no browser) registers tasks and serves results; `worker` (dramatiq) runs them in
  Chrome. Tasks go through RabbitMQ, statuses and results live in Redis with a TTL.
- **API**: `POST /v2/tasks` + `GET /v2/tasks/{id}` (asynchronous, poll by link), `DELETE /v2/sessions/current`;
  `POST /v1/solve` keeps the synchronous contract on top of the same queue; `GET /health`.
- **Concurrency**: each worker has `BROWSER_SLOTS` Chrome windows. Tasks without a free slot wait in the queue;
  `/v1/solve` answers `503` with `Retry-After` when no slot is free.
- **Window holding**: with `browser_idle_timeout` (minutes, 0–120) the session's Chrome stays open after the task;
  the next task with the same `session_id` is routed to that worker's own queue and runs in the open window, even
  without a `url` (on the current page). Idle time is a Redis key TTL; an open window occupies a slot.
- **Anti-bot**: detect → solve → verify loop for Cloudflare Turnstile and reCAPTCHA; captchas inside pages (Turnstile,
  reCAPTCHA, hCaptcha) are solved via the client's 2captcha key.
- **Proxies**: not managed by the service — the client may supply their own proxy per request. Browser locale and
  timezone are derived from the proxy's exit IP.
- **Sessions**: `session_id` cookie → persistent Chrome profile on disk, removed after `PROFILE_TTL_DAYS` (7) unused days. A fingerprint (screen, DPR, canvas noise seed) is generated on
  the first request of a session; cookies passed in a request replace the profile's cookies.
- **Browser identity is not configurable**: user agent and window size are the real browser's own — a spoofed UA
  mismatches `userAgentData` and Cloudflare blocks it. Clients take `user_agent` from the response.
- **Clients and usage**: Postgres — API keys (sha256 only), a journal of browser windows (opened, last used,
  closed) and of requests (queued, started, finished, outcome, task data without secrets). A task and a window are
  visible only to the client that created them; no keys — no access.
- **Logging**: stdout, level from `LOG_LEVEL`; client secrets (proxy credentials, 2captcha key, `fill` values,
  cookie values) are masked.
- **Deployment**: Docker Compose — `api`, `worker`, `rabbitmq`, `redis`, `postgres`; TLS is terminated by Caddy on the host — see
  [docs/Deployment.md](docs/Deployment.md).

### Project layout

```
src/
  main.py              # api service: /v2/tasks, /v1/solve, /health, sessions
  db/                  # Postgres, shared by api and worker
    models.py          # clients, windows, requests
    clients.py         # API keys: issue, check, revoke
    journal.py         # worker: windows and requests journal
    stats.py, cli.py   # usage report; python -m db.cli create|list|revoke|stats
  tasks/               # shared by api and worker, no browser
    store.py           # Redis: tasks, results, open windows, worker heartbeats
    broker.py          # dramatiq broker (RabbitMQ), queues, messages
    sessions.py        # session_id cookie
  worker/
    actors.py          # dramatiq actors, worker lifecycle, heartbeat
    windows.py         # browser slots, windows held between tasks, task execution
  api/
    profile.py         # per-profile generated fingerprint
    schemas.py         # SolveRequest / TaskRequest / ActionRequest / responses + validation
    actions.py         # action registry and execution
    geoip.py           # proxy → locale / timezone
    logs.py            # request/response dumps without client secrets
  engine/              # Chrome session wrapper (ported from scrapling)
    stealthy.py        # AsyncStealthySession
    session.py         # browser / context launch options
    navigation.py, page_pool.py, validators.py, proxy_rotation.py, ad_domains.py, ...
  antibot/
    orchestrator.py    # detect → solve → verify loop
    registry.py, base.py
    known_providers/   # cloudflare.py, recaptcha.py
    humanize.py        # human-like mouse moves and typing
  captcha/             # 2captcha solvers: Turnstile, reCAPTCHA, hCaptcha
client/                # Python client package robotex (pip install, make client-build)
docs/                  # API, deployment, anti-bot notes
tests/
  unit/                # pytest without a browser: make test
  browser/             # end-to-end with real Chrome in Docker: make test-browser
  harness.py           # api + worker in one process (fakeredis, no RabbitMQ) for tests
  smoke.py             # check a running stack: make smoke
  loadtest.py          # concurrent load test + docker stats
  test_main.http       # manual requests (JetBrains HTTP client)
wheels/                # wheels for offline install on Ubuntu Noble
Dockerfile               # worker: Chrome + Xvfb
Dockerfile.api           # api: no browser
docker-compose.yml       # local / load testing
docker-compose.prod.yml  # template for releases (make release)
Makefile
```

## API

Synchronous:

```
POST /v1/solve
X-API-Key: <key>

{
	"url": "https://freelancehunt.com/projects",
	"proxy": "http://user:pass@host:port",
	"captcha_type": "cloudflare",
	"actions": [
		{"type": "wait_for", "selector": "a[href*=\"/project/\"]", "timeout": 60}
	],
	"html": true
}
```

Response:

```json
{
	"status": "ok",
	"user_agent": "Mozilla/5.0 (...) Chrome/155.0.0.0 Safari/537.36",
	"cookies": [{"name": "cf_clearance", "value": "..."}],
	"actions": [{"type": "wait_for", "ok": true, "duration_ms": 14}],
	"html": "<head>...</head><body>...</body>",
	"duration_ms": 6405
}
```

On failure `status` is `"err"` with a machine-readable `code` (`no_slots`, `request_failed`, `challenge_failed`,
`action_failed`) and a human-readable `error`.

Asynchronous, keeping the browser window open between tasks:

```
POST /v2/tasks          {"url": "https://example.com/login", "actions": [...], "browser_idle_timeout": 10}
→ 202 {"task_id": "...", "status": "queued", "url": "/v2/tasks/..."}  + Set-Cookie: session_id
GET  /v2/tasks/{id}     → {"status": "queued|running|done|failed", "result": {... same as /v1/solve ...}}
POST /v2/tasks          {"actions": [...], "browser_idle_timeout": 10}   # same session: runs in the open window
DELETE /v2/sessions/current                                              # close the window, free the slot
```

Full reference — fields, action types, errors, sessions, v2:
[docs/API.md](docs/API.md). How to use the API from your client (same UA and proxy as the browser, sessions,
retries) — [docs/Usage.md](docs/Usage.md).

Python client: [client/README.md](client/README.md).

```bash
pip install "robotex @ git+https://git.sokolab.xyz/s0k0l/robotex.git@v0.1.0#subdirectory=client"
```

## Roadmap

Not done yet:

- Workers on several hosts (needs shared or per-host routing of browser profiles)
- Real billing/usage plans beyond a flat API-key check
- Broader action vocabulary
- Second engine (Firefox, e.g. Camoufox) for targets where the Chrome fingerprint doesn't fit

## Local development

```bash
make sync-offline        # venv in ~/.py_envs/robotex; another path: make VENV=... <target>
docker compose up -d rabbitmq redis                       # queue and storage
cd src && RABBITMQ_URL=... REDIS_URL=... uvicorn main:app --reload
cd src && dramatiq worker.actors --processes 1 --threads 4
```

The compose network is not published, so for running the services on the host publish `5672` and `6379` yourself,
or just use `make up`. Chrome runs headful, so a display is required (locally — your desktop, in Docker — Xvfb).

Tests:

```bash
make sync-dev        # pytest into .venv
make test            # unit tests: api + worker + dramatiq, no browser, no RabbitMQ/Redis (~10 s)
make test-browser    # build the images and run end-to-end tests with real Chrome (needs network)
make smoke           # check the running stack from make up
```

## Docker

```bash
make up      # build robotex-api and robotex-worker, start api, worker, rabbitmq, redis
make smoke   # health, /v1/solve, /v2/tasks, window holding
make logs
make down
```

The API listens on `http://localhost:8000`. Browser profiles (`extra/user_data_dir`) and the GeoIP cache live in the
worker's `extra` named volume, so sessions survive container restarts.

Releases:

```bash
make build ver=v0.1.0      # docker images robotex-api:v0.1.0 and robotex-worker:v0.1.0
make release ver=v0.1.0    # images + deploy bundle in build/v0.1.0/ (compose, .env, images .tar.gz)
```

Details (Xvfb, tini, `/dev/shm`, env settings, load measurements) — [docs/Deployment.md](docs/Deployment.md).