robotex/README.md

# Robotex

Browser automation API with built-in anti-bot stealth. Send a URL and a sequence of declarative actions — fill a field,
click, wait for an element — and get back cookies, the rendered page and, optionally, a screenshot, including sites
behind Cloudflare-style JS challenges.

## Status

Early MVP (v0), under active development. Single browser engine, no billing yet. Tasks and project wiki —
[Redmine](https://redmine.sokolab.xyz/projects/robotex).

Every request needs a client API key in `X-API-Key` (keys are issued with `make key-create`). Usage — requests,
their duration, browser time per client — is recorded in Postgres (`make stats`).

## Architecture

```
Client → Caddy (TLS) → api (FastAPI) ──task──▶ RabbitMQ ──▶ worker (dramatiq, Chrome + Xvfb)
            ▲            │  POST /v2/tasks       common queue        browser slots, open windows
            │            │  POST /v1/solve       or the worker's     pass anti-bot → run actions
            │            ▼                       own queue                    │
            └─── GET /v2/tasks/{id} ◀── Redis: task status and result (TTL) ◀─┘
```

- **Engine**: real Google Chrome (`channel="chrome"`) driven via [Patchright](https://github.com/Kaliiiiiiiiii-Vinyzu/patchright-python)
  (undetected Playwright fork), headful under Xvfb. Browser session wrapper ported from scrapling. Firefox is not
  supported yet.
- **Two services**: `api` (FastAPI, no browser) registers tasks and serves results; `worker` (dramatiq) runs them in
  Chrome. Tasks go through RabbitMQ, statuses and results live in Redis with a TTL.
- **API**: `POST /v2/tasks` + `GET /v2/tasks/{id}` (asynchronous, poll by link), `DELETE /v2/sessions/current`;
  `POST /v1/solve` keeps the synchronous contract on top of the same queue; `GET /health`.
- **Concurrency**: each worker has `BROWSER_SLOTS` Chrome windows. Tasks without a free slot wait in the queue;
  `/v1/solve` answers `503` with `Retry-After` when no slot is free.
- **Window holding**: with `browser_idle_timeout` (minutes, 0–120) the session's Chrome stays open after the task;
  the next task with the same `session_id` is routed to that worker's own queue and runs in the open window, even
  without a `url` (on the current page). Idle time is a Redis key TTL; an open window occupies a slot.
- **Anti-bot**: detect → solve → verify loop for Cloudflare Turnstile and reCAPTCHA; captchas inside pages (Turnstile,
  reCAPTCHA, hCaptcha) are solved via the client's 2captcha key.
- **Proxies**: not managed by the service — the client may supply their own proxy per request. Browser locale and
  timezone are derived from the proxy's exit IP.
- **Sessions**: `session_id` cookie → persistent Chrome profile on disk, removed after `PROFILE_TTL_DAYS` (7) unused days. A fingerprint (screen, DPR, canvas noise seed) is generated on
  the first request of a session; cookies passed in a request replace the profile's cookies.
- **Browser identity is not configurable**: user agent and window size are the real browser's own — a spoofed UA
  mismatches `userAgentData` and Cloudflare blocks it. Clients take `user_agent` from the response.
- **Clients and usage**: Postgres — API keys (sha256 only), a journal of browser windows (opened, last used,
  closed) and of requests (queued, started, finished, outcome, task data without secrets). A task and a window are
  visible only to the client that created them; no keys — no access.
- **Logging**: stdout, level from `LOG_LEVEL`; client secrets (proxy credentials, 2captcha key, `fill` values,
  cookie values) are masked.
- **Deployment**: Docker Compose — `api`, `worker`, `rabbitmq`, `redis`, `postgres`; TLS is terminated by Caddy on the host — see
  [docs/Deployment.md](docs/Deployment.md).

### Project layout

```
src/
  main.py              # api service: /v2/tasks, /v1/solve, /health, sessions
  db/                  # Postgres, shared by api and worker
    models.py          # clients, windows, requests
    clients.py         # API keys: issue, check, revoke
    journal.py         # worker: windows and requests journal
    stats.py, cli.py   # usage report; python -m db.cli create|list|revoke|stats
  tasks/               # shared by api and worker, no browser
    store.py           # Redis: tasks, results, open windows, worker heartbeats
    broker.py          # dramatiq broker (RabbitMQ), queues, messages
    sessions.py        # session_id cookie
  worker/
    actors.py          # dramatiq actors, worker lifecycle, heartbeat
    windows.py         # browser slots, windows held between tasks, task execution
  api/
    profile.py         # per-profile generated fingerprint
    schemas.py         # SolveRequest / TaskRequest / ActionRequest / responses + validation
    actions.py         # action registry and execution
    geoip.py           # proxy → locale / timezone
    logs.py            # request/response dumps without client secrets
  engine/              # Chrome session wrapper (ported from scrapling)
    stealthy.py        # AsyncStealthySession
    session.py         # browser / context launch options
    navigation.py, page_pool.py, validators.py, proxy_rotation.py, ad_domains.py, ...
  antibot/
    orchestrator.py    # detect → solve → verify loop
    registry.py, base.py
    known_providers/   # cloudflare.py, recaptcha.py
    humanize.py        # human-like mouse moves and typing
  captcha/             # 2captcha solvers: Turnstile, reCAPTCHA, hCaptcha
client/                # Python client package robotex (pip install, make client-build)
docs/                  # API, deployment, anti-bot notes
tests/
  unit/                # pytest without a browser: make test
  browser/             # end-to-end with real Chrome in Docker: make test-browser
  harness.py           # api + worker in one process (fakeredis, no RabbitMQ) for tests
  smoke.py             # check a running stack: make smoke
  loadtest.py          # concurrent load test + docker stats
  test_main.http       # manual requests (JetBrains HTTP client)
wheels/                # wheels for offline install on Ubuntu Noble
Dockerfile               # worker: Chrome + Xvfb
Dockerfile.api           # api: no browser
docker-compose.yml       # local / load testing
docker-compose.prod.yml  # template for releases (make release)
Makefile
```

## API

Synchronous:

```
POST /v1/solve
X-API-Key: <key>

{
	"url": "https://freelancehunt.com/projects",
	"proxy": "http://user:pass@host:port",
	"captcha_type": "cloudflare",
	"actions": [
		{"type": "wait_for", "selector": "a[href*=\"/project/\"]", "timeout": 60}
	],
	"html": true
}
```

Response:

```json
{
	"status": "ok",
	"user_agent": "Mozilla/5.0 (...) Chrome/155.0.0.0 Safari/537.36",
	"cookies": [{"name": "cf_clearance", "value": "..."}],
	"actions": [{"type": "wait_for", "ok": true, "duration_ms": 14}],
	"html": "<head>...</head><body>...</body>",
	"duration_ms": 6405
}
```

On failure `status` is `"err"` with a machine-readable `code` (`no_slots`, `request_failed`, `challenge_failed`,
`action_failed`) and a human-readable `error`.

Asynchronous, keeping the browser window open between tasks:

```
POST /v2/tasks          {"url": "https://example.com/login", "actions": [...], "browser_idle_timeout": 10}
→ 202 {"task_id": "...", "status": "queued", "url": "/v2/tasks/..."}  + Set-Cookie: session_id
GET  /v2/tasks/{id}     → {"status": "queued|running|done|failed", "result": {... same as /v1/solve ...}}
POST /v2/tasks          {"actions": [...], "browser_idle_timeout": 10}   # same session: runs in the open window
DELETE /v2/sessions/current                                              # close the window, free the slot
```

Full reference — fields, action types, errors, sessions, v2:
[docs/API.md](docs/API.md). How to use the API from your client (same UA and proxy as the browser, sessions,
retries) — [docs/Usage.md](docs/Usage.md).

Python client: [client/README.md](client/README.md).

```bash
pip install "robotex @ git+https://git.sokolab.xyz/s0k0l/robotex.git@v0.1.0#subdirectory=client"
```

## Roadmap

Not done yet:

- Workers on several hosts (needs shared or per-host routing of browser profiles)
- Real billing/usage plans beyond a flat API-key check
- Broader action vocabulary
- Second engine (Firefox, e.g. Camoufox) for targets where the Chrome fingerprint doesn't fit

## Local development

```bash
make sync-offline              # venv in ~/.py_envs/robotex; another path: make VENV=... <target>
make dev                       # api + worker on the host, Chrome windows visible on your desktop; API on :8010
make dev-key name=local        # API key for it (stored in the same postgres volume as make up)
```

`make dev` stops the containerized api and worker (otherwise the Docker worker also takes tasks from the common
queue and runs them invisibly), starts rabbitmq, redis and postgres with ports published on 127.0.0.1
(`docker-compose.dev.yml`: 5673, 6380, 5433), then runs uvicorn with `--reload` and the dramatiq worker in the
foreground. Ctrl+C stops both; `make down` stops the infrastructure. Ports, slots: `DEV_API_PORT`,
`DEV_RABBITMQ_PORT`, `DEV_REDIS_PORT`, `DEV_POSTGRES_PORT`, `DEV_SLOTS` (default 2). Chrome runs headful, so a display
is required (locally — your desktop, in Docker — Xvfb).

Tests:

```bash
make sync-dev        # pytest into .venv
make test            # unit tests: api + worker + dramatiq, no browser, no RabbitMQ/Redis (~10 s)
make test-browser    # build the images and run end-to-end tests with real Chrome (needs network)
make smoke           # check the running stack from make up
```

## Docker

```bash
make up      # build robotex-api and robotex-worker, start api, worker, rabbitmq, redis
make smoke   # health, /v1/solve, /v2/tasks, window holding
make logs
make down
```

The API listens on `http://localhost:8000`. Browser profiles (`extra/user_data_dir`) and the GeoIP cache live in the
worker's `extra` named volume, so sessions survive container restarts.

Releases:

```bash
make build ver=v0.1.0      # docker images robotex-api:v0.1.0 and robotex-worker:v0.1.0
make release ver=v0.1.0    # push images to docker.sokolab.xyz + deploy bundle in build/v0.1.0/ (compose, .env)
make release ver=v0.1.0 out=file  # same, but images go to build/v0.1.0/robotex-v0.1.0.tar.gz instead of the registry
```

Details (Xvfb, tini, `/dev/shm`, env settings, load measurements) — [docs/Deployment.md](docs/Deployment.md).