robotex/README.md

# Robotex

Browser automation API with built-in anti-bot stealth. Send a URL and a sequence of declarative actions — fill a field,
click, wait for an element — and get back cookies, the rendered page and, optionally, a screenshot, including sites
behind Cloudflare-style JS challenges.

## Status

Early MVP (v0), under active development. Single browser engine, no queue, no billing yet. Tasks and project wiki —
[Redmine](https://redmine.sokolab.xyz/projects/robotex).

Not implemented yet, though planned for v1:

- API-key auth (`X-API-Key` is accepted but **not checked**) — [#4](https://redmine.sokolab.xyz/issues/4)
- persisting request outcomes (success rate) — [#17](https://redmine.sokolab.xyz/issues/17)
- cleanup of old browser profiles — [#20](https://redmine.sokolab.xyz/issues/20)

## Architecture

```
Client → Caddy (TLS, on the host) → FastAPI POST /v1/solve → browser slot (semaphore, 503 if none)
                                                                     │
                                                                     ▼
                                                      Chrome (Patchright), profile per session_id
                                                      pass anti-bot → run actions → collect result
                                                                     │
                                                                     ▼
                                                      release slot → response
```

- **Engine**: real Google Chrome (`channel="chrome"`) driven via [Patchright](https://github.com/Kaliiiiiiiiii-Vinyzu/patchright-python)
  (undetected Playwright fork), headful under Xvfb. Browser session wrapper ported from scrapling. Firefox is not
  supported yet.
- **API**: FastAPI, single versioned endpoint (`POST /v1/solve`) plus `GET /health`.
- **Concurrency**: bounded by an `asyncio.Semaphore` of `BROWSER_SLOTS` (env, default 10) — no external job queue in
  v1. When all slots are busy the request gets `503` with `Retry-After` immediately, it is not queued.
- **Anti-bot**: detect → solve → verify loop for Cloudflare Turnstile and reCAPTCHA; captchas inside pages (Turnstile,
  reCAPTCHA, hCaptcha) are solved via the client's 2captcha key.
- **Proxies**: not managed by the service — the client may supply their own proxy per request. Browser locale and
  timezone are derived from the proxy's exit IP.
- **Sessions**: `session_id` cookie → persistent Chrome profile on disk. User agent and fingerprint (screen, DPR) are
  fixed on the first request of a session; cookies passed in a request replace the profile's cookies.
- **Logging**: stdout, level from `LOG_LEVEL`; client secrets (proxy credentials, 2captcha key, `fill` values,
  cookie values) are masked.
- **Deployment**: single Docker container; TLS is terminated by Caddy on the host — see
  [docs/Deployment.md](docs/Deployment.md).

### Project layout

```
src/
  main.py              # FastAPI app: /v1/solve, /health, slots, sessions, response
  api/
    profile.py         # per-profile user agent and generated fingerprint
    schemas.py         # SolveRequest / ActionRequest / SolveResponse + validation
    actions.py         # action registry and execution
    geoip.py           # proxy → locale / timezone
  engine/              # Chrome session wrapper (ported from scrapling)
    stealthy.py        # AsyncStealthySession
    session.py         # browser / context launch options
    navigation.py, page_pool.py, validators.py, proxy_rotation.py, ad_domains.py, ...
  antibot/
    orchestrator.py    # detect → solve → verify loop
    registry.py, base.py
    known_providers/   # cloudflare.py, recaptcha.py
    humanize.py        # human-like mouse moves and typing
  captcha/             # 2captcha solvers: Turnstile, reCAPTCHA, hCaptcha
docs/                  # API, deployment, anti-bot notes
tests/
  unit/                # pytest without a browser: make test
  browser/             # end-to-end with real Chrome in Docker: make test-browser
  loadtest.py          # concurrent load test + docker stats
  test_main.http       # manual requests (JetBrains HTTP client)
wheels/                # wheels for offline install on Ubuntu Noble
Dockerfile
docker-compose.yml       # local / load testing
docker-compose.prod.yml  # template for releases (make release)
Makefile
```

## API

```
POST /v1/solve
X-API-Key: <key>

{
	"url": "https://freelancehunt.com/projects",
	"proxy": "http://user:pass@host:port",
	"captcha_type": "cloudflare",
	"actions": [
		{"type": "wait_for", "selector": "a[href*=\"/project/\"]", "timeout": 60}
	],
	"html": true
}
```

Response:

```json
{
	"status": "ok",
	"user_agent": "Mozilla/5.0 (...) Chrome/155.0.0.0 Safari/537.36",
	"cookies": [{"name": "cf_clearance", "value": "..."}],
	"actions": [{"type": "wait_for", "ok": true, "duration_ms": 14}],
	"html": "<head>...</head><body>...</body>",
	"duration_ms": 6405
}
```

On failure `status` is `"err"` with a machine-readable `code` (`no_slots`, `request_failed`, `challenge_failed`,
`action_failed`) and a human-readable `error`. Full reference — fields, action types, errors, sessions:
[docs/API.md](docs/API.md).

## Roadmap

Deliberately out of scope for v1:

- Job queue (Redis/RabbitMQ) + multiple worker VPS for horizontal scaling
- Real billing/usage plans beyond a flat API-key check
- Broader action vocabulary
- Second engine (Firefox, e.g. Camoufox) for targets where the Chrome fingerprint doesn't fit

## Local development

```bash
python -m venv .venv
source .venv/bin/activate
make sync-offline
cd src && uvicorn main:app --reload
```

Chrome runs headful, so a display is required (locally — your desktop, in Docker — Xvfb).

Tests:

```bash
make sync-dev        # pytest into .venv
make test            # unit tests, no browser (~5 s)
make test-browser    # build the image and run end-to-end tests with real Chrome (needs network)
```

## Docker

```bash
docker compose up -d --build
```

The API listens on `http://localhost:8000`. Browser profiles (`extra/user_data_dir`) and the GeoIP cache live in the
`extra` named volume, so sessions survive container restarts.

Releases:

```bash
make build ver=v0.0.2      # docker image robotex:v0.0.2
make release ver=v0.0.2    # image + deploy bundle in build/v0.0.2/ (compose, .env, image .tar.gz)
```

Details (Xvfb, tini, `/dev/shm`, env settings, load measurements) — [docs/Deployment.md](docs/Deployment.md).