Robotex

Browser automation API with built-in anti-bot stealth. Send a URL and a sequence of declarative actions — fill a field, click, wait for an element — and get back cookies, the rendered page and, optionally, a screenshot, including sites behind Cloudflare-style JS challenges.

Status

Early MVP (v0), under active development. Single browser engine, no queue, no billing yet. Tasks and project wiki — Redmine.

Not implemented yet, though planned for v1:

• API-key auth (X-API-Key is accepted but not checked) — #4
• persisting request outcomes (success rate) — #17
• cleanup of old browser profiles — #20

Architecture

Client → Caddy (TLS, on the host) → FastAPI POST /v1/solve → browser slot (semaphore, 503 if none)
                                                                     │
                                                                     ▼
                                                      Chrome (Patchright), profile per session_id
                                                      pass anti-bot → run actions → collect result
                                                                     │
                                                                     ▼
                                                      release slot → response

• Engine: real Google Chrome (channel="chrome") driven via Patchright (undetected Playwright fork), headful under Xvfb. Browser session wrapper ported from scrapling. Firefox is not supported yet.
• API: FastAPI, single versioned endpoint (POST /v1/solve) plus GET /health.
• Concurrency: bounded by an asyncio.Semaphore of BROWSER_SLOTS (env, default 10) — no external job queue in v1. When all slots are busy the request gets 503 with Retry-After immediately, it is not queued.
• Anti-bot: detect → solve → verify loop for Cloudflare Turnstile and reCAPTCHA; captchas inside pages (Turnstile, reCAPTCHA, hCaptcha) are solved via the client's 2captcha key.
• Proxies: not managed by the service — the client may supply their own proxy per request. Browser locale and timezone are derived from the proxy's exit IP.
• Sessions: session_id cookie → persistent Chrome profile on disk. User agent and fingerprint (screen, DPR) are fixed on the first request of a session; cookies passed in a request replace the profile's cookies.
• Logging: stdout, level from LOG_LEVEL; client secrets (proxy credentials, 2captcha key, fill values, cookie values) are masked.
• Deployment: single Docker container; TLS is terminated by Caddy on the host — see docs/Deployment.md.

Project layout

src/
  main.py              # FastAPI app: /v1/solve, /health, slots, sessions, response
  api/
    profile.py         # per-profile user agent and generated fingerprint
    schemas.py         # SolveRequest / ActionRequest / SolveResponse + validation
    actions.py         # action registry and execution
    geoip.py           # proxy → locale / timezone
  engine/              # Chrome session wrapper (ported from scrapling)
    stealthy.py        # AsyncStealthySession
    session.py         # browser / context launch options
    navigation.py, page_pool.py, validators.py, proxy_rotation.py, ad_domains.py, ...
  antibot/
    orchestrator.py    # detect → solve → verify loop
    registry.py, base.py
    known_providers/   # cloudflare.py, recaptcha.py
    humanize.py        # human-like mouse moves and typing
  captcha/             # 2captcha solvers: Turnstile, reCAPTCHA, hCaptcha
docs/                  # API, deployment, anti-bot notes
tests/
  unit/                # pytest without a browser: make test
  browser/             # end-to-end with real Chrome in Docker: make test-browser
  loadtest.py          # concurrent load test + docker stats
  test_main.http       # manual requests (JetBrains HTTP client)
wheels/                # wheels for offline install on Ubuntu Noble
Dockerfile
docker-compose.yml       # local / load testing
docker-compose.prod.yml  # template for releases (make release)
Makefile

API

POST /v1/solve
X-API-Key: <key>

{
	"url": "https://freelancehunt.com/projects",
	"proxy": "http://user:pass@host:port",
	"captcha_type": "cloudflare",
	"actions": [
		{"type": "wait_for", "selector": "a[href*=\"/project/\"]", "timeout": 60}
	],
	"html": true
}

Response:

{
	"status": "ok",
	"user_agent": "Mozilla/5.0 (...) Chrome/155.0.0.0 Safari/537.36",
	"cookies": [{"name": "cf_clearance", "value": "..."}],
	"actions": [{"type": "wait_for", "ok": true, "duration_ms": 14}],
	"html": "<head>...</head><body>...</body>",
	"duration_ms": 6405
}

On failure status is "err" with a machine-readable code (no_slots, request_failed, challenge_failed, action_failed) and a human-readable error. Full reference — fields, action types, errors, sessions: docs/API.md.

Roadmap

Deliberately out of scope for v1:

• Job queue (Redis/RabbitMQ) + multiple worker VPS for horizontal scaling
• Real billing/usage plans beyond a flat API-key check
• Broader action vocabulary
• Second engine (Firefox, e.g. Camoufox) for targets where the Chrome fingerprint doesn't fit

Local development

python -m venv .venv
source .venv/bin/activate
make sync-offline
cd src && uvicorn main:app --reload

Chrome runs headful, so a display is required (locally — your desktop, in Docker — Xvfb).

Tests:

make sync-dev        # pytest into .venv
make test            # unit tests, no browser (~5 s)
make test-browser    # build the image and run end-to-end tests with real Chrome (needs network)

Docker

docker compose up -d --build

The API listens on http://localhost:8000. Browser profiles (extra/user_data_dir) and the GeoIP cache live in the extra named volume, so sessions survive container restarts.

Releases:

make build ver=v0.0.2      # docker image robotex:v0.0.2
make release ver=v0.0.2    # image + deploy bundle in build/v0.0.2/ (compose, .env, image .tar.gz)

Details (Xvfb, tini, /dev/shm, env settings, load measurements) — docs/Deployment.md.