s0k0l/robotex
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feature: архитектура v2 — api + очередь + worker, удержание окна браузера (#78)

Browse Source 
- Сервис api (FastAPI, без Chrome): POST /v2/tasks, GET /v2/tasks/{id} (доступ по cookie
  сессии), DELETE /v2/sessions/current; /v1/solve — прежний контракт поверх очереди.
- Сервис worker (dramatiq + RabbitMQ): слоты браузера, окна сессий между задачами
  (browser_idle_timeout 0–120 мин, простой — TTL ключа в Redis), heartbeat.
- Маршрутизация задач сессии в персональную очередь воркера с открытым окном.
- Redis: задачи, результаты (RESULT_TTL_HOURS), окна, воркеры. Прокси и ключ 2captcha
  в Redis не сохраняются.
- Движок: AsyncStealthySession.run() — действия на текущей странице без навигации.
- Два образа (robotex-api, robotex-worker), compose с rabbitmq и redis,
  make up/down/logs/smoke, релиз собирает все образы.
- Тесты: api + worker в одном процессе (fakeredis), настоящий воркер dramatiq на
  StubBroker, браузерные тесты на новой схеме, smoke для запущенного стенда.
- Документация: README, API (v2), Deployment, Usage.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
protokeyandClaude Opus 5.5 committed 2026-10-13 19:13:52 +04:00
1 parent fe509c25c3
commit 7b511d5d38
38 files changed
+2142 -545
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

README.md
+63 -28
View File
@@ -6,7 +6,7 @@ behind Cloudflare-style JS challenges.
## Status
Early MVP (v0), under active development. Single browser engine, no queue, no billing yet. Tasks and project wiki —
Early MVP (v0), under active development. Single browser engine, no billing yet. Tasks and project wiki —
[Redmine](https://redmine.sokolab.xyz/projects/robotex).
Not implemented yet, though planned for v1:
@@ -18,22 +18,25 @@ Not implemented yet, though planned for v1:
## Architecture
```
Client → Caddy (TLS, on the host) → FastAPI POST /v1/solve → browser slot (semaphore, 503 if none)
│
▼
Chrome (Patchright), profile per session_id
pass anti-bot → run actions → collect result
│
▼
release slot → response
Client → Caddy (TLS) → api (FastAPI) ──task──▶ RabbitMQ ──▶ worker (dramatiq, Chrome + Xvfb)
▲ │ POST /v2/tasks common queue browser slots, open windows
│ │ POST /v1/solve or the worker's pass anti-bot → run actions
│ ▼ own queue │
└─── GET /v2/tasks/{id} ◀── Redis: task status and result (TTL) ◀─┘
```
- **Engine**: real Google Chrome (`channel="chrome"`) driven via [Patchright](https://github.com/Kaliiiiiiiiii-Vinyzu/patchright-python)
(undetected Playwright fork), headful under Xvfb. Browser session wrapper ported from scrapling. Firefox is not
supported yet.
- **API**: FastAPI, single versioned endpoint (`POST /v1/solve`) plus `GET /health`.
- **Concurrency**: bounded by an `asyncio.Semaphore` of `BROWSER_SLOTS` (env, default 10) — no external job queue in
v1. When all slots are busy the request gets `503` with `Retry-After` immediately, it is not queued.
- **Two services**: `api` (FastAPI, no browser) registers tasks and serves results; `worker` (dramatiq) runs them in
Chrome. Tasks go through RabbitMQ, statuses and results live in Redis with a TTL.
- **API**: `POST /v2/tasks` + `GET /v2/tasks/{id}` (asynchronous, poll by link), `DELETE /v2/sessions/current`;
`POST /v1/solve` keeps the synchronous contract on top of the same queue; `GET /health`.
- **Concurrency**: each worker has `BROWSER_SLOTS` Chrome windows. Tasks without a free slot wait in the queue;
`/v1/solve` answers `503` with `Retry-After` when no slot is free.
- **Window holding**: with `browser_idle_timeout` (minutes, 0–120) the session's Chrome stays open after the task;
the next task with the same `session_id` is routed to that worker's own queue and runs in the open window, even
without a `url` (on the current page). Idle time is a Redis key TTL; an open window occupies a slot.
- **Anti-bot**: detect → solve → verify loop for Cloudflare Turnstile and reCAPTCHA; captchas inside pages (Turnstile,
reCAPTCHA, hCaptcha) are solved via the client's 2captcha key.
- **Proxies**: not managed by the service — the client may supply their own proxy per request. Browser locale and
@@ -44,19 +47,27 @@ Client → Caddy (TLS, on the host) → FastAPI POST /v1/solve → browser slot
mismatches `userAgentData` and Cloudflare blocks it. Clients take `user_agent` from the response.
- **Logging**: stdout, level from `LOG_LEVEL`; client secrets (proxy credentials, 2captcha key, `fill` values,
cookie values) are masked.
- **Deployment**: single Docker container; TLS is terminated by Caddy on the host — see
- **Deployment**: Docker Compose — `api`, `worker`, `rabbitmq`, `redis`; TLS is terminated by Caddy on the host — see
[docs/Deployment.md](docs/Deployment.md).
### Project layout
```
src/
main.py # FastAPI app: /v1/solve, /health, slots, sessions, response
main.py # api service: /v2/tasks, /v1/solve, /health, sessions
tasks/ # shared by api and worker, no browser
store.py # Redis: tasks, results, open windows, worker heartbeats
broker.py # dramatiq broker (RabbitMQ), queues, messages
sessions.py # session_id cookie
worker/
actors.py # dramatiq actors, worker lifecycle, heartbeat
windows.py # browser slots, windows held between tasks, task execution
api/
profile.py # per-profile user agent and generated fingerprint
schemas.py # SolveRequest / ActionRequest / SolveResponse + validation
profile.py # per-profile generated fingerprint
schemas.py # SolveRequest / TaskRequest / ActionRequest / responses + validation
actions.py # action registry and execution
geoip.py # proxy → locale / timezone
logs.py # request/response dumps without client secrets
engine/ # Chrome session wrapper (ported from scrapling)
stealthy.py # AsyncStealthySession
session.py # browser / context launch options
@@ -71,10 +82,13 @@ docs/ # API, deployment, anti-bot notes
tests/
unit/ # pytest without a browser: make test
browser/ # end-to-end with real Chrome in Docker: make test-browser
harness.py # api + worker in one process (fakeredis, no RabbitMQ) for tests
smoke.py # check a running stack: make smoke
loadtest.py # concurrent load test + docker stats
test_main.http # manual requests (JetBrains HTTP client)
wheels/ # wheels for offline install on Ubuntu Noble
Dockerfile
Dockerfile # worker: Chrome + Xvfb
Dockerfile.api # api: no browser
docker-compose.yml # local / load testing
docker-compose.prod.yml # template for releases (make release)
Makefile
@@ -82,6 +96,8 @@ Makefile
## API
Synchronous:
```
POST /v1/solve
X-API-Key: <key>
@@ -111,15 +127,27 @@ Response:
```
On failure `status` is `"err"` with a machine-readable `code` (`no_slots`, `request_failed`, `challenge_failed`,
`action_failed`) and a human-readable `error`. Full reference — fields, action types, errors, sessions:
`action_failed`) and a human-readable `error`.
Asynchronous, keeping the browser window open between tasks:
```
POST /v2/tasks {"url": "https://example.com/login", "actions": [...], "browser_idle_timeout": 10}
→ 202 {"task_id": "...", "status": "queued", "url": "/v2/tasks/..."} + Set-Cookie: session_id
GET /v2/tasks/{id} → {"status": "queued|running|done|failed", "result": {... same as /v1/solve ...}}
POST /v2/tasks {"actions": [...], "browser_idle_timeout": 10} # same session: runs in the open window
DELETE /v2/sessions/current # close the window, free the slot
```
Full reference — fields, action types, errors, sessions, v2:
[docs/API.md](docs/API.md). How to use the API from your client (same UA and proxy as the browser, sessions,
retries) — [docs/Usage.md](docs/Usage.md).
## Roadmap
Deliberately out of scope for v1:
Not done yet:
- Job queue (Redis/RabbitMQ) + multiple worker VPS for horizontal scaling
- Workers on several hosts (needs shared or per-host routing of browser profiles)
- Real billing/usage plans beyond a flat API-key check
- Broader action vocabulary
- Second engine (Firefox, e.g. Camoufox) for targets where the Chrome fingerprint doesn't fit
@@ -130,33 +158,40 @@ Deliberately out of scope for v1:
python -m venv .venv
source .venv/bin/activate
make sync-offline
cd src && uvicorn main:app --reload
docker compose up -d rabbitmq redis # queue and storage
cd src && RABBITMQ_URL=... REDIS_URL=... uvicorn main:app --reload
cd src && dramatiq worker.actors --processes 1 --threads 4
```
Chrome runs headful, so a display is required (locally — your desktop, in Docker — Xvfb).
The compose network is not published, so for running the services on the host publish `5672` and `6379` yourself,
or just use `make up`. Chrome runs headful, so a display is required (locally — your desktop, in Docker — Xvfb).
Tests:
```bash
make sync-dev # pytest into .venv
make test # unit tests, no browser (~5 s)
make test-browser # build the image and run end-to-end tests with real Chrome (needs network)
make test # unit tests: api + worker + dramatiq, no browser, no RabbitMQ/Redis (~10 s)
make test-browser # build the images and run end-to-end tests with real Chrome (needs network)
make smoke # check the running stack from make up
```
## Docker
```bash
docker compose up -d --build
make up # build robotex-api and robotex-worker, start api, worker, rabbitmq, redis
make smoke # health, /v1/solve, /v2/tasks, window holding
make logs
make down
```
The API listens on `http://localhost:8000`. Browser profiles (`extra/user_data_dir`) and the GeoIP cache live in the
`extra` named volume, so sessions survive container restarts.
worker's `extra` named volume, so sessions survive container restarts.
Releases:
```bash
make build ver=v0.0.2 # docker image robotex:v0.0.2
make release ver=v0.0.2 # image + deploy bundle in build/v0.0.2/ (compose, .env, image .tar.gz)
make build ver=v0.1.0 # docker images robotex-api:v0.1.0 and robotex-worker:v0.1.0
make release ver=v0.1.0 # images + deploy bundle in build/v0.1.0/ (compose, .env, images .tar.gz)
```
Details (Xvfb, tini, `/dev/shm`, env settings, load measurements) — [docs/Deployment.md](docs/Deployment.md).