chore: keep social announcements in social/<post-slug>/
The dev.to text for the Robotex post moves here from the untracked drafts/. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
protokeyandClaude Opus 5.5 committed
1 parent
06647e39c3
commit
ee17899090
1 file changed
+181
@@ -0,0 +1,181 @@
|
||||
---
|
||||
title: "Robotex: a real browser for your bots, behind a plain HTTP API"
|
||||
published: false
|
||||
description: "Robotex passes anti-bot checks in real Chrome, runs your scenario on the page and hands your bot the cookies. After that, the bot goes back to cheap HTTP requests."
|
||||
tags: webscraping, python, automation, api
|
||||
---
|
||||
|
||||
Every bot that works with a website sooner or later hits one of three walls:
|
||||
|
||||
1. a captcha;
|
||||
2. anti-bot protection (Cloudflare's "Just a moment…" page, Turnstile, reCAPTCHA);
|
||||
3. heavy JavaScript that has to run before you can even submit a form.
|
||||
|
||||
A spider built on raw HTTP requests is helpless here. You need a real browser. But a browser is heavy: it eats half a gigabyte of RAM per window, it has to be hidden from detection, and it has to live next to your bot. Most of the time, though, the bot doesn't need the browser at all. It only needs it to get *through the door*.
|
||||
|
||||
That's the idea behind **[Robotex](https://robotex.sokolab.xyz/)**: a browser-as-a-service that you call only where there's no way around it.
|
||||
|
||||
## The idea: the browser opens the door, the bot walks in
|
||||
|
||||
Robotex opens the page in real Chrome, passes the anti-bot checks, runs a declarative scenario (fill, click, wait, grab HTML or text) and returns:
|
||||
|
||||
- the **cookies**, including `cf_clearance`;
|
||||
- the browser's own **User-Agent**;
|
||||
- optionally, the rendered HTML and a screenshot.
|
||||
|
||||
From there your bot keeps working with a plain HTTP client, which is fast and cheap. When the site shows a challenge again, the bot calls Robotex once more in the same session.
|
||||
|
||||
It's a service, not a library, so the bot can be written in anything that speaks HTTP.
|
||||
|
||||
## A first request
|
||||
|
||||
The synchronous endpoint is the simplest way in:
|
||||
|
||||
```http
|
||||
POST https://app.robotex.sokolab.xyz/v1/solve
|
||||
X-API-Key: <your key>
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"url": "https://example.com/login",
|
||||
"proxy": "http://user:pass@host:port",
|
||||
"captcha_type": "cloudflare",
|
||||
"actions": [
|
||||
{"type": "fill", "selector": "#email", "value": "user@example.com"},
|
||||
{"type": "fill", "selector": "#password", "value": "..."},
|
||||
{"type": "click", "selector": "#submit"},
|
||||
{"type": "wait_for", "selector": ".dashboard", "timeout": 30}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "ok",
|
||||
"user_agent": "Mozilla/5.0 (X11; Linux x86_64) ... Chrome/155.0.0.0 Safari/537.36",
|
||||
"cookies": [{"name": "cf_clearance", "value": "...", "domain": ".example.com"}],
|
||||
"actions": [
|
||||
{"type": "fill", "ok": true, "duration_ms": 41},
|
||||
{"type": "fill", "ok": true, "duration_ms": 37},
|
||||
{"type": "click", "ok": true, "duration_ms": 120},
|
||||
{"type": "wait_for", "ok": true, "duration_ms": 1830}
|
||||
],
|
||||
"duration_ms": 9120
|
||||
}
|
||||
```
|
||||
|
||||
The response also sets a `session_id` cookie. That's a persistent Chrome profile on the server: cookies, local storage and a browser fingerprint. Send it back next time, and to the site you're still the same visitor, so the anti-bot check usually doesn't come back.
|
||||
|
||||
## Using the result in Python
|
||||
|
||||
The important part is what you do *after* the browser:
|
||||
|
||||
```python
|
||||
import httpx
|
||||
|
||||
ROBOTEX = "https://app.robotex.sokolab.xyz"
|
||||
PROXY = "http://user:pass@host:port"
|
||||
|
||||
robotex = httpx.Client(base_url=ROBOTEX, timeout=300, headers={"X-API-Key": "rbx_..."})
|
||||
|
||||
|
||||
def solve(url: str) -> dict:
|
||||
data = robotex.post("/v1/solve", json={
|
||||
"url": url,
|
||||
"proxy": PROXY,
|
||||
"captcha_type": "cloudflare",
|
||||
}).json() # session_id stays in robotex's cookie jar for the next calls
|
||||
if data.get("status") != "ok":
|
||||
raise RuntimeError(f"{data.get('code')}: {data.get('error')}")
|
||||
return data
|
||||
|
||||
|
||||
solution = solve("https://example.com/projects")
|
||||
|
||||
site = httpx.Client(
|
||||
proxy=PROXY, # the same proxy
|
||||
headers={"User-Agent": solution["user_agent"]}, # the same UA
|
||||
cookies={c["name"]: c["value"] for c in solution["cookies"]},
|
||||
)
|
||||
page = site.get("https://example.com/projects")
|
||||
if page.status_code == 403:
|
||||
solution = solve("https://example.com/projects") # same session, fresh cf_clearance
|
||||
```
|
||||
|
||||
Two rules make this work: **the same User-Agent** and **the same proxy**. Cloudflare ties `cf_clearance` to the UA and usually to the IP. Change either and the cookie is worthless.
|
||||
|
||||
## Lessons learned the hard way
|
||||
|
||||
### Don't spoof the User-Agent
|
||||
|
||||
Early versions let clients pass their own `user_agent`. It seemed harmless: it's just a string. But you can only change the string. `navigator.userAgentData`, client hints and the engine's behaviour still belong to the real Chrome, and Cloudflare sees the mismatch. I measured it on Chrome 155 / Linux against a site behind Cloudflare:
|
||||
|
||||
| User-Agent in the browser | Cloudflare |
|
||||
|---|---|
|
||||
| native (Linux Chrome 155) | passed 3 of 3, 7–9 s |
|
||||
| Linux Chrome 147 | failed 3 of 3 |
|
||||
| Windows Chrome 155 | failed |
|
||||
|
||||
So the parameter is gone (passing it returns `422`), and the rule is: **the service defines the browser's identity, the client adapts to the response.** The same went for the window size. Each profile instead gets a plausible fingerprint (screen size and `devicePixelRatio` drawn from a real-world distribution) generated once and applied natively by Chrome, with no JS patches, since those broke the Cloudflare check.
|
||||
|
||||
### Use real Chrome, not "Chromium with flags"
|
||||
|
||||
Robotex drives real Google Chrome through [Patchright](https://github.com/Kaliiiiiiiiii-Vinyzu/patchright-python), an undetected Playwright fork without the CDP leaks anti-bots look for. Chrome doesn't run headless: it runs as a normal windowed browser inside a virtual display (Xvfb).
|
||||
|
||||
### Make the proxy's geo and the browser agree
|
||||
|
||||
Locale and timezone are derived from the proxy's exit IP. A German IP with a browser in `America/New_York` is a red flag. And no fingerprint will save you on a dirty IP: proxy quality matters more than any trick on the page.
|
||||
|
||||
## Architecture v2: a queue between the API and the browsers
|
||||
|
||||
The first version ran Chrome inside the API process. When all browser slots were busy, a request simply got a `503`. The current version splits it in two:
|
||||
|
||||
```
|
||||
Client → api (FastAPI) ──task──▶ RabbitMQ ──▶ worker (dramatiq, Chrome + Xvfb)
|
||||
▲ │
|
||||
└── GET /v2/tasks/{id} ◀── Redis ◀────────┘
|
||||
```
|
||||
|
||||
- **api** has no browser at all: it queues tasks and serves results;
|
||||
- **RabbitMQ** holds tasks until a worker has a free slot, so they wait instead of failing;
|
||||
- **workers** run Chrome, several windows each;
|
||||
- **Redis** keeps task statuses and results for a day. Proxy credentials and captcha keys never go there.
|
||||
|
||||
`/v1/solve` still exists and runs on top of the same queue, so old clients didn't change a line. New clients can go async:
|
||||
|
||||
```http
|
||||
POST /v2/tasks
|
||||
X-API-Key: <your key>
|
||||
|
||||
{"url": "https://example.com/login", "actions": [...], "browser_idle_timeout": 10}
|
||||
|
||||
→ 202 {"task_id": "0b9c…", "status": "queued", "url": "/v2/tasks/0b9c…"}
|
||||
```
|
||||
|
||||
Then poll `GET /v2/tasks/{id}` (with the same `session_id` cookie) until the status is `done`.
|
||||
|
||||
### Keeping the browser window open
|
||||
|
||||
The feature I like most in v2 is `browser_idle_timeout`. With it, the session's Chrome window **stays open** for up to 120 idle minutes after the task. The next task with the same `session_id` is routed to the personal queue of the worker holding that window and runs right there: no Chrome start, no second anti-bot check. Leave out the `url`, and the actions run on the page that's already open.
|
||||
|
||||
That makes multi-step scenarios natural: log in, let the bot do its HTTP work, come back and continue filling the same form. When you're done, `DELETE /v2/sessions/current` closes the window and frees the slot.
|
||||
|
||||
## What else it does
|
||||
|
||||
- **Captchas inside pages:** the `captcha` step solves Turnstile, reCAPTCHA and hCaptcha through your own 2captcha key before the form is submitted.
|
||||
- **Anti-bot pages:** a detect → solve → verify loop for Cloudflare, before and during the scenario. Turnstile passes reliably.
|
||||
- **Secrets stay out of logs:** proxy credentials, captcha keys, `fill` values and cookie values are masked.
|
||||
|
||||
## Try it
|
||||
|
||||
Robotex is free. You only pay for your own proxies and, if you need it, 2captcha.
|
||||
|
||||
- **Landing:** [robotex.sokolab.xyz](https://robotex.sokolab.xyz/)
|
||||
- **API and Swagger:** [app.robotex.sokolab.xyz/docs](https://app.robotex.sokolab.xyz/docs)
|
||||
- **API key:** a test key for the demo instance is in the [project wiki](https://redmine.sokolab.xyz/projects/robotex/wiki)
|
||||
- **Source code:** [git.sokolab.xyz/s0k0l/robotex](https://git.sokolab.xyz/s0k0l/robotex)
|
||||
- **Questions and bugs:** [Redmine forum and issues](https://redmine.sokolab.xyz/projects/robotex)
|
||||
|
||||
Next on the roadmap: workers on several hosts and a second engine (Firefox via Camoufox) for sites where a Chrome fingerprint doesn't fit.
|
||||
|
||||
If you've fought Cloudflare from a scraper, I'd love to hear what worked for you, and what broke. And please use Robotex only on sites where you have the right to do so.
|
||||
Reference in new issue
Block a user