feat: English version of the site, CV as text
- add English language: config, menus, i18n, translations of all pages and posts - CV page renders the full resume in both languages with PDF download buttons - remove salary from CV PDFs, Telegram is @xs0k0lx everywhere, English level B2 - fix broken CV menu link, update theme submodule to Public/blowfish Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
protokeyandClaude Opus 5.5 committed
1 parent
02ff33128c
commit
a4d92fecd3
23 files changed
+1145
-8
No files matched your search
@@ -0,0 +1,4 @@
|
||||
---
|
||||
title: "Blog"
|
||||
description: "Notes on projects, software and hardware"
|
||||
---
|
||||
@@ -0,0 +1,21 @@
|
||||
---
|
||||
title: "I launched my website"
|
||||
date: 2026-10-05T21:14:48+04:00
|
||||
draft: false
|
||||
tags: ["server", "hugo"]
|
||||
summary: "How I set up a website and Gitea on my own server"
|
||||
---
|
||||
|
||||
Today I launched [sokolab.xyz](https://sokolab.xyz/), a landing page and a blog. Everything runs on my own server in Docker. Here I'll publish reports on my work, useful things about computers and internet technologies, maybe some reviews and lecture courses.
|
||||
|
||||
The home page is a landing page with a summary about me and my statuses. The blog page has all posts in chronological order. And if a course appears, or my work turns into some kind of product, it will get its own subdomain where it can live its own life.
|
||||
|
||||
## How it works
|
||||
|
||||
The site is built with Hugo, the sources live in my [Gitea](https://git.sokolab.xyz/Public/site).
|
||||
|
||||
|
||||
|
||||
- I write a post in Markdown;
|
||||
- I do `git push`;
|
||||
- a minute later it's on the site.
|
||||
@@ -12,7 +12,7 @@ summary: "Как я поднял сайт и Gitea на своём сервер
|
||||
|
||||
## Как это устроено
|
||||
|
||||
Сайт собран на Hugo, исходники лежат в моей [Gitea](https://git.sokolab.xyz/s0k0l).
|
||||
Сайт собран на Hugo, исходники лежат в моей [Gitea](https://git.sokolab.xyz/Public/site).
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,369 @@
|
||||
---
|
||||
title: How a programmer can start their own blog
|
||||
date: 2026-10-08T19:22:00+04:00
|
||||
draft: false
|
||||
tags:
|
||||
- hugo
|
||||
- server
|
||||
summary: How to set up your own blog
|
||||
---
|
||||
|
||||
## Intro
|
||||
|
||||
The first thing you need is a server. Just google "Debian VPS" and pick from what comes up.
|
||||
|
||||
**Important:** if you're in Russia, it's better to choose a hosting provider and a server in your own country. Then search Yandex for "аренда виртуального сервера Debian".
|
||||
|
||||
While the server is being set up, you have about an hour to buy a domain name and register a Cloudflare account. The first one is obvious, and CF is needed to hide your server's IP address from site visitors. This definitely improves server security, but it has to be done before the domain is first pointed at the server. So after buying the domain, go to the control panel (where you bought it) and change the NS servers to the ones Cloudflare gives you after you add the project in your account. After that it takes some time (up to 3 days) for the domain to be re-parked, since the whole internet has to learn about it. But after that no visitor will be able to find out your site's IP, unless you leak it, of course. We'll talk about that some other time.
|
||||
|
||||
What you have now:
|
||||
1. the IP and root password
|
||||
2. a domain name parked at CF
|
||||
3. records set up in CF:
|
||||
1. yourdomain.com - IP - the root record, the domain name opens the blog home page
|
||||
2. git.yourdomain.com - IP - a subdomain for Gitea
|
||||
|
||||
## Let's go
|
||||
|
||||
[Initial Debian server setup ->](/blog/zero-setup-debian-server/)
|
||||
|
||||
Install Docker with
|
||||
|
||||
```
|
||||
curl -fsSL https://get.docker.com -o install-docker.sh
|
||||
sh install-docker.sh
|
||||
```
|
||||
|
||||
Log out of the server, because now we need to prepare everything locally.
|
||||
|
||||
On your computer, install Hugo from the GitHub repository, because the Debian package repository has an old version.
|
||||
|
||||
https://github.com/gohugoio/hugo/releases
|
||||
|
||||
Install Git on your computer to work with the repository.
|
||||
|
||||
On the server we'll run 2 applications in Docker:
|
||||
- **git** - Gitea, a git service, something like GitHub
|
||||
- **hugo** - the site builder and Caddy. Caddy serves the finished pages and is also the only entry point from outside, Gitea traffic goes through it too
|
||||
|
||||
Hugo builds blog pages from Markdown files, converting everything to HTML. The workflow looks like this:
|
||||
1. create an md file in the repository
|
||||
2. fill it in, commit and push
|
||||
3. the builder in Docker checks the repository once a minute and sees a new commit
|
||||
4. it builds the site and swaps the old version for the new one
|
||||
|
||||
Okay, now pick a place on your computer and create an `apps` folder, where we'll keep the configuration of the Docker applications on the server. The final structure will look like this:
|
||||
|
||||
```
|
||||
apps/
|
||||
├── git/
|
||||
│ └── docker-compose.yaml
|
||||
└── hugo/
|
||||
├── docker-compose.yaml
|
||||
├── caddy.conf
|
||||
├── dot_env
|
||||
├── .gitignore
|
||||
└── data/
|
||||
└── builder/
|
||||
└── build.sh
|
||||
```
|
||||
|
||||
There are only configs here. All data (repositories, certificates, the built site) lives in Docker named volumes, not in folders next to the configs. This way `apps` can safely be kept in git and copied to the server without fear of overwriting data. I'll cover backups below.
|
||||
|
||||
Both applications talk over a shared `web` network. Only Caddy exposes ports to the outside.
|
||||
|
||||
### git
|
||||
|
||||
`apps/git/docker-compose.yaml`:
|
||||
|
||||
```yaml
|
||||
# Git application for docker-server
|
||||
|
||||
services:
|
||||
gitea:
|
||||
image: gitea/gitea:28.0-rootless
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
GITEA__server__DOMAIN: git.yourdomain.com
|
||||
GITEA__server__ROOT_URL: https://git.yourdomain.com/
|
||||
GITEA__server__DISABLE_SSH: "true"
|
||||
GITEA__database__DB_TYPE: sqlite3
|
||||
GITEA__service__DISABLE_REGISTRATION: "true"
|
||||
volumes:
|
||||
- data:/var/lib/gitea
|
||||
- config:/etc/gitea
|
||||
networks: [ web ]
|
||||
|
||||
|
||||
networks:
|
||||
web:
|
||||
external: true
|
||||
|
||||
|
||||
volumes:
|
||||
data:
|
||||
config:
|
||||
```
|
||||
|
||||
We use the rootless image, so inside the container Gitea doesn't run as root. This is where named volumes really fit: when creating a volume, Docker sets its owner from the image, and you don't have to `chown` anything by hand.
|
||||
|
||||
We don't expose any ports, only Caddy reaches Gitea over the `web` network. SSH is disabled because Cloudflare doesn't proxy it, and we don't want to expose the server's IP. We'll push over https. Registration is closed so nobody but you can create accounts there.
|
||||
|
||||
### hugo
|
||||
|
||||
`apps/hugo/docker-compose.yaml`:
|
||||
|
||||
```yaml
|
||||
# Landing page and blog site on Hugo
|
||||
|
||||
services:
|
||||
builder:
|
||||
image: ghcr.io/gohugoio/hugo:v0.167.0
|
||||
restart: unless-stopped
|
||||
user: root
|
||||
entrypoint: [ "sh", "/build.sh" ]
|
||||
environment:
|
||||
REPO_URL: ${REPO_URL}
|
||||
volumes:
|
||||
- ./data/builder/build.sh:/build.sh:ro
|
||||
- src:/src
|
||||
- public:/public
|
||||
networks: [ web ] # to reach git-gitea-1
|
||||
|
||||
web:
|
||||
image: caddy:2.10.2-alpine
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "80:80"
|
||||
- "443:443"
|
||||
volumes:
|
||||
- ./caddy.conf:/etc/caddy/Caddyfile:ro
|
||||
- public:/srv:ro
|
||||
- caddy_data:/data
|
||||
- caddy_config:/config
|
||||
networks: [ web ] # to proxy to git-gitea-1
|
||||
|
||||
|
||||
networks:
|
||||
web:
|
||||
external: true
|
||||
|
||||
|
||||
volumes:
|
||||
src:
|
||||
public:
|
||||
caddy_data:
|
||||
caddy_config:
|
||||
```
|
||||
|
||||
You don't need to build your own image. `builder` is the official Hugo image, it already includes git. Use the same version as on your computer, you can check it with `hugo version`. `user: root` is needed because the image runs as a regular user by default, while Docker creates volumes as root.
|
||||
|
||||
`web` is Caddy. It faces the outside on 80 and 443, serves the site from the `public` volume shared with `builder`, and proxies the git subdomain to Gitea. It keeps its certificates in `caddy_data`.
|
||||
|
||||
`apps/hugo/caddy.conf`:
|
||||
|
||||
```
|
||||
yourdomain.com {
|
||||
tls internal
|
||||
encode gzip
|
||||
root * /srv/live
|
||||
file_server
|
||||
|
||||
handle_errors {
|
||||
rewrite * /404.html
|
||||
file_server
|
||||
}
|
||||
}
|
||||
|
||||
www.yourdomain.com {
|
||||
tls internal
|
||||
redir https://yourdomain.com{uri} permanent
|
||||
}
|
||||
|
||||
git.yourdomain.com {
|
||||
tls internal
|
||||
reverse_proxy git-gitea-1:3000
|
||||
}
|
||||
```
|
||||
|
||||
The site is served from `/srv/live`, why exactly from there will become clear from the build script. `handle_errors` makes non-existent pages show the theme's nice 404 instead of an empty response. `www` simply redirects to the main domain so the site has a single address.
|
||||
|
||||
Docker Compose builds the name `git-gitea-1` itself from the folder name and the service name: folder `git`, service `gitea`, first instance. That's why it matters that the folder is named exactly like that.
|
||||
|
||||
`tls internal` means Caddy issues a certificate for itself. The browser never sees it, because visitors connect to Cloudflare, and Cloudflare connects to us. So in the Cloudflare panel, under SSL/TLS, set the mode to **Full**. Not Flexible, otherwise traffic from Cloudflare to the server goes unencrypted, and not Full (strict), which won't accept such a certificate.
|
||||
|
||||
`apps/hugo/dot_env` is a template for the variables:
|
||||
|
||||
```
|
||||
REPO_URL=http://git-gitea-1:3000/<name>/site.git
|
||||
```
|
||||
|
||||
On the server you copy it to `.env` and fill in your username and repository. The `.env` itself doesn't go into git, that's what the `.gitignore` next to it with a single `.env` line is for. The builder clones the repository straight from the Gitea container over the internal network, without going out to the internet. If the repository is private, add a token: `http://<name>:<token>@git-gitea-1:3000/<name>/site.git`. The token is issued in the Gitea user settings, under Applications. In that case hide the file from prying eyes with `chmod 600 .env`.
|
||||
|
||||
`apps/hugo/data/builder/build.sh`:
|
||||
|
||||
```sh
|
||||
#!/bin/sh
|
||||
# Once a minute: new commit -> build into /public/next -> swap /public/live.
|
||||
# Build error - the previous site stays.
|
||||
# The repository address is taken from REPO_URL on every start: change in .env + redeploy = new source.
|
||||
cd /src
|
||||
if [ -d .git ]; then
|
||||
git remote set-url origin "$REPO_URL"
|
||||
else
|
||||
git clone --recurse-submodules "$REPO_URL" . || exit 1
|
||||
fi
|
||||
last=''
|
||||
while true; do
|
||||
if git fetch -q origin HEAD && git reset -q --hard FETCH_HEAD \
|
||||
&& git submodule sync -q --recursive && git submodule update -q --init --recursive; then
|
||||
rev=$(git rev-parse HEAD)
|
||||
if [ "$rev" != "$last" ]; then
|
||||
rm -rf /public/next
|
||||
if hugo --minify -d /public/next; then
|
||||
rm -rf /public/old
|
||||
[ -d /public/live ] && mv /public/live /public/old
|
||||
mv /public/next /public/live
|
||||
last=$rev
|
||||
echo "published $rev"
|
||||
else
|
||||
echo "build of $rev failed, site unchanged"
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
sleep 60
|
||||
done
|
||||
```
|
||||
|
||||
The main trick of the script is that the site is built into a separate `next` folder, and only if the build succeeds does it replace the working `live` one. If you push a broken commit, the site simply stays as it was, and the log shows what broke. The previous version is kept in `old` in case you need to roll back quickly.
|
||||
|
||||
`git reset --hard` is used here on purpose instead of `git pull`. If you force push or rewrite history, a regular pull will break, while reset just takes whatever is in the repository. And `set-url` at startup lets you change the source: edit `REPO_URL` in `.env`, restart the container, and the builder pulls from the new place.
|
||||
|
||||
## Uploading to the server
|
||||
|
||||
The configs are ready, let's send the folder to the server. `<name>` is the host name from `~/.ssh/config` that we set up in the previous article:
|
||||
|
||||
```
|
||||
s0k0l:~$ scp -r apps <name>:/opt/
|
||||
```
|
||||
|
||||
Before starting, we need to open the firewall for the site. In the previous article we closed all incoming traffic, and I warned there that Docker lives by its own rules. Let's sort it out now.
|
||||
|
||||
Open `/etc/nftables.conf` on the server. In the `input` chain, uncomment the line for the site:
|
||||
|
||||
```
|
||||
tcp dport { 80, 443 } accept
|
||||
```
|
||||
|
||||
And change the `forward` chain to this:
|
||||
|
||||
```
|
||||
chain forward {
|
||||
type filter hook forward priority 0; policy drop;
|
||||
|
||||
ct state established,related accept
|
||||
ct status dnat accept
|
||||
iifname "docker0" accept
|
||||
iifname "br-*" accept
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The thing is, traffic to containers goes not through `input` but through `forward`. With our strict `policy drop`, containers can neither accept connections nor reach the internet. These rules let through traffic to the ports Docker published (in our case only Caddy's 80 and 443) and outgoing traffic from the containers themselves. Everything else is still closed.
|
||||
|
||||
Check and apply the same way as last time, with a timer just in case:
|
||||
|
||||
```
|
||||
nft -c -f /etc/nftables.conf
|
||||
(sleep 120 && nft flush ruleset) &
|
||||
nft -f /etc/nftables.conf
|
||||
```
|
||||
|
||||
Check that ssh is alive, cancel the timer with `kill %1` and restart Docker:
|
||||
|
||||
```
|
||||
systemctl restart docker
|
||||
```
|
||||
|
||||
This is mandatory. Our config starts with `flush ruleset`, which also wipes the rules Docker created for itself. After a restart Docker creates them again. Remember this: every time you restart nftables, restart Docker too.
|
||||
|
||||
## Launch
|
||||
|
||||
Create the shared network through which the containers will see each other:
|
||||
|
||||
```
|
||||
docker network create web
|
||||
```
|
||||
|
||||
Bring up Gitea and, for now, only Caddy without the builder. The builder has nothing to clone until there's a repository in Gitea:
|
||||
|
||||
```
|
||||
cd /opt/apps/git && docker compose up -d
|
||||
cd /opt/apps/hugo && cp dot_env .env && docker compose up -d web
|
||||
```
|
||||
|
||||
Open `https://git.yourdomain.com` in the browser. Gitea will show the initial setup page, the database is already set in the config. At the bottom of the page, in the administrator account section, create your user. We closed registration, so this is the only way to get an account.
|
||||
|
||||
Create a `site` repository in Gitea. Now push the site there from your computer:
|
||||
|
||||
```
|
||||
s0k0l:~$ cd my-blog
|
||||
s0k0l:~$ git remote add origin https://git.yourdomain.com/<name>/site.git
|
||||
s0k0l:~$ git push -u origin main
|
||||
```
|
||||
|
||||
If the theme is included as a git submodule, the theme repository must also be reachable by the builder at the URL in `.gitmodules`. The easiest way is to make a mirror of the theme in your Gitea and put its address in `.gitmodules`.
|
||||
|
||||
Put your real username into `/opt/apps/hugo/.env` and start the builder:
|
||||
|
||||
```
|
||||
cd /opt/apps/hugo && docker compose up -d
|
||||
```
|
||||
|
||||
Check that it built the site:
|
||||
|
||||
```
|
||||
docker logs -f hugo-builder-1
|
||||
```
|
||||
|
||||
If `published <commit hash>` shows up in the log, open `https://yourdomain.com`, the blog is working.
|
||||
|
||||
## Backups
|
||||
|
||||
Since the data lives in named volumes, you can't see it in `apps`. You can list them with `docker volume ls`. Two of them matter here: `git_data` and `git_config`, they hold all the repositories and Gitea settings. The site doesn't need backing up, it's built from the repository, and Caddy will issue certificates again.
|
||||
|
||||
Back up Gitea:
|
||||
|
||||
```
|
||||
docker compose -f /opt/apps/git/docker-compose.yaml stop
|
||||
docker run --rm -v git_data:/data -v git_config:/config -v /root:/backup alpine \
|
||||
tar czf /backup/gitea-$(date +%F).tgz /data /config
|
||||
docker compose -f /opt/apps/git/docker-compose.yaml start
|
||||
```
|
||||
|
||||
We stop Gitea during the backup so the SQLite database isn't written halfway. Pull the finished archive to your machine with `scp`.
|
||||
|
||||
## How to write now
|
||||
|
||||
The whole process looks like this:
|
||||
|
||||
1. create a post with `hugo new content blog/my-post/index.md`
|
||||
2. write it and preview it locally with `hugo server`
|
||||
3. commit and push
|
||||
4. a minute later the post is on the site
|
||||
|
||||
For convenience I added a `Makefile` to the repository so I don't have to type the commands by hand:
|
||||
|
||||
```makefile
|
||||
all:
|
||||
hugo new content $(path)
|
||||
|
||||
publish:
|
||||
git add .
|
||||
git commit -m 'chore: $(msg)'
|
||||
git push
|
||||
```
|
||||
|
||||
A new post is `make path=blog/my-post/index.md`, publishing is `make publish msg="new post"`.
|
||||
|
||||
That's it. A server, your own git, a blog and auto-deploy, all on one inexpensive VPS with no third-party services except Cloudflare.
|
||||
@@ -0,0 +1,152 @@
|
||||
---
|
||||
title: Robotex, a Browser Automation API
|
||||
date: 2026-10-09T01:59:39+04:00
|
||||
draft: false
|
||||
tags:
|
||||
- parsing
|
||||
- anti-bot
|
||||
- api
|
||||
- saas
|
||||
summary: How a bot can scrape a site behind anti-bot protection or with heavy JS
|
||||
aliases:
|
||||
- /blog/robotex-brouser-automation-api/
|
||||
---
|
||||
|
||||
Every website bot sooner or later runs into one of three problems:
|
||||
|
||||
1. a captcha
|
||||
2. anti-bot protection
|
||||
3. heavy JS that has to be rendered to submit a form
|
||||
|
||||
A spider that walks a site with raw http requests is helpless here. There's only one solution: launch a browser and let it handle the hard part. But the browser has to be hidden too. Under the hood it sends the same http requests as a Python spider, it just does so from the context of running JS. And that context is full of markers that tell the site someone is controlling the browser.
|
||||
|
||||
For this task I wrote a microservice, [Robotex](https://git.sokolab.xyz/s0k0l/robotex). Below I'll explain how it works and why it's needed when Scrapling and FlareSolverr already exist.
|
||||
|
||||
## What was used before and now
|
||||
|
||||
I used to go with Selenium. It runs Firefox through geckodriver over the Marionette protocol and gives itself away quite noticeably: `navigator.webdriver`, driver traces in the page environment and so on. Almost all of that can be hidden with an add-on that adjusts the page context before the site loads. That's what I did, but it's a constant race against every new check.
|
||||
|
||||
Now there are tools that solve this at the level of the browser itself:
|
||||
|
||||
- **Camoufox** - a Firefox build where fingerprints (screen, fonts, WebGL, core count, etc.) are spoofed in the engine code rather than through JS.
|
||||
- **Patchright** - a patched Playwright that removes the CDP leaks anti-bots use to detect Chromium automation.
|
||||
|
||||
Scrapling works on top of them. It has a ready-made mode for getting through anti-bot pages, and I borrowed its algorithm. It turned out that passing Cloudflare isn't about solving a puzzle. If the browser environment looks human and the IP is clean, it's enough to wait for the widget and click the checkbox with a random offset and delay. The whole difficulty is correctly detecting that you're facing a check, and making sure the browser no longer looks suspicious by the time it clicks.
|
||||
|
||||
For small volumes without accounts, Scrapling is enough. Problems start when accounts come in. Then the IP, User-Agent, cookies and browser fingerprint have to be tied into one profile and live together. Change the proxy but keep the old cookies, and the account goes into verification or gets banned. The protection starts nagging you with captchas, and accounts die.
|
||||
|
||||
## What Robotex is
|
||||
|
||||
Robotex takes over all the browser work: it gets past the anti-bot, runs a scenario on the page and hands the bot cookies it can use to keep browsing the site with plain requests.
|
||||
|
||||
It differs from FlareSolverr and Byparr in that those can only open a page and get `cf_clearance`. Robotex runs a scenario: fill a form, click, solve a captcha, wait for an element. And it differs from Scrapling in that it's not a library inside your process but a separate service. The bot can be written in anything, all it needs is an http client.
|
||||
|
||||
Under the hood are FastAPI and a real Chrome driven by Patchright. I ported the browser wrapper from Scrapling. I started with Camoufox but eventually settled on Chrome.
|
||||
|
||||
## How it works
|
||||
|
||||
The bot sends a scenario:
|
||||
|
||||
```
|
||||
POST /v1/solve
|
||||
Content-Type: application/json
|
||||
X-API-Key: <key>
|
||||
|
||||
{
|
||||
"url": "https://example.com/login",
|
||||
"actions": [
|
||||
{"type": "fill", "selector": "#email", "value": "user@example.com"},
|
||||
{"type": "fill", "selector": "#password", "value": "..."},
|
||||
{"type": "mouse_move", "selector": "#submit"},
|
||||
{"type": "click", "selector": "#submit"},
|
||||
{"type": "wait_for", "selector": ".dashboard", "timeout": 30}
|
||||
],
|
||||
"proxy": "socks5://user:pass@host:port",
|
||||
"html": true,
|
||||
"timeout": 120
|
||||
}
|
||||
```
|
||||
|
||||
The scenario fills in the login form and waits for the dashboard to load. Besides these actions there are `submit` (a click that waits for navigation to a new page), `delay`, `inner_html` (grab a piece of the page) and `captcha`, more on that below.
|
||||
|
||||
Since the request has no `session_id` cookie, the service creates a new browser profile and returns it in the `Set-Cookie` header. The profile is stored on disk: cookies, local storage and fingerprint. The browser locale and timezone are matched to the proxy's geo, so the IP and the environment don't contradict each other. When the bot needs a browser again, it sends this cookie, and Robotex continues in the same profile. To the site it's still the same user.
|
||||
|
||||
The response:
|
||||
|
||||
```
|
||||
{
|
||||
"status": "ok",
|
||||
"user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) ...",
|
||||
"cookies": [
|
||||
{"name": "sessionid", "value": "...", "domain": ".example.com", "path": "/"}
|
||||
],
|
||||
"actions": [
|
||||
{"type": "fill", "ok": true},
|
||||
{"type": "fill", "ok": true},
|
||||
{"type": "mouse_move", "ok": true},
|
||||
{"type": "click", "ok": true},
|
||||
{"type": "wait_for", "ok": true}
|
||||
],
|
||||
"html": "<head>...</head><body>...</body>"
|
||||
}
|
||||
```
|
||||
|
||||
From there the bot browses the site on its own, with these cookies, **the same User-Agent and through the same proxy**. If an action fails, `status` will be `err`, and `actions` shows at which step and why.
|
||||
|
||||
### Captcha
|
||||
|
||||
If a captcha is expected on the form, add a step for it and a recognition service key (2captcha is supported right now):
|
||||
|
||||
```
|
||||
{
|
||||
"url": "https://example.com/login",
|
||||
"actions": [
|
||||
{"type": "fill", "selector": "#email", "value": "user@example.com"},
|
||||
{"type": "fill", "selector": "#password", "value": "..."},
|
||||
{"type": "captcha", "selector": "#captcha", "value": "hcaptcha"},
|
||||
{"type": "click", "selector": "#submit"},
|
||||
{"type": "wait_for", "selector": ".dashboard", "timeout": 30}
|
||||
],
|
||||
"captcha": "<2captcha key>",
|
||||
"proxy": "socks5://user:pass@host:port"
|
||||
}
|
||||
```
|
||||
|
||||
The captcha will be solved before the submit click.
|
||||
|
||||
### Anti-bot page
|
||||
|
||||
An anti-bot page can appear at any moment, or not at all. So it's not a scenario step but a setting for the whole request:
|
||||
|
||||
```
|
||||
{
|
||||
"url": "https://example.com/login",
|
||||
"actions": [ ... ],
|
||||
"captcha_type": "cloudflare",
|
||||
"captcha_detect_locator": "#anti-bot-page-only-id",
|
||||
"captcha_success_locator": "#site-only-id",
|
||||
"captcha": "<2captcha key>",
|
||||
"proxy": "socks5://user:pass@host:port"
|
||||
}
|
||||
```
|
||||
|
||||
`captcha_detect_locator` is a selector that exists only on the check page, `captcha_success_locator` exists only on the site itself. Right after the page loads and before running the scenario, Robotex checks whether there's a barrier in front of it. If there is, it gets through on its own: waits, clicks the checkbox. If the check escalated to a full image captcha, it sends it to the recognition service. If the check pops up in the middle of a scenario, there's a separate `pass_challenge` action for that.
|
||||
|
||||
For heavy sites there are a couple more useful options: `disable_resources` turns off loading of images and fonts, `blocked_domains` cuts requests to unneeded domains like analytics, and `cookies` lets you pass in ready-made cookies.
|
||||
|
||||
## Limitations
|
||||
|
||||
There's no silver bullet here:
|
||||
|
||||
- This is an early MVP. API key checks, a task queue and billing are still planned
|
||||
- a browser eats hundreds of megabytes of memory, so one instance currently handles 10 concurrent sessions
|
||||
- proxy quality matters a lot, no fingerprint will help on a dirty IP
|
||||
- Turnstile doesn't pass on every site, I'm still working on stability. Anti-bots get updated, and getting through a specific site has to be tested and maintained
|
||||
|
||||
On the upside, the bot stays lightweight: it needs a browser only where there's no way around it, and the rest of the time it works with fast http requests.
|
||||
|
||||
## Code
|
||||
|
||||
Sources: [git.sokolab.xyz/s0k0l/robotex](https://git.sokolab.xyz/s0k0l/robotex). The `docs` folder also has the spec for the anti-bot bypass module, if you're curious how it works inside.
|
||||
|
||||
Use it only on sites where you have the right to do so.
|
||||
@@ -0,0 +1,187 @@
|
||||
---
|
||||
title: "Initial Debian server setup"
|
||||
date: 2026-10-08T19:29:38+04:00
|
||||
draft: false
|
||||
tags: [server, linux]
|
||||
summary: "What to do with a Debian server after you buy it. An article for Linux users."
|
||||
---
|
||||
|
||||
Usually, after you buy a server, you get an email with an IP address and the root password. That's enough to connect to the server over ssh and start setting it up.
|
||||
|
||||
Just 3 steps:
|
||||
1. System update
|
||||
2. ssh setup
|
||||
3. nftables setup
|
||||
|
||||
Step zero is preparing ssh keys and a config for the server. To generate keys, run:
|
||||
|
||||
```
|
||||
s0k0l:~$ ssh-keygen
|
||||
Generating public/private ed25519 key pair.
|
||||
Enter file in which to save the key (/~/.ssh/id_ed25519):
|
||||
```
|
||||
|
||||
It asks for a path for the key. You can just type the key name you want, and the pair will be created in the directory you ran the command from. The first time you can skip this and press Enter.
|
||||
|
||||
The second question is about a passphrase for the key. For ssh keys to production infrastructure it's better to set one. If you don't want to, just hit Enter.
|
||||
|
||||
Once the hosting provider brings the server up, copy the key over with
|
||||
|
||||
```
|
||||
s0k0l:~$ ssh-copy-id root@<ip>
|
||||
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
|
||||
/usr/bin/ssh-copy-id: INFO: 3 key(s) remain to be installed -- if you are prompted now it is to install the new keys
|
||||
root@<ip>'s password:
|
||||
```
|
||||
|
||||
Enter the password and you should see a success message suggesting you connect to the server, which is exactly what we'll do. But first add a new entry to `~/.ssh/config`:
|
||||
|
||||
```
|
||||
Host <name>
|
||||
HostName <ip>
|
||||
Port 22
|
||||
IdentityFile ~/.ssh/id_ed25519
|
||||
```
|
||||
|
||||
Now you can connect by name
|
||||
|
||||
```
|
||||
ssh <name>
|
||||
```
|
||||
|
||||
On the server, the first thing to do is run an update:
|
||||
|
||||
```
|
||||
apt update && apt dist-upgrade -y && reboot
|
||||
```
|
||||
|
||||
This updates the whole system and reboots it. Wait about 5 minutes and try to connect again.
|
||||
|
||||
Now we need to configure ssh so that nobody else can connect to the server on the default port or log in with a password. This is the most important part.
|
||||
|
||||
Open `/etc/ssh/sshd_config` for editing and find the parameters for the listen address and port.
|
||||
|
||||
```
|
||||
Port 5872
|
||||
AddressFamily inet
|
||||
ListenAddress <ip>
|
||||
#ListenAddress ::
|
||||
```
|
||||
|
||||
Set them like this to strictly define how the server can be reached. We have an ipv4 address, so we can put it in the config explicitly. And the port should be changed to a random one in the range from 1024 to 65535. To check which ports are already taken, run:
|
||||
|
||||
```
|
||||
ss -tlnup
|
||||
```
|
||||
|
||||
If the port you want isn't in the output, it's free.
|
||||
|
||||
Next, find the following in the ssh config:
|
||||
|
||||
```
|
||||
PermitRootLogin prohibit-password
|
||||
|
||||
PubkeyAuthentication yes
|
||||
|
||||
PasswordAuthentication no
|
||||
```
|
||||
|
||||
These settings disable ssh password authentication for all users, including root (PermitRootLogin).
|
||||
|
||||
After saving the config, restart the ssh service. Stay in your session until you've connected in parallel.
|
||||
|
||||
```
|
||||
systemctl restart ssh
|
||||
```
|
||||
|
||||
Open another terminal, fix the port in `~/.ssh/config` and connect to the server. If the connection works, you can close the first terminal. SSH is done.
|
||||
|
||||
The last step is setting up nftables. It's the standard firewall in Debian, iptables is no longer needed.
|
||||
|
||||
The rules live in a single file, `/etc/nftables.conf`. The principle of a strict config is simple: all incoming traffic is denied, we allow only what is actually needed. Outgoing traffic stays open, otherwise updates and DNS will break.
|
||||
|
||||
Open `/etc/nftables.conf` and replace its contents entirely:
|
||||
|
||||
```
|
||||
#!/usr/sbin/nft -f
|
||||
|
||||
flush ruleset
|
||||
|
||||
table inet filter {
|
||||
chain input {
|
||||
type filter hook input priority 0; policy drop;
|
||||
|
||||
# let established connections through, drop garbage
|
||||
ct state established,related accept
|
||||
ct state invalid drop
|
||||
|
||||
# loopback interface
|
||||
iif lo accept
|
||||
|
||||
# ping, but no more than 5 per second
|
||||
ip protocol icmp icmp type echo-request limit rate 5/second accept
|
||||
ip protocol icmp accept
|
||||
meta l4proto ipv6-icmp accept
|
||||
|
||||
# ssh on our port, no more than 10 new connections per minute from one ip
|
||||
tcp dport 5872 ct state new meter ssh_limit { ip saddr limit rate 10/minute } accept
|
||||
|
||||
# uncomment if the server will host a website
|
||||
# tcp dport { 80, 443 } accept
|
||||
}
|
||||
|
||||
chain forward {
|
||||
type filter hook forward priority 0; policy drop;
|
||||
}
|
||||
|
||||
chain output {
|
||||
type filter hook output priority 0; policy accept;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Change port `5872` to the one you set in `sshd_config`. This is the most important line in the config, a mistake here will lock you out of the server.
|
||||
|
||||
First, check the config for errors without applying anything:
|
||||
|
||||
```
|
||||
nft -c -f /etc/nftables.conf
|
||||
```
|
||||
|
||||
If the command printed nothing, the syntax is fine. Now a safety net in case we do lock ourselves out. Start a timer that will flush all rules in 2 minutes:
|
||||
|
||||
```
|
||||
(sleep 120 && nft flush ruleset) &
|
||||
```
|
||||
|
||||
And apply the config:
|
||||
|
||||
```
|
||||
nft -f /etc/nftables.conf
|
||||
```
|
||||
|
||||
Then, as with ssh, open another terminal and connect. If you got in, everything is fine, cancel the timer:
|
||||
|
||||
```
|
||||
kill %1
|
||||
```
|
||||
|
||||
If you didn't get in, just wait 2 minutes, the rules will be flushed automatically and you can calmly look for the mistake.
|
||||
|
||||
To see what is actually applied right now, run:
|
||||
|
||||
```
|
||||
nft list ruleset
|
||||
```
|
||||
|
||||
All that's left is to enable loading the rules at boot, otherwise after a reboot the server will be left without a firewall:
|
||||
|
||||
```
|
||||
systemctl enable --now nftables
|
||||
```
|
||||
|
||||
If the server will run Docker, keep in mind that it writes its own rules and can open container ports bypassing your config. That's a topic for a separate article.
|
||||
|
||||
That's it. The server is updated, ssh only lets you in with a key and on a non-standard port, and the firewall blocks everything we haven't explicitly allowed. From here you can start installing your services.
|
||||
|
||||
Disclaimer: the timer trick is needed because a firewall can drop even an already established connection. The example config has the line `ct state established,related accept`, which means "allow connections that are already established". So in theory there shouldn't be a disconnect.
|
||||
Reference in new issue
Block a user