# Protecting a Website from Bot Scanning


You open `access.log` — and there it is, the knocking. Regular. Brainless. Mechanical, like
a woodpecker running on numerical control.

`/wp-login.php`. `/.env`. `/.git/config`. `/phpmyadmin/`.

The site, incidentally, is built on Hugo. WordPress has never lived here. Not for one single day.

And yet they keep knocking. Forty seconds pass — and there it is again. Day and night. No lunch
breaks, no weekends, no conscience.

And no. This is no break-in.

This is an inventory. A stroll through with a notepad: so, what have you got here worth taking?

<!--more-->

## The regular knock at the door

Scanning is the internet's background noise, like the hum of a refrigerator: until you truly listen
for it, it isn't quite there.

Your domain flickered through the CT logs the moment a certificate was issued. Then it showed up
in passive DNS. Then somebody pointed Shodan at your address range — and that's it, you're on
the list.

Nobody singled you out personally, don't flatter yourself. A combine harvester is simply rolling
along the whole address range, rattling every doorknob in turn. Your door is the third on the left.
No better than your neighbour's.

Go and look at your own logs, right this moment. Seriously, give it a minute:

`grep -aoE '"(GET|POST) [^ ]+' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -30`

I'd wager something from this gentleman's kit floats up to the top:

* `/.env`, `/.git/config`, `/config.json` — secrets somebody carelessly left exposed to the
  outside. A classic;
* `/wp-login.php`, `/xmlrpc.php`, `/wp-admin/` — the everlasting WordPress hunt. Even if you don't
  have it and never did;
* `/phpmyadmin/`, `/adminer.php` — little panels left behind back in 2014. The bots, unlike you,
  still remember them. Every one;
* `/backup.sql`, `/dump.sql.gz`, `/site.zip` — what if the database dump is still lying in the web
  root? Well, what if?
* `/actuator/env`, `/api/v1/pods` — Spring and Kubernetes. The fashion never goes out of style;
* plus the never-ending grind through `/admin`, `/manager`, `/cgi-bin/` with every imaginable
  suffix.

And running alongside is a second layer. A nastier one.

Scrapers methodically drain your product catalogue — the lot, down to the very last item page.

Password-guessers hammer the login form: five passwords per second, from three hundred addresses,
all at once.

AI crawlers carry off the texts you paid a real author to write.

Click farms pump up behavioural metrics and burn through your ad budget.

Everybody is having fun. Except you.

> The frightening one isn't whoever found a hole. The frightening one is whoever keeps looking for it,
> continuously, for free, and without tiring. They will find it eventually.

## Who exactly are we catching

"Bot" is far too vague a word, like "animal" at a zoo. Each kind needs catching in its own way, and
mixing up the categories costs you dearly: swat the wrong ones and tomorrow your site quietly slips
out of search.

Meet them, one by one, by name:

* **Vulnerability scanners** — grinding through paths, hunting for admin panels, `.env`, exploit
  URLs. What it costs you: logs crammed with junk, and a hole once found is already a disaster.
  Cured by a filter in front of the server;
* **Password-guessers** — hammering `/login` with dictionaries. Stolen accounts, a database brought
  to its knees. Cured by rate limits and checks;
* **Content scrapers** — pulling down the whole catalogue and every article. Content theft plus
  a bandwidth bill. Screened at the door;
* **AI crawlers** — harvesting texts to train models. Your content drives off into somebody else's
  model. Switched off in the known-bots list;
* **SEO crawlers** — building link maps for competitors. Server load, your site's structure leaked.
  Switched off in the known-bots list;
* **Fraud farms** — cloned browsers from hundreds of addresses. Ad click fraud, skewed analytics.
  Handed to the protection layer;
* **Search engines** — they index. Meaning they work for you. Nothing about them needs fixing.
  Do not touch them, let them pass. Always.

That last line is the important one, in case it slipped past.

Half the "we switched protection on and lost our traffic" stories begin exactly there: Googlebot was
sent to a captcha. It took offence. And it never came back.

## robots.txt is not a fence

`robots.txt` is a polite request. A request, to be precise — not a rule, not a fence, not a barrier.

Search engines honour it, because looking good pays off for them. A vulnerability scanner, on the
other hand, feels about your politeness… let's put it this way: it could not care less.

Worse still. A file like this:

> `User-agent: *`
> `Disallow: /admin-panel-2019/`
> `Disallow: /backup/`
> `Disallow: /internal-api/`

…is not protection. It's a map. Directions to the interesting parts, drawn in your own hand.

*Thank you*, says the bot, *how very kind* — and walks straight down those addresses.

The rule, in short: put in `robots.txt` only what you wouldn't mind showing to any passer-by.
Everything else gets closed off with access controls. Polite requests don't close off things like
that. Ever.

## Three places you can put the filter

* **In the application** — captchas in forms, honeypot fields. Upside: it knows the context of the
  action. Downside: the request has already made it to your code, your database and your disk;
* **On the server** — nginx `limit_req`, fail2ban, a WAF. Full control, free of charge. But it only
  sees its own IP, and against a distributed attack it's almost useless;
* **At the perimeter, in front of the server** — a proxy with a browser check. The junk never gets
  to origin. The trade-off: traffic flows through a middleman, so you need one you trust.

The server layer, by the way, is not to be sniffed at — it works, and it works honestly.

The minimum gentleman's kit for nginx looks like this: cap the rate of requests against the login
form.

```nginx
limit_req_zone $binary_remote_addr zone=login:10m rate=5r/m;

location = /login {
    limit_req zone=login burst=3 nodelay;
    proxy_pass http://backend;
}
```

And on top of that, fail2ban, so the especially stubborn characters get banned right at the kernel
level. For good. Well, for a day to begin with:

```ini
[nginx-scan]
enabled  = true
port     = http,https
filter   = nginx-scan
logpath  = /var/log/nginx/access.log
maxretry = 6
findtime = 120
bantime  = 86400
```

`bantime 86400` is a day, in case you were wondering. In seconds, yes.

Does it work? Of course it works.

Right up until the day those same requests come in from four thousand addresses — one or two from
each. Every one of them quiet. Every one below any threshold you care to set, however far down you
drag it.

And the server is down on the floor. And you're standing over it, bewildered.

That is where the server layer ends. And the perimeter begins.

We'll go through it using [WebShield](https://webshield.pro/) as the example: the domain moves onto
their NS, traffic flows through their proxy, and the server receives only what has already been
sieved. The site's code stays untouched. Not a single line. Zero edits.

## Connection to WebShield

### Step 1. Pointing the domain to their NS

Sign up, open **Domains → Add domain**, and type in the bare name — no `https://`, no paths: just
`example.com`.

Then over to your registrar, and switch the NS records to `nsbox.webshield.pro` and
`nshub.webshield.pro`.

Remove the old NS records. Entirely. Every last one.

Keep one "just in case" and you'll earn the classic "it works every other time": half the resolvers
dutifully route around your protection. Around it, yes. That kind of surprise.

The update usually propagates within 10–60 minutes. Sometimes it stretches out to 48 hours. There's
no hurrying it along — DNS lives its own life, don't ask.

Check from the terminal with `dig +short NS example.com`, or, if you'd like to see it through
Google's own eyes, `dig +short NS example.com @8.8.8.8`.

Once the `nsbox`/`nshub` pair shows up in the answer, press **Check** in the dashboard. The status
should change to *Delegated*. Should, though not always straight away: DNS, as we've established,
thinks slowly.

{{< admonition type=tip title="Connect before you need it" open=true >}}
Protection goes up ahead of an attack, not in the middle of one. While the NS records are still
spreading across resolvers, the botnet keeps pounding away at the old address. And you sit there and
wait. Waiting under fire is no fun at all — tested and confirmed.
{{< /admonition >}}

### Step 2. Records and the Proxy toggle

In the **DNS** section, add the usual records: `A` or `AAAA` for `@`, `CNAME` for `www`. If mail
lives on the domain, add `MX` and `TXT`.

Supported types are `A`, `AAAA`, `CNAME`, `MX`, `TXT`, `SRV`, `CAA`; `SOA` and `NS` are system
records and can't be edited by hand.

And now — the single most important move in this entire article. On `A`, `AAAA` and `CNAME` records
there's a **Proxy** toggle.

One click, and visitor requests no longer travel straight to you but through WebShield's nodes —
with caching, WAF and filtering.

Service records (`MX`, `TXT`, `SRV`, `CAA`) are never proxied, and thank heavens: mail has no use
for a proxy.

Next comes **Hosts**, and that's where you tune the behaviour of a specific name:

* **Host mode** — proxying, or a 301 redirect to a different host. Handy for `www` when the site
  lives on the apex;
* **SSL required** and **Redirect HTTP to HTTPS** — the certificate is issued and renewed
  automatically, for free, subdomains included;
* **HTTP/2** and **HTTP/3** — for client connections. The browser picks HTTP/3 up from the `Alt-Svc`
  header on the next request; if the visitor's network blocks UDP, no harm done, they quietly stay
  on HTTP/2;
* **Protocol to your server** — the "WebShield → origin" leg. Leave it on *Automatic*. Seriously,
  leave it.

On that last one, briefly and to the point. Force HTTP/2 at a server that doesn't speak it and you
get a 502. Across the entire site. With no rollback. None whatsoever.

When in doubt, don't touch it. Like the manual for an electrical appliance, honestly.

### Step 3. Hide the origin

The most galling mistake — and the most common one: protection is on, the dashboards are glowing,
lovely, and the attack sails straight past. Directly at the server's IP.

Why? Because that IP is still sticking out in plain sight, like an antenna.

Check three things. Three, no more than that:

1. no DNS records left pointing at the real address. Old subdomains, `mail.`, `ftp.`, `old.`,
   `dev.` and other greetings from the past — clear every one of them out;
2. has the IP already been exposed anywhere? DNS history, for your information, is kept for years.
   Change the IP at your hosting provider, don't be shy;
3. on the server, close 80 and 443 to everyone except WebShield's nodes. Everyone. Except them.

With ufw it looks like this:

* `ufw default deny incoming`
* `ufw allow 22/tcp`
* `ufw allow from <WebShield-node-address> to any port 443 proto tcp`
* `ufw enable`

Take the current node address list from the dashboard or from support. There's nothing to guess at
here; you'll only end up arguing with your firewall over nothing.

### Step 4. Browser check: Off, Browser or Captcha

This is the core of anti-scanning protection, the heart of the entire enterprise.

The level is chosen for each host: in **Hosts** for a proxied site, in the site's card under
**Sites** for static hosting. There are three modes in total:

* **Off** — no extra checks, basic filtering stays in place. The default, for quiet sites;
* **Browser** — suspicious clients get put through a browser check. The working choice for most
  people, take this one;
* **Captcha** — a check with a captcha, stricter. Mutually exclusive with Browser. This is the mode
  for an active attack, a flood of scrapers, click fraud.

The browser check is a quick automatic test at the door.

A real person usually doesn't even notice it: the page opened by itself, nothing needed clicking,
off we go. An automated client that is only pretending to be a browser, though, fails the test — and
never gets to the server.

What exactly gets checked, and how, is not something you need to know. Neither do you. And certainly
not whoever runs into that screen. Let them guess.

Malicious-request filtering, meanwhile, runs constantly, whatever you pick with that switch:
injections, service files, exploit paths — all of it is cut off even on **Off**.

A scanner yanking at `/.env` gets bounced in any mode. No exceptions.

How do you know it's time to tighten up? Simple: regular spikes on the "Attacks over time" chart, or
the "Direct hits on the check" card creeping upward — that's it, switch the captcha on. That's what
it's there for.

### Step 5. Exceptions — surgically, one at a time

The browser check breaks everything that can't be a browser: websockets, API calls, mobile and
server-side SDKs. The check is of no use to them — they're already on your side.

For those, the host settings have exceptions. One per line, along these lines: `/robots.txt`,
`/sitemap*.xml`, `/api/*`, `/ws`.

The rules are strict. And that, oddly enough, is for the better:

* no more than 50 entries;
* each one starts with `/` — strictly;
* the root `/` can't be specified. Otherwise the whole point is lost entirely, it truly is;
* `..` is forbidden;
* up to 200 characters: letters, digits and `/`, `_`, `.`, `-`, `*`;
* the asterisk `*` works as a mask at the end of a segment. There only.

An exception removes the check screen. And only that. Malicious-request filtering, rate limits and
bans keep working on those paths exactly as before.

Even so — don't put login forms, the admin panel or sections holding user data into exceptions. The
temptation is strong, sure. The cost of getting it wrong is stronger.

### Step 6. Known bots: who gets in, and who gets a 403

A separate layer. And my favourite one, honestly.

WebShield recognises known bots by sight and lets you decide about each one individually — right on
the host, in the **Known bots** block. All are allowed by default. Checkbox ticked, we let them
through. Unticked, it's a 403 — and not just anywhere, but out at the edge, before your server.

They're divided into groups, like so:

* **AI training crawlers** — GPTBot, ClaudeBot and anthropic-ai, CCBot, Bytespider, Amazonbot,
  Meta-ExternalAgent, Cohere, Diffbot. And the company;
* **AI search crawlers** — OAI-SearchBot, PerplexityBot;
* **AI assistants** — ChatGPT-User, Claude-User, Perplexity-User, DuckAssistBot. These fetch a page
  because a living human asked for it in a chat — helping them out even feels rather nice;
* **Search engines** — Googlebot, Bingbot, YandexBot, Applebot, DuckDuckBot and the rest;
* **Social networks and messengers** — link previews: Telegram, VK, X, LinkedIn, WhatsApp, Discord;
* **SEO and analysis** — AhrefsBot, SemrushBot, MJ12bot, DotBot, DataForSeoBot;
* **Monitoring** — UptimeRobot, Pingdom, StatusCake, Site24x7.

A typical setup for a content project: untick both AI groups entirely and leave the search engines
alone.

Indexing carries on, and the texts don't drive off to be trained on.

Assistants are your call: behind each of them sits a living person, and they're reading about you
right now. Or they asked about you — which is also rather nice.

Link vendors and competitors' crawlers (`AhrefsBot`, `SemrushBot`, `MJ12bot`) get shut out on pure
economics: they know how to drain a site. Very diligently, credit where it's due.

The result shows up in the statistics with the naked eye: on the "Visitors" tab, on the "Humans and
bots" chart, the layer for the blocked category collapses within a day or two. And traffic to origin
drops by exactly its share. Arithmetic, that.

This layer is built for bots that identify themselves honestly. Big companies don't play games about
that — blocking applies to them reliably.

A disguised scraper, by contrast, won't be cut off by this list: other mechanisms handle those. The
layers don't replace one another — they stack. Like blankets in winter.

And separately, to prevent confusion: `Google-Extended` and `Applebot-Extended` are not bots from
that list. They are directives for `robots.txt`, written into the file like this:
`User-agent: Google-Extended` and, below it, `Disallow: /`.

### Step 7. The script on your site — seeing without showing a screen

Not everyone is willing to show visitors a check. And I get it: an online store, every second
counts, and an extra screen between the ad and the shopping cart isn't an "overhead", it's a direct
loss. Money down the drain.

The compromise they came up with is elegant — a script on your pages:
`<script src="/webshieldpro/t.js" async></script>`.

No keys. No settings. It's served from your own domain, so no third party enters the chain.

Why it exists: some visitor signals are only visible from inside the page, and without the script
that zone stays blind. With it, the real picture emerges in the statistics: how much traffic comes
from farms and automated browsers. Not "how many visits there were", but who they actually were.

And here is what it does **not** do. This needs to be clear, with no illusions:

* it blocks nothing. It merely supplies data the protection uses to make decisions;
* it does not replace the browser check: that one denies entry, this one merely makes things
  visible;
* it has no API — you can't reach it from your own code, so don't even try;
* it's no use at all for decisions about forms. There's a separate tool for that, covered below.

What it collects: technical characteristics of the environment and anonymised counters. What it does
not collect: field contents, typed text, passwords, page contents, browsing history. None of that.

Do reflect the fact that visitor data gets processed technically in your privacy policy, though —
that's your area of responsibility, and nobody is going to cover for you there.

### Step 8. Forms: a separate problem, a separate tool

Sign-ups, enquiries, promo codes, logins — all of this is about trusting one specific action, not
traffic in general.

The tool is a separate one: **Form protection**. In the dashboard you create a project, receive
a key, put a tag in `<head>` and list your forms by selector:

`<script src="https://af.webshield.pro/t/v1.js?k=YOUR_KEY" data-forms="#signup, #contact_form" async></script>`

And that's it, away you go.

From then on, before accepting a submission, your backend asks for a score by session identifier and
receives a verdict with reasons.

Your code applies the decision — physically there is no way to stop a request from outside the
traffic path, and the documentation says so honestly, without embellishment.

The domain, incidentally, doesn't have to be moved to WebShield at all: forms are protected
separately. Convenient if a migration isn't on the cards just yet.

## Let's see what we got

After a day, open **Statistics**. That's where all the interesting stuff is, I promise.

**The "Protection" tab.** Attacks repelled, IPs blocked, fake search bots, browser checks shown,
direct hits on the check, suspicious ad clicks. A green zero in the first card is good. Excellent,
even.

**The "Bans" tab.** A live list of everyone the system is currently holding. Without trial or
investigation, but with a reason:

* **Blocked requests** — how many got cut off over the period;
* **Still knocking** — the bot hasn't given up and keeps trying. Persistent;
* **Action** — refusal at the edge, or being sent to the check;
* **Reason** — scanner, error spike, login brute force, bot farm;
* **Expires** — when the current block gets lifted.

A line reading "scanner · Still knocking · 14,302 requests" reconciles you with life better than
any advertisement. Honestly.

There they are — fourteen thousand-odd requests that no longer reached your PHP. And never will.

**The "Visitors" tab.** The "Humans and bots" chart, layered by category: humans, search engines, AI
bots, social networks, SEO, monitoring, scanners.

Unique visitors here means confirmed live browsers, so the figure is noticeably lower than in Yandex
Metrica. And, frankly, noticeably closer to the truth. Metrica counts everybody, vacuum cleaners
included.

**The "Site health" tab → "Search engines".** A table with crawl status: 🟢 crawling without
problems, 🟡 there are errors, 🔴 running into the protection, grey — hasn't visited in a long time.

Look in here after every tightening of the settings. See red next to Yandex or Google — don't think
twice: ease the mode or add exceptions. Immediately.

And a quick check by hand while the tab is open: `curl -sI https://example.com/robots.txt | head -1`,
and the same for `https://example.com/sitemap.xml`.

Both should return `200`. Got the check screen instead — congratulations, you've just closed your
sitemap off from the search engines. A genre classic, sadly.

## What stays on your side

The perimeter doesn't cancel hygiene, that's the thing. No proxy, however fancy, will save you from
an admin panel with the password `admin123`. Not one:

* CMS, plugin and theme updates — on a schedule, not "whenever we remember". We usually remember
  right after the break-in;
* `.env`, `.git`, database dumps — outside the web root. Always. Even "for five minutes".
  Especially for five minutes;
* two-factor on the admin panel, and the admin panel itself not at `/admin`, I beg you;
* a honeypot field in forms: hidden, invisible to a human. Filled in means the sender is a bot,
  simple as that;
* minimal file permissions, a separate user for the application;
* backups that somebody has actually tried restoring at least once. Not "we make them", but
  specifically "we've tried restoring them".

## What the protection doesn't do

The section that usually doesn't make it into the marketing brochures. Which is a shame. Here it is:

* **It doesn't catch 100% of malicious traffic.** And don't promise yourself it will. No system in
  the world does. The goal is more modest: strip away the bulk of it and raise the cost of an attack
  enough that it stops being interesting;
* **It doesn't parse the contents of submissions.** Manual spam, written by a living person, gets
  through. Well, what did you expect — they really are a person;
* **It doesn't affect rankings.** It reduces the bot load that skews your metrics. What Yandex or
  Google make of it in their results is their business; there are no guarantees on positions and
  there can't be;
* **It doesn't refund click fraud.** It identifies suspicious clicks and gives you an export of IPs
  and timestamps as evidence. The refund is processed by the ad network, and the decision is theirs;
* **It doesn't fix a hole in your code.** The perimeter cuts down the number of attempts, but the
  vulnerability isn't going anywhere. See the section above — that's what it's there for.

## Checklist

Run your eye over this before closing the tab. Everything ticked means you can sleep soundly:

* the domain is delegated, `dig +short NS` shows `nsbox`/`nshub`, no old NS records left — not
  a single one;
* `Proxy` is on for all of the site's `A`, `AAAA` and `CNAME` records;
* the server's real IP isn't sticking out anywhere, ports 80/443 are open only to the protection
  nodes;
* `Bot protection` is in `Browser` mode. At minimum;
* exceptions are set surgically: `/robots.txt`, `/sitemap*.xml`, API, websockets. The root —
  untouched;
* AI crawlers and surplus SEO bots are unticked, search engines are allowed;
* `robots.txt` and `sitemap.xml` return `200` with no check screen;
* "Site health → Search engines" — green for Yandex and Google;
* a day later, the "Bans" tab shows who the system is holding and why. And that's a pleasant sight.

Your logs will never fall completely silent, don't dream of it — the knocking never stops. But from
now on they'll be knocking at somebody else's door. One a block away.