Crawler IP Verifier - real Googlebot or fake

Name: Crawler IP Verifier - real Googlebot or fake
Author: pathwren

Tell real Googlebot/GPTBot from fake: 500 IPs a call, firewall allowlist out.

От сообществаРаботаетБез входаГлобальныйБесплатноМожет изменять данные

Что умеет

Verify Batch: Check up to 500 addresses against every mirrored operator prefix list in one call. Returns, per address, the matching prefix, the source that published it and when it was mirrored, plus
Lookup Prefix: Give a CIDR and get every published operator prefix that contains it, is contained by it, or partially overlaps it — the check for 'does my network collide with a crawler range' and for
Export Ip Acl: Turn a set of operators into a config you can paste: nginx geo, nginx allow/deny, Apache, HAProxy, a Cloudflare firewall expression, an ipset script, a Caddy matcher, or a plain CIDR li

Какие данные видит

Нужен ли аккаунт

Не нужен: сервер работает без входа

Tell real Googlebot/GPTBot from fake: 500 IPs a call, firewall allowlist out. Free, no key. Direct endpoint, no gateway: https://www.pathwren.workers.dev/c/smithery-direct/mcp/netcheck (streamable HTTP, no key, no signup). Tools: verify_batch, lookup_prefix, export_ip_acl, verification_methods, range_stats, example. Call the example tool with no arguments for a worked answer against real data before you write any code. What it does and how, in one page: https://www.pathwren.workers.dev/c/smithery-direct/mcp-netcheck.html

Список инструментов сервера (7)

Технические названия из tools/list. Нужны только разработчикам.

verify_batch	Check up to 500 addresses against every mirrored operator prefix list in one call. Returns, per address, the matching prefix, the source that published it and when it was mirrored, plus a rollup by operator. Accepts {ip, claim} pairs to test a claimed identity against the ranges, and says plainly that a miss is not evidence of a fake. Example: addresses=[{ip:'66.249.66.1', claim:'Googlebot'}, {ip:'203.0.113.9', claim:'GPTBot'}] — the first matches a published Google prefix, the second matches nothing. Also callable without MCP, same implementation: GET https://www.pathwren.workers.dev/tools/verify-crawler?ip=66.249.66.1&ua=Googlebot&s=client-dossiers
lookup_prefix	Give a CIDR and get every published operator prefix that contains it, is contained by it, or partially overlaps it — the check for 'does my network collide with a crawler range' and for auditing an allowlist somebody handed you. Give an operator or source slug instead and get everything that source publishes. Example: cidr='66.249.66.0/24'.
export_ip_acl	Turn a set of operators into a config you can paste: nginx geo, nginx allow/deny, Apache, HAProxy, a Cloudflare firewall expression, an ipset script, a Caddy matcher, or a plain CIDR list. Every export carries a provenance header naming each source URL and the mirror time, and reports the rule cost. Example: operators='all', format='cidr-list', action='allow'.
verification_methods	Per crawler: whether the operator publishes IP ranges, documents reverse DNS, or publishes no verification method at all — with the exact source URL or the exact dig commands, and how fresh this mirror is. Call it before reading anything into an address that matched nothing. Reverse DNS is never run here; the command is handed back instead. Example: crawler='claudebot'. Also callable without MCP, same implementation: GET https://www.pathwren.workers.dev/tools/verification-methods?crawler=claudebot&s=client-dossiers
range_stats	Takes no arguments. Safe to call. Deterministic. Touches no third party. Per source: prefix counts, total addresses and /24-equivalents, smallest and largest prefix, the publishing URL and when it was last fetched; plus prefixes published by more than one operator. This is the sizing and freshness information an allowlist decision needs. Example: {} — no arguments at all.
whoami	Takes no arguments. Safe to call. Deterministic. Touches no third party. Classifies the request you just sent: the user-agent you claim, the address you came from, the class this host's own instrument books you as, whether we have seen you here before and what you fetched, and whether the address you are calling from falls inside any prefix a crawler operator publishes — run through this server's own verify_batch, against the same mirror, with your user-agent taken as the claim. Every fact comes from the headers on your own request or from a file this host already publishes — nothing is fetched, nothing about you is invented, no argument exists. Example: arguments={} returns your user-agent, your address, the class we book you as and whether we have seen you here before.
example	Takes no arguments. Safe to call. Deterministic. Touches no third party. Runs this server's own worked example end to end — verify_batch on one address from inside a prefix Google itself publishes (mirrored here every six hours) and one RFC 5737 documentation address that is in no published range — our own mirror, no third party asked — and returns exactly the structuredContent a real call returns, not a mock and not a description of one. Use it to see the shape of an answer before you decide what to send. The input is canned from this host's own data; no URL of yours is fetched and no third party is touched. Example: arguments={} runs verify_batch with {"addresses":[{"ip":"66.249.66.1","claim":"Googlebot"},{"ip":"203.0.113.9","claim":"GPTBot and returns its real answer.