AI Crawler Index

Name any bot user-agent, verify a crawler IP, or generate robots.txt.

CommunityOnlineNo sign-inGlobalFreeCan modify data

What it can do

  • Classify User Agent: Identify the crawler behind a raw User-Agent header: operator, category, robots.txt stance, how to verify it, and what blocking it costs. A user-agent is a claim, not proof — conf
  • Lookup Crawler: The full index record for one crawler: operator, category, robots.txt token and stance, user-agent substring and example, verification method, mirrored IP-range endpoint, and what bloc
  • List Crawlers: The index, filtered or whole: each row says what the crawler is and what blocking it costs, and the reply lists every valid filter value. Example: category='ai-training', limit=20.

What data it sees

Do you need an account

No: the server works without sign-in

Name any bot user-agent, verify a crawler IP, or generate robots.txt. Free, no key, no signup. Direct endpoint, no gateway: https://www.pathwren.workers.dev/c/smithery-direct/mcp (streamable HTTP, no key, no signup). Tools: classify_user_agent, lookup_crawler, list_crawlers, generate_robots_txt, is_verified_crawler_ip, whats_changed, example. Call the example tool with no arguments for a worked answer against real data before you write any code. What it does and how, in one page: https://www.pathwren.workers.dev/c/smithery-direct/mcp.html

How to connect

How to connect AI Crawler Index to Claude

The server address to paste into Claude settings:

https://ai-crawler-index--pathwren.run.tools

  1. Open Claude (claude.ai or the desktop app).
  2. Go to Settings → Connectors.
  3. Click Add custom connector.
  4. Paste the server address copied below into Remote MCP server URL and click Add.
  5. In a chat, click + → Connectors and switch the new connector on.

Custom connectors are available on Free, Pro, Max, Team and Enterprise plans (Free is limited to one). On Team and Enterprise an organization Owner adds the connector first under Organization settings → Connectors.

Authorization

After you click Add, a sign-in window for the service opens. Sign in with your own account and approve access. Claude never sees your password.

How to connect AI Crawler Index to ChatGPT

The server address to paste into ChatGPT settings:

https://ai-crawler-index--pathwren.run.tools

  1. Open ChatGPT in a browser (chatgpt.com). A Plus, Pro, Business, Enterprise or Edu plan is required.
  2. Turn on developer mode once: Settings → Apps → Advanced settings → Developer mode.
  3. Open Settings → Connectors and click Create.
  4. Fill in the form: Name (anything), Description (one line about what the service does), MCP server URL (copy it below).
  5. Under Authentication choose OAuth if the service requires sign-in, otherwise None. Click Create.
  6. In a new chat open + → Apps/Connectors and enable the connector.

OpenAI has renamed this section before (Connectors → Apps/Plugins). If the label differs, search settings for "developer mode". On Business/Enterprise workspaces an admin must allow custom connectors first.

Authorization

On first use ChatGPT opens the service's sign-in window. Sign in and approve access.

How to connect AI Crawler Index to Cursor

The server address to paste into Cursor settings:

https://ai-crawler-index--pathwren.run.tools

Fastest: click Open in Cursor below and confirm the prompt.

Manually:

  1. In Cursor open Settings → Cursor Settings → MCP and click Add new global MCP server.
  2. Paste the JSON copied below into ~/.cursor/mcp.json (per project: .cursor/mcp.json in the repo root).
  3. Save the file. The server appears in the MCP list; authorize it there if asked.

Authorization

If the service needs sign-in, an authorize button appears next to the server in the MCP list.

Server tool list (9)

Raw names from tools/list. Only developers need these.

classify_user_agentIdentify the crawler behind a raw User-Agent header: operator, category, robots.txt stance, how to verify it, and what blocking it costs. A user-agent is a claim, not proof — confirm with is_verified_crawler_ip. Example: user_agent='GPTBot/1.2' returns GPTBot, OpenAI, ai-training. Also callable without MCP, same implementation: GET https://www.pathwren.workers.dev/tools/classify-ua?ua=GPTBot%2F1.2&s=client-dossiers
lookup_crawlerThe full index record for one crawler: operator, category, robots.txt token and stance, user-agent substring and example, verification method, mirrored IP-range endpoint, and what blocking it costs. An unknown slug returns near matches. Example: slug='claudebot'.
list_crawlersThe index, filtered or whole: each row says what the crawler is and what blocking it costs, and the reply lists every valid filter value. Example: category='ai-training', limit=20.
generate_robots_txtA ready-to-paste robots.txt block for one stance, with the reasoning and the crawlers it names. Omit the argument to get all eight stances described. Example: stance='block-ai-training'.
is_verified_crawler_ipTest an IPv4 or IPv6 address against every crawler prefix the operators themselves publish, remirrored every six hours. Returns the matching prefix, its source and the crawlers using it; a miss is reported as a miss, not as a forgery. Example: ip='20.171.206.5'. Also callable without MCP, same implementation: GET https://www.pathwren.workers.dev/tools/verify-crawler?ip=20.171.206.5&s=client-dossiers
whats_changedFreshness and change state: when the index was built, when each operator source was last fetched, which changed, which are failing, and the dated changelog. The cron check before regenerating a config from this data. Example: since='2026-08-01'.
changes_sinceOnly what changed since your last call: prefixes added or removed, upstreams failed or recovered, records edited. Send back the `cursor` from the previous result; it advances only on a real change, and the data refreshes every 6h. Example: since='0'.
whoamiTakes no arguments. Safe to call. Deterministic. Touches no third party. Classifies the request you just sent: the user-agent you claim, the address you came from, the class this host's own instrument books you as, whether we have seen you here before and what you fetched, and what the crawler index makes of the user-agent you sent, and whether the address you came from is inside a prefix its operator publishes — both run through this server's own classify_user_agent and is_verified_crawler_ip, on the same data a stranger can curl. Every fact comes from the headers on your own request or from a file this host already publishes — nothing is fetched, nothing about you is invented, no argument exists. Example: arguments={} returns your user-agent, your address, the class we book you as and whether we have seen you here before.
exampleTakes no arguments. Safe to call. Deterministic. Touches no third party. Runs this server's own worked example end to end — classify_user_agent on the exact user-agent string of the busiest client in this host's own published log (/data/observed-clients.json, 1164 requests in the window) — our data, and no third party is asked anything — and returns exactly the structuredContent a real call returns, not a mock and not a description of one. Use it to see the shape of an answer before you decide what to send. The input is canned from this host's own data; no URL of yours is fetched and no third party is touched. Example: arguments={} runs classify_user_agent with {"user_agent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1. and returns its real answer.