Agent Web Index — AI crawler readiness

Can AI assistants actually read a website?

CommunityOnlineNo sign-inGlobalFreeCan modify data

What it can do

  • Domain Readiness: Can AI assistants actually read this domain? Returns the measured readiness score, the per-crawler verdict (served / blocked in robots.txt / refused by the edge despite robots.txt al
  • Web Openness: How much of the most-visited web is readable by AI assistants right now: the share of measured domains that block at least one AI crawler, and the served / robots-blocked / edge-blocked
  • Edge Blocking: Who is actually doing the blocking: for each CDN/WAF vendor, the share of (domain x crawler) pairs that robots.txt ALLOWS and the server refuses anyway — i.e. how much of the blocking i

What data it sees

Do you need an account

No: the server works without sign-in

Can AI assistants actually read a website? This server answers from the Agent Web Index: 45,000+ domains probed with real HTTP requests — once as a browser, once as each published AI crawler user-agent (GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot and others) — so it can tell apart what robots.txt says from what the server actually does.

Tools

  • domain_readiness — readiness score, per-crawler verdict and the edge vendor answering for a host.
  • web_readiness — the aggregate state of the measured web.
  • edge_blocking — refusal rate per CDN/WAF vendor, over domain×crawler pairs robots.txt allows.
  • measure_domain — measures any domain on demand: six real requests made while you wait.

No key, no signup. Dataset is CC BY 4.0. Most AI-crawler blocking is not in robots.txt — it is an edge default nobody chose, and this is the data that shows it.

How to connect

How to connect Agent Web Index — AI crawler readiness to Claude

The server address to paste into Claude settings:

https://ai-crawler-readiness--pfmcp-hq.run.tools

  1. Open Claude (claude.ai or the desktop app).
  2. Go to Settings → Connectors.
  3. Click Add custom connector.
  4. Paste the server address copied below into Remote MCP server URL and click Add.
  5. In a chat, click + → Connectors and switch the new connector on.

Custom connectors are available on Free, Pro, Max, Team and Enterprise plans (Free is limited to one). On Team and Enterprise an organization Owner adds the connector first under Organization settings → Connectors.

Authorization

After you click Add, a sign-in window for the service opens. Sign in with your own account and approve access. Claude never sees your password.

How to connect Agent Web Index — AI crawler readiness to ChatGPT

The server address to paste into ChatGPT settings:

https://ai-crawler-readiness--pfmcp-hq.run.tools

  1. Open ChatGPT in a browser (chatgpt.com). A Plus, Pro, Business, Enterprise or Edu plan is required.
  2. Turn on developer mode once: Settings → Apps → Advanced settings → Developer mode.
  3. Open Settings → Connectors and click Create.
  4. Fill in the form: Name (anything), Description (one line about what the service does), MCP server URL (copy it below).
  5. Under Authentication choose OAuth if the service requires sign-in, otherwise None. Click Create.
  6. In a new chat open + → Apps/Connectors and enable the connector.

OpenAI has renamed this section before (Connectors → Apps/Plugins). If the label differs, search settings for "developer mode". On Business/Enterprise workspaces an admin must allow custom connectors first.

Authorization

On first use ChatGPT opens the service's sign-in window. Sign in and approve access.

How to connect Agent Web Index — AI crawler readiness to Cursor

The server address to paste into Cursor settings:

https://ai-crawler-readiness--pfmcp-hq.run.tools

Fastest: click Open in Cursor below and confirm the prompt.

Manually:

  1. In Cursor open Settings → Cursor Settings → MCP and click Add new global MCP server.
  2. Paste the JSON copied below into ~/.cursor/mcp.json (per project: .cursor/mcp.json in the repo root).
  3. Save the file. The server appears in the MCP list; authorize it there if asked.

Authorization

If the service needs sign-in, an authorize button appears next to the server in the MCP list.

Server tool list (5)

Raw names from tools/list. Only developers need these.

domain_readinessCan AI assistants actually read this domain? Returns the measured readiness score, the per-crawler verdict (served / blocked in robots.txt / refused by the edge despite robots.txt allowing it), and which edge vendor answers for the host. Measured with real HTTP requests, not guessed from robots.txt alone. Google-Extended and Applebot-Extended are robots.txt tokens, not crawlers: they are flagged robotsOnly, carry no server verdict, and are excluded from every count about what the server did. Also returns `fixes`: the ordered, domain-specific list of what to change to let the blocked crawlers in, separating what is written in the site own robots.txt from what the CDN/WAF applies on top of it.
web_opennessHow much of the most-visited web is readable by AI assistants right now: the share of measured domains that block at least one AI crawler, and the served / robots-blocked / edge-blocked breakdown per crawler.
edge_blockingWho is actually doing the blocking: for each CDN/WAF vendor, the share of (domain x crawler) pairs that robots.txt ALLOWS and the server refuses anyway — i.e. how much of the blocking is an infrastructure default rather than a decision the site owner made. Each vendor also comes broken down per crawler, which separates a blanket wall (same rate for every crawler) from a managed block list that names some AI user-agents and not others.
recent_changesWhat CHANGED: the domains that recently started or stopped blocking a specific AI crawler, with the day the flip was observed and whether it happened in robots.txt or at the edge. This cannot be reconstructed after the fact from any public source — it exists only because the index made the same requests the day before and the day after. Use it to answer 'who just blocked/unblocked ChatGPT, Claude, Perplexity...' or to watch one domain over time.
measure_domainMeasure a domain RIGHT NOW instead of reading the archive: 9 real HTTP requests, one from a browser user-agent and one per AI crawler user-agent, plus robots.txt / llms.txt / sitemap. Use it for any site the index has not reached yet, or when the caller wants a fresh verdict after changing robots.txt or a WAF rule. Returns the same `fixes` list as domain_readiness, derived from the fresh measurement.