Crawl Census

Ask before you fetch: will this domain serve your crawler, refuse it, or charge it?

CommunityOnlineNo sign-inGlobalFreeRead-only

What it can do

    What data it sees

    Do you need an account

    No: the server works without sign-in

    Ask before you fetch: will this domain serve your crawler, refuse it, or charge it?

    How to connect

    How to connect Crawl Census to Claude

    The server address to paste into Claude settings:

    https://crawlcensus.com/mcp

    1. Open Claude (claude.ai or the desktop app).
    2. Go to Settings → Connectors.
    3. Click Add custom connector.
    4. Paste the server address copied below into Remote MCP server URL and click Add.
    5. In a chat, click + → Connectors and switch the new connector on.

    Custom connectors are available on Free, Pro, Max, Team and Enterprise plans (Free is limited to one). On Team and Enterprise an organization Owner adds the connector first under Organization settings → Connectors.

    Authorization

    After you click Add, a sign-in window for the service opens. Sign in with your own account and approve access. Claude never sees your password.

    How to connect Crawl Census to ChatGPT

    The server address to paste into ChatGPT settings:

    https://crawlcensus.com/mcp

    1. Open ChatGPT in a browser (chatgpt.com). A Plus, Pro, Business, Enterprise or Edu plan is required.
    2. Turn on developer mode once: Settings → Apps → Advanced settings → Developer mode.
    3. Open Settings → Connectors and click Create.
    4. Fill in the form: Name (anything), Description (one line about what the service does), MCP server URL (copy it below).
    5. Under Authentication choose OAuth if the service requires sign-in, otherwise None. Click Create.
    6. In a new chat open + → Apps/Connectors and enable the connector.

    OpenAI has renamed this section before (Connectors → Apps/Plugins). If the label differs, search settings for "developer mode". On Business/Enterprise workspaces an admin must allow custom connectors first.

    Authorization

    On first use ChatGPT opens the service's sign-in window. Sign in and approve access.

    How to connect Crawl Census to Cursor

    The server address to paste into Cursor settings:

    https://crawlcensus.com/mcp

    Fastest: click Open in Cursor below and confirm the prompt.

    Manually:

    1. In Cursor open Settings → Cursor Settings → MCP and click Add new global MCP server.
    2. Paste the JSON copied below into ~/.cursor/mcp.json (per project: .cursor/mcp.json in the repo root).
    3. Save the file. The server appears in the MCP list; authorize it there if asked.

    Authorization

    If the service needs sign-in, an authorize button appears next to the server in the MCP list.

    Server tool list (7)

    Raw names from tools/list. Only developers need these.

    scan_siteRun a live AI-accessibility audit of a domain: robots.txt policy for every tracked AI crawler, live user-agent probes, JavaScript-free readability, structured data and llms.txt. Returns a score out of 100 with per-check detail.
    site_reportReturn the most recent stored audit for a domain without triggering a new scan. Faster and free of load on the target site.
    census_statsCorpus-level statistics: how many measured domains block each AI crawler, mean access score, llms.txt adoption.
    crawl_preflightDecide whether a crawler may fetch a list of domains before spending requests on them. Works for any crawler token, not only the ones this census tracks: an unrecognised agent is resolved from each domain's stored robots.txt rather than refused. For each domain returns one of: allow (robots permits it and a live request carrying that agent's user agent was served), disallow (robots.txt forbids it), refuse (robots permits it but the edge refused the agent anyway, so the allowance is not real), pay (the origin answered HTTP 402 Payment Required, meaning it will serve this agent on commercial terms), or unknown. The full definition of each, including what it obliges a crawler to do, is published at https://crawlcensus.com/api/v1/verdicts. Built for crawler operators rather than site owners: it prevents wasted fetches against doors that are shut, and flags content an operator is trying to sell rather than withhold.
    agent_profileWhat this census measures and publishes about one AI crawler: how often it is disallowed in robots.txt, how often live requests carrying its user agent are refused at the network edge whatever robots.txt says, whether its operator documents it as honouring robots.txt, and where to correct any of that. Intended for the operator of the agent as much as for anyone studying it, so it includes the correction channel and the public page a claim can be disputed against.
    census_factsEvery headline finding from the census as discrete, dated records rather than prose. Each carries its value, unit, denominator, measurement date, the page it comes from and a ready-made citation line, plus the caveats that apply to all of them. Use this when answering a question about how open the web is to AI crawlers: lifting a percentage out of a rendered page loses the denominator and the date, which is what makes the number wrong when it is repeated.
    submit_domainsQueue domains the census has not measured yet so a later crawl_preflight can answer them. This closes the loop crawl_preflight starts: anything it returns as unknown with measurable true is worth submitting, and the reply names any that were already fresh or that this census will never measure, so a caller looping over its own unknowns converges instead of resubmitting the same set. Queueing is a database write rather than a fetch, so the allowance is far higher than scan_site and submitted domains are measured ahead of the ranked backlog.