Crawlie

Technical SEO + AI-search audits for agents.

CommunityOnlineNo sign-inGlobalFreeCan modify data

What it can do

  • Crawl Site: Run a hosted technical-SEO + GEO audit of a whole website in Crawlie Cloud. Returns a compact digest (headline, scores, prioritized top fixes, issues grouped by rule with sample URLs) plus
  • Audit Url: Audit a single URL in Crawlie Cloud (no crawling beyond the page). Returns the compact SEO + GEO digest and a reportId. Counts as one crawl against your plan.
  • Audit Urls: Audit an explicit list of URLs in Crawlie Cloud (no link following). One report for the whole batch — use for spot-checking a set of pages. Each page audited counts against your plan's cre

What data it sees

Do you need an account

No: the server works without sign-in

Technical SEO + AI-search audits for agents. Crawl any website and get a structured report your agent can actually work with — broken links, redirect chains, missing titles/canonicals, duplicate content, structured data, and GEO (generative engine optimization) readiness: is your site answerable by ChatGPT, Claude, and Perplexity?

What your agent can do

  • Audit a single URL, a list, or crawl an entire site
  • Slice saved reports without re-crawling — top fixes by impact, affected URLs per issue, link graph, GEO gaps
  • Explain any issue — why it matters and how to fix it, with agent-actionable output
  • Monitor continuously — create a project, set a crawl schedule, diff reports to catch regressions after deploys
  • Gate releases — run a pre-launch check and fail on broken links, 5xx, or noindex slips

Try asking

  • "Crawl mysite.com and give me the top 10 fixes by impact"
  • "What changed since the last crawl of my docs site?"
  • "Is my site ready to be cited by AI search engines?"

Sign in with OAuth — no API key to paste. Free to start. Also available as an open-source desktop crawler at crawlie.co.

How to connect

How to connect Crawlie to Claude

The server address to paste into Claude settings:

https://crawlie.run.tools

  1. Open Claude (claude.ai or the desktop app).
  2. Go to Settings → Connectors.
  3. Click Add custom connector.
  4. Paste the server address copied below into Remote MCP server URL and click Add.
  5. In a chat, click + → Connectors and switch the new connector on.

Custom connectors are available on Free, Pro, Max, Team and Enterprise plans (Free is limited to one). On Team and Enterprise an organization Owner adds the connector first under Organization settings → Connectors.

Authorization

After you click Add, a sign-in window for the service opens. Sign in with your own account and approve access. Claude never sees your password.

How to connect Crawlie to ChatGPT

The server address to paste into ChatGPT settings:

https://crawlie.run.tools

  1. Open ChatGPT in a browser (chatgpt.com). A Plus, Pro, Business, Enterprise or Edu plan is required.
  2. Turn on developer mode once: Settings → Apps → Advanced settings → Developer mode.
  3. Open Settings → Connectors and click Create.
  4. Fill in the form: Name (anything), Description (one line about what the service does), MCP server URL (copy it below).
  5. Under Authentication choose OAuth if the service requires sign-in, otherwise None. Click Create.
  6. In a new chat open + → Apps/Connectors and enable the connector.

OpenAI has renamed this section before (Connectors → Apps/Plugins). If the label differs, search settings for "developer mode". On Business/Enterprise workspaces an admin must allow custom connectors first.

Authorization

On first use ChatGPT opens the service's sign-in window. Sign in and approve access.

How to connect Crawlie to Cursor

The server address to paste into Cursor settings:

https://crawlie.run.tools

Fastest: click Open in Cursor below and confirm the prompt.

Manually:

  1. In Cursor open Settings → Cursor Settings → MCP and click Add new global MCP server.
  2. Paste the JSON copied below into ~/.cursor/mcp.json (per project: .cursor/mcp.json in the repo root).
  3. Save the file. The server appears in the MCP list; authorize it there if asked.

Authorization

If the service needs sign-in, an authorize button appears next to the server in the MCP list.

Server tool list (18)

Raw names from tools/list. Only developers need these.

crawl_siteRun a hosted technical-SEO + GEO audit of a whole website in Crawlie Cloud. Returns a compact digest (headline, scores, prioritized top fixes, issues grouped by rule with sample URLs) plus a reportId you can re-slice later without re-crawling. Long crawls stream progress; if one exceeds the request window you get a jobId to poll with crawl_status. Counts as one crawl against your plan.
audit_urlAudit a single URL in Crawlie Cloud (no crawling beyond the page). Returns the compact SEO + GEO digest and a reportId. Counts as one crawl against your plan.
audit_urlsAudit an explicit list of URLs in Crawlie Cloud (no link following). One report for the whole batch — use for spot-checking a set of pages. Each page audited counts against your plan's credits.
crawl_statusPoll a crawl started by crawl_site that exceeded the request window. Returns progress while running, or the full digest + reportId once done.
list_reportsList saved hosted reports for your team (id, url, scores, page counts).
get_reportLoad a saved hosted report. Compact digest by default; set includeIssues/includePages for the full flat lists.
top_fixesPrioritized fixes for a saved report, optionally scoped to a category (e.g. category='geo'). Operates on the latest report by default.
geo_gapsAI-readiness (GEO) issues for a saved report, grouped by rule with sample URLs. Latest report by default.
affected_urlsList URLs flagged by a specific rule in a saved report (e.g. rule='geo-no-author').
diff_reportsCompare two saved crawls of the same site: score deltas, pages added/removed, and issues that newly appeared or were resolved. Use it to verify fixes landed between crawls.
link_graphInternal-link analytics for a saved report: top authority pages (internal PageRank), biggest hubs, orphan pages, dead ends, max click depth, and reciprocal-link counts. Latest report by default.
explain_issueWhy a Crawlie rule matters, how to fix it, and its impact — e.g. rule='geo-no-llms-txt'. Use list_rules to browse rule ids.
list_rulesThe catalog of built-in SEO/GEO/accessibility checks Crawlie runs (rule id, title, category, severity), optionally filtered by category. Use explain_issue for the full why/how-to-fix text.
whoamiThe signed-in Crawlie Cloud account: email, active team, plan limits, and this period's crawl/credit usage.
list_projectsSaved sites (projects) for your team, with their crawl schedule, alert settings, last scores and next run time.
create_projectSave a site as a project so it can be re-crawled on a schedule with regression alerts. Set the schedule here or later with set_schedule. Scheduling requires a paid plan; the project itself is free.
set_scheduleUpdate a project's re-crawl schedule and regression alerts (email and/or a Slack/Discord-compatible webhook). Alerts fire when a scheduled crawl's health drops or new error-severity issues appear. Scheduling requires a paid plan.
project_trendScore history for a monitored project: health/GEO/accessibility, errors, warnings and page counts per crawl, oldest first. Use it to see whether a site is improving.