crawlforge-mcp
CrawlForge MCP is a production-ready MCP server that gives AI agents the power to scrape websites, extract structured data, run deep research, bypass anti-bot…
What it can do
- Fetch Url: Fetch content from a URL with optional headers and timeout
- Extract Text: Extract clean text content from a webpage
- Extract Links: Extract all links from a webpage with optional filtering
What data it sees
Do you need an account
No: the server works without sign-in
CrawlForge MCP is a production-ready MCP server that gives AI agents the power to scrape websites, extract structured data, run deep research, bypass anti-bot detection, and process documents. It packages 27 specialized web scraping tools into a single MCP server with credit-based pricing and a free tier.
How to connect
How to connect crawlforge-mcp to Claude
The server address to paste into Claude settings:
https://crawlforge-mcp--crawlforgedev.run.tools
- Open Claude (claude.ai or the desktop app).
- Go to Settings → Connectors.
- Click Add custom connector.
- Paste the server address copied below into Remote MCP server URL and click Add.
- In a chat, click + → Connectors and switch the new connector on.
Custom connectors are available on Free, Pro, Max, Team and Enterprise plans (Free is limited to one). On Team and Enterprise an organization Owner adds the connector first under Organization settings → Connectors.
Authorization
After you click Add, a sign-in window for the service opens. Sign in with your own account and approve access. Claude never sees your password.
How to connect crawlforge-mcp to ChatGPT
The server address to paste into ChatGPT settings:
https://crawlforge-mcp--crawlforgedev.run.tools
- Open ChatGPT in a browser (chatgpt.com). A Plus, Pro, Business, Enterprise or Edu plan is required.
- Turn on developer mode once: Settings → Apps → Advanced settings → Developer mode.
- Open Settings → Connectors and click Create.
- Fill in the form: Name (anything), Description (one line about what the service does), MCP server URL (copy it below).
- Under Authentication choose OAuth if the service requires sign-in, otherwise None. Click Create.
- In a new chat open + → Apps/Connectors and enable the connector.
OpenAI has renamed this section before (Connectors → Apps/Plugins). If the label differs, search settings for "developer mode". On Business/Enterprise workspaces an admin must allow custom connectors first.
Authorization
On first use ChatGPT opens the service's sign-in window. Sign in and approve access.
How to connect crawlforge-mcp to Cursor
The server address to paste into Cursor settings:
https://crawlforge-mcp--crawlforgedev.run.tools
Fastest: click Open in Cursor below and confirm the prompt.
Manually:
- In Cursor open Settings → Cursor Settings → MCP and click Add new global MCP server.
- Paste the JSON copied below into
~/.cursor/mcp.json(per project:.cursor/mcp.jsonin the repo root). - Save the file. The server appears in the MCP list; authorize it there if asked.
Authorization
If the service needs sign-in, an authorize button appears next to the server in the MCP list.
Server tool list (20)
Raw names from tools/list. Only developers need these.
| fetch_url | Fetch content from a URL with optional headers and timeout |
| extract_text | Extract clean text content from a webpage |
| extract_links | Extract all links from a webpage with optional filtering |
| extract_metadata | Extract metadata from a webpage (title, description, keywords, etc.) |
| scrape_structured | Extract structured data from a webpage using CSS selectors |
| search_web | Search the web using Google Search API (proxied through CrawlForge) |
| crawl_deep | Crawl websites deeply using breadth-first search |
| map_site | Discover and map website structure |
| extract_content | Extract and analyze main content from web pages with enhanced readability detection |
| process_document | Process documents from multiple sources and formats including PDFs and web pages |
| summarize_content | Generate intelligent summaries of text content with configurable options |
| analyze_content | Perform comprehensive content analysis including language detection and topic extraction |
| extract_structured | Extract structured data from a webpage using LLM-powered analysis and a JSON Schema. Falls back to CSS selector extraction when no LLM provider is configured. |
| batch_scrape | Process multiple URLs simultaneously with support for async job management and webhook notifications |
| scrape_with_actions | Execute browser action chains before scraping content, with form auto-fill and intermediate state capture |
| deep_research | Conduct comprehensive multi-stage research with intelligent query expansion, source verification, and conflict detection |
| track_changes | Enhanced content change tracking with baseline capture, comparison, scheduled monitoring, advanced comparison engine, alert system, and historical analysis |
| generate_llms_txt | Analyze websites and generate standard-compliant LLMs.txt and LLMs-full.txt files defining AI model interaction guidelines |
| stealth_mode | Advanced anti-detection browser management with stealth features, fingerprint randomization, and human behavior simulation |
| localization | Multi-language and geo-location management with country-specific settings, browser locale emulation, timezone spoofing, and geo-blocked content handling |