Web scraping and crawling APIs and MCP servers for AI agents
Crawl and scrape websites, convert pages to Markdown/JSON, and extract structured data through scraping APIs with proxies and JS rendering.
12 web scraping and crawling APIs an AI agent can call: auth method and header, official MCP server (9), free tier (8), docs links. Machine-readable JSON included. Agents: fetch catalog/web-scraping-browser.json instead of parsing this page. Last checked 2026-11-04.
Ratings and usage (★) fetched: .
| Service | What an agent can do | Auth | MCP server | Free tier | Details (with sources) | Notes | Proof (ratings, usage) |
|---|---|---|---|---|---|---|---|
| Apify | Run web scraping/automation 'Actors' from a marketplace of thousands, and manage datasets, via REST API or MCP server. | API key / OAuthAuthorization: Bearer <token> for REST; browser-based OAuth2 or bearer token for hosted MCP | official | free plan: $5/mo platform usage credit, no card required | Base URL: https://api.apify.com/v2 Free plan: $5 prepaid platform usage, per month, no card ↗ Rate limits: 250,000 requests/min global per user; 60 req/s per resource by default (higher for some endpoints) ↗ Async jobs: POST run returns run object; poll GET run or use webhooks; sync run endpoints wait up to 300 s ↗ Example call: run an Actor synchronously and get its dataset items: POST https://api.apify.com/v2/actors/{actorId}/run-sync-get-dataset-items ↗ Unit price: Platform usage: $0.20 per compute unit (1 GB RAM for 1 hour) on Free and Starter plans; some Store Actors add their own per-result or per-event fees ↗ MCP tools: 7: search-actors, fetch-actor-details, call-actor, search-apify-docs, fetch-apify-docs, get-dataset-items … ↗ | MCP search/docs tools work unauthenticated; running paid Actors consumes platform usage credit. Actor endpoints use /v2/actors/{actorId}/... (e.g. run-sync-get-dataset-items) | |
| Bright Data | Access Web Unlocker, SERP API, Scraping Browser, and structured datasets via REST API or MCP to pull web data at scale. | API keyAuthorization: Bearer <token> | official | Free tier: 5,000 requests/month via the MCP server (renews on the 1st, no card required); other products pay-as-you-go | Base URL: https://api.brightdata.com Free plan: 5,000 MCP requests, per month ↗ Example call: fetch a web page through Web Unlocker: POST https://api.brightdata.com/request ↗ Unit price: Web Unlocker pay-as-you-go: $1.5 per 1,000 successful requests ↗ MCP tools: 5: search_engine, search_engine_batch, scrape_as_markdown, scrape_batch, discover ↗ | Large multi-product suite; pricing and free-tier limits vary significantly by product (Web Unlocker vs Scraping Browser vs Datasets). | |
| Browse AI API | Trigger pre-built or custom 'robots' to extract and monitor web data via REST API, returning structured JSON. | API keyAuthorization: Bearer <key> | — | free forever plan: 50 credits/mo, no card required | Base URL: https://api.browse.ai/v2 Free plan: 50 credits, per month, no card ↗ Async jobs: robot tasks run asynchronously; get results via webhooks or by retrieving the task ↗ Example call: run a robot (extraction task): POST https://api.browse.ai/v2/robots/{robotId}/tasks ↗ Unit price: 1 credit per 10 rows extracted or per screenshot; premium sites cost 2-10 credits minimum per task; extra credits about $0.013-$0.030 each depending on plan and billing period ↗ | — | |
| Diffbot | Automatically extract structured data (articles, products, people) from any URL via REST API, no custom parsing rules needed. | API keytoken= query parameter | — | free plan: 10,000 credits/mo, no card required (5 req/min limit) | Base URL: https://api.diffbot.com/v3 Free plan: 10,000 credits, per month, no card ↗ Rate limits: Free plan: 5 requests/min ↗ Example call: extract an article from a URL: GET https://api.diffbot.com/v3/article?url={url}&token={token} ↗ Unit price: Extract: 1 credit per page; overage $0.001/credit (Startup), $0.0009/credit (Plus) ↗ | Also offers Knowledge Graph and Crawl products with separate pricing; official client library docs listed as 'coming soon'. | |
| Firecrawl | Scrape, crawl, and extract structured data from any website via a single REST API call. | API keyAuthorization: Bearer <fc-key> | official | free tier: 1,000 credits/mo, no card required | Base URL: https://api.firecrawl.dev Free plan: 1,000 credits, per month, no card ↗ Rate limits: Free: 2 concurrent requests; scrape/map/search 10/min; crawl and agent 2/min ↗ Async jobs: crawl returns a job id; poll GET /v2/crawl/{id} or use webhooks (crawl.page, crawl.completed) ↗ Example call: scrape a page to markdown: POST https://api.firecrawl.dev/v2/scrape ↗ Unit price: Scrape: 1 credit per page; Hobby plan 5,000 credits/month for $16/mo billed yearly ($19 monthly); free plan 1,000 credits ↗ MCP tools: 11: firecrawl_scrape, firecrawl_search, firecrawl_map, firecrawl_crawl, firecrawl_check_crawl_status, firecrawl_parse … ↗ | Also offers OAuth-based remote MCP endpoint (mcp.firecrawl.dev/v2/mcp-oauth) and a keyless trial with daily limits. | |
| Oxylabs | Fetch web pages, search-engine results, and e-commerce data at scale via REST API with built-in proxy/unblocking infrastructure. | API keyHTTP Basic Auth with API-user credentials (--user USERNAME:PASSWORD) | official | Trial: Web Scraper API free trial with up to 2,000 results one-time, no card required | Base URL: https://realtime.oxylabs.io/v1 Free plan: up to 2,000 results (Web Scraper API), one-time, no card ↗ Example call: scrape a web page (realtime): POST https://realtime.oxylabs.io/v1/queries ↗ Unit price: Web Scraper API (Free Trial/Micro): $1.15 per 1,000 results for other sites without JS ($1.00 Google, $0.50 Amazon), $1.35 per 1,000 with JS rendering; failed requests not charged ↗ MCP tools: 10: universal_scraper, google_search_scraper, amazon_search_scraper, amazon_product_scraper, ai_scraper, ai_crawler … ↗ | Enterprise-oriented pricing overall, but self-serve API-user creation and free trial are available via the dashboard. Official MCP is hosted at mcp.oxylabs.io; its GitHub repo was archived on 2026-09-16. | |
| ScrapeGraphAI API | Use AI-driven prompts to scrape and extract structured data from webpages via REST API or Python/JS SDK. | API keySGAI-APIKEY: <key> header | official | Trial: 500 API credits one-time on the Free plan (10 requests/min), no card required | Base URL: https://v2-api.scrapegraphai.com/api Free plan: 500 API credits one-time (10 requests/min), one-time, no card ↗ Rate limits: Free plan: 10 requests/min, 1 concurrent crawl, 1 monitor; status polling not counted ↗ Async jobs: crawl is async: start returns an id, poll the crawl get-status endpoint ↗ Example call: extract structured data from a web page with a prompt: POST https://v2-api.scrapegraphai.com/api/extract ↗ Unit price: Extract: 5 credits per call (+5 with stealth); scrape to markdown 1 credit. Starter $20/mo for 10,000 credits (about $0.01 per extract); credit packs from $5 per 1,000 credits ↗ MCP tools: 20: scrape, extract, search, crawl_start, crawl_get, crawl_pages … ↗ | Also offers the open-source scrapegraph-ai library with LangChain/LlamaIndex integrations. | |
| ScraperAPI | Scrape any webpage with automatic proxy rotation, CAPTCHA handling, and JS rendering via a single REST API call. | API keyapi_key query parameter | official | Free: free plan with 1,000 API credits per month (max 5 concurrent connections); new accounts start with a 7-day trial of 5,000 one-time credits, no card required | Base URL: https://api.scraperapi.com Free plan: 1,000 API credits (after a 7-day 5,000-credit trial), max 5 concurrent connections, per month ↗ Rate limits: Free plan: max 5 concurrent connections ↗ Async jobs: optional async endpoint https://async.scraperapi.com returns a job to fetch later ↗ Example call: scrape a web page: GET https://api.scraperapi.com/?api_key={API_KEY}&url={URL} ↗ Unit price: 1 API credit per normal request; render=true or premium=true 10 credits; Google/Bing SERP 25 credits; only successful (200/404) requests are charged ↗ | Widely used alternative to ScrapingBee/Zyte; official hosted MCP server at mcp.scraperapi.com/mcp. | — |
| Scrapfly | Scrape webpages, run cloud browsers, and capture screenshots via REST API or hosted MCP server with anti-bot bypass. | API key / OAuthkey= query param for REST API; API key or OAuth2 for MCP server | official | 1,000 free credits/mo, no card required, no time limit | Base URL: https://api.scrapfly.io Free plan: 1,000 API credits, no card ↗ Rate limits: Free plan: 5 concurrent requests; concurrency defined by subscription ↗ Async jobs: optional: pass webhook_name to queue the request and receive the result on a webhook ↗ Example call: scrape a web page: GET https://api.scrapfly.io/scrape?key={API_KEY}&url={url} ↗ Unit price: Basic scrape (datacenter proxy, no JS) = 1 credit; with JS rendering = 6 credits; residential proxy = 25 credits. Extra credits $5.00 per 10k on Discovery ($30/mo, 200k credits) ↗ MCP tools: 11: scraping_instruction_enhanced, web_get_page, web_scrape, screenshot, check_if_blocked, cloud_browser_open … ↗ | MCP usage draws from the same credit balance as direct API calls. | |
| ScrapingBee | Scrape any webpage with JS rendering, proxy rotation, and screenshot capture via a single REST API call. | API keyAuthorization: Bearer <key> (legacy: api_key query param, deprecated) | official | Trial: 1,000 free API credits one-time on signup, no card required; then Hobby plan $19/mo for 75,000 credits | Base URL: https://app.scrapingbee.com/api/v1 Free plan: 1,000 free API credits, one-time, no card ↗ Example call: scrape a web page: GET https://app.scrapingbee.com/api/v1?url={url} ↗ Unit price: 1 credit per request without JS rendering, 5 credits with JS rendering (default), 10/25 with premium proxy; Hobby plan $19/mo for 75,000 credits ↗ MCP tools: 14: get_page_text, get_page_html, extract_page_data, get_screenshot, get_file, fast_search … ↗ | Postman collection provided in place of a published OpenAPI spec. Hosted MCP takes the API key as an api_key query parameter on the server URL. | |
| Spider.cloud | Crawl and scrape websites at scale, returning JSON/XML/CSV, via REST API, CLI, or MCP server. | API keyAuthorization: Bearer sk-<key> | official | Free: keyless /scrape endpoint, rate-limited (quota not published); other endpoints return 402 without a key | Base URL: https://api.spider.cloud Free plan: keyless /scrape, rate-limited (quota not published), no card ↗ Example call: scrape one page to markdown: POST https://api.spider.cloud/scrape ↗ Unit price: Pay-as-you-go per request: $1 per GB of fetched content plus $0.0001 per CPU-minute; failed requests free (10,000 credits = $1) ↗ MCP tools: 22: spider_crawl, spider_scrape, spider_search, spider_links, spider_screenshot, spider_unblocker … ↗ | ||
| Zyte API | Fetch and auto-extract web pages (HTML, browser rendering, product/article data) via one HTTP API with ban avoidance. | API keyHTTP Basic Auth: API key as username, blank password | — | Trial: $5 free credit one-time on signup, valid 30 days, no subscription needed; then pay-as-you-go or a monthly commitment | Base URL: https://api.zyte.com/v1 Free plan: $5 free credit, one-time ↗ Rate limits: Per-API-key rate limit (429 /limits/over-user-limit); retry with exponential backoff ↗ Example call: fetch a web page (raw HTTP response body): POST https://api.zyte.com/v1/extract ↗ Unit price: Pay-as-you-go: HTTP requests $0.13-$1.27 per 1,000 and browser-rendered requests $1.01-$16.08 per 1,000, depending on the site's difficulty tier; lower with monthly commitments ↗ | Part of Zyte's broader suite (Scrapy Cloud, Zyte IDE). |
Other categories
- Web search
- Browser automation and cloud browsers
- Knowledge and research data
- Image generation and editing
- Video generation and editing
- Speech and audio
- Music generation
- 3D generation and assets
- Architecture, CAD and BIM
- Diagrams and software architecture
- Design and UI
- Documents, OCR and presentations
- Translation and language
- Code execution sandboxes
- Developer platforms and DevOps
- Cloud and hosting
- Databases
- Vector search, memory and RAG
- Email, team chat and notifications
- SMS and messaging apps
- Voice agents and telephony
- Productivity and workspace
- CRM, support and marketing
- Social media
- Maps, geolocation and weather
- Finance, payments and market data
- E-commerce
- Automation and integration platforms
- Model APIs and inference