Web scraping and browser automation APIs and MCP servers for AI agents

Crawl, scrape and convert pages to Markdown/JSON, or drive a remote headless browser (clicks, forms, logins).

18 web scraping and browser automation APIs an AI agent can call: auth method and header, official MCP server (15), free tier (12), docs links. Machine-readable JSON included. Agents: fetch catalog/web-scraping-browser.json instead of parsing this page. Last checked 2026-09-24.

Ratings and usage (★) fetched: 2026-09-24 16:54 UTC – 2026-09-24 17:02 UTC.

Service	What an agent can do	Auth	MCP server	Free tier	Details (with sources)	Notes	Proof (ratings, usage)
Anchor Browser	Provision authenticated cloud browser sessions and turn any website into callable MCP tools for agents.	API key `anchor-api-key header`	official	$5 free credits/mo, no card required	Base URL: https://api.anchorbrowser.io Free plan: $5 free credits, per month, no card ↗ Rate limits: Free plan: session creation capped at 5/min, up to 5 concurrent browsers ↗ Example call: start a cloud browser session: POST https://api.anchorbrowser.io/v1/sessions ↗ Unit price: $0.01 per browser created + $0.05 per browser hour; Anchor proxy $8/GB ↗	Authenticated (logged-in) browser sessions require a paid tier; free tier covers unauthenticated browsing only.	SourceForge 5.0★/1 review (2026-09-24 16:55 UTC)
Apify	Run web scraping/automation 'Actors' from a marketplace of thousands, and manage datasets, via REST API or MCP server.	API key / OAuth `Authorization: Bearer <token> for REST; browser-based OAuth2 or bearer token for hosted MCP`	official	free plan: $5/mo platform usage credit, no card required	Base URL: https://api.apify.com/v2 Free plan: $5 prepaid platform usage, per month, no card ↗ Rate limits: 250,000 requests/min global per user; 60 req/s per resource by default (higher for some endpoints) ↗ Async jobs: POST run returns run object; poll GET run or use webhooks; sync run endpoints wait up to 300 s ↗ Example call: run an Actor synchronously and get its dataset items: POST https://api.apify.com/v2/actors/{actorId}/run-sync-get-dataset-items ↗ Unit price: Platform usage: $0.20 per compute unit (1 GB RAM for 1 hour) on Free and Starter plans; some Store Actors add their own per-result or per-event fees ↗ MCP tools: 7: search-actors, fetch-actor-details, call-actor, search-apify-docs, fetch-apify-docs, get-dataset-items … ↗	MCP search/docs tools work unauthenticated; running paid Actors consumes platform usage credit. Actor endpoints use /v2/actors/{actorId}/... (e.g. run-sync-get-dataset-items)	Smithery 218 uses (2026-09-24 16:54 UTC) SourceForge 4.8★/1,714 reviews (2026-09-24 17:02 UTC)
Bright Data	Access Web Unlocker, SERP API, Scraping Browser, and structured datasets via REST API or MCP to pull web data at scale.	API key `Authorization: Bearer <token>`	official	Free tier: 5,000 requests/month via the MCP server (renews on the 1st, no card required); other products pay-as-you-go	Base URL: https://api.brightdata.com Free plan: 5,000 MCP requests, per month ↗ Example call: fetch a web page through Web Unlocker: POST https://api.brightdata.com/request ↗ Unit price: Web Unlocker pay-as-you-go: $1.5 per 1,000 successful requests ↗ MCP tools: 5: search_engine, search_engine_batch, scrape_as_markdown, scrape_batch, discover ↗	Large multi-product suite; pricing and free-tier limits vary significantly by product (Web Unlocker vs Scraping Browser vs Datasets).	GitHub 2,655★ brightdata/brightdata-mcp (2026-09-24 16:54 UTC) Smithery 300 uses (2026-09-24 16:54 UTC) SourceForge 4.4★/1 review (2026-09-24 16:55 UTC)
Browse AI API	Trigger pre-built or custom 'robots' to extract and monitor web data via REST API, returning structured JSON.	API key `Authorization: Bearer <key>`	—	free forever plan: 50 credits/mo, no card required	Base URL: https://api.browse.ai/v2 Free plan: 50 credits, per month, no card ↗ Async jobs: robot tasks run asynchronously; get results via webhooks or by retrieving the task ↗ Example call: run a robot (extraction task): POST https://api.browse.ai/v2/robots/{robotId}/tasks ↗ Unit price: 1 credit per 10 rows extracted or per screenshot; premium sites cost 2-10 credits minimum per task; extra credits about $0.013-$0.030 each depending on plan and billing period ↗		—
Browser Use Cloud API	Launch cloud browser sessions and run AI browser-automation agent tasks via REST API or Python/TypeScript SDK.	API key `X-Browser-Use-API-Key header (key prefix bu_)`	official	$15 one-time signup credit for new accounts (Google/GitHub/Microsoft); paid usage afterward	Base URL: https://api.browser-use.com/api/v4 Free plan: $15 signup credits (Google/GitHub/Microsoft signups), one-time, no card ↗ Rate limits: Per-project default max(25, 2x concurrency) req/s, capped at 100 RPS; edge WAF 1000 RPS per IP ↗ Async jobs: tasks run asynchronously; poll the run/session status until complete ↗ Example call: run an AI browser task: POST https://api.browser-use.com/api/v4/runs ↗ Unit price: Agent run: model token cost + 20% service fee (e.g. Claude Sonnet 5 $2.40 in / $12.00 out per 1M tokens); cloud browser $0.02/hour ↗ MCP tools: 7: run_session, get_session, send_task, stop_session, get_session_messages, list_sessions … ↗	Pay-as-you-go: model cost + 20% and browsers at $0.02/hour, no per-run fees; no ongoing free tier beyond the signup credit.	—
Browserbase (+ Stagehand)	Run and control cloud headless browsers programmatically, with Stagehand adding natural-language page actions.	API key `X-BB-API-Key header, plus Project ID`	official	Free tier: 1 browser hour/month, 3 concurrent browsers, 15 min/session, no card required	Base URL: https://api.browserbase.com/v1 Free plan: 1 browser hour, 3 concurrent browsers, 15 min/session, per month ↗ Rate limits: Free plan: 3 concurrent browsers, 15 minutes per session ↗ Data policy: Free plan data retention limited to 7 days ↗ Example call: create a cloud browser session: POST https://api.browserbase.com/v1/sessions ↗ Unit price: Browser time beyond plan: $0.12 per browser hour (Developer, $20/mo incl. 100 h) or $0.10 (Startup, $99/mo incl. 500 h); proxies $12 or $10 per GB ↗ MCP tools: 6: start, end, navigate, act, observe, extract ↗	Billed by browser-hours; Stagehand SDK is open source but needs a browser backend such as Browserbase. Hosted MCP takes the key as ?browserbaseApiKey=; the mcp-server-browserbase repo is archived since 2026-07-20.	Smithery 433 uses (2026-09-24 16:54 UTC) SourceForge 3.0★/2 reviews (2026-09-24 16:55 UTC)
Diffbot	Automatically extract structured data (articles, products, people) from any URL via REST API, no custom parsing rules needed.	API key `token= query parameter`	—	free plan: 10,000 credits/mo, no card required (5 req/min limit)	Base URL: https://api.diffbot.com/v3 Free plan: 10,000 credits, per month, no card ↗ Rate limits: Free plan: 5 requests/min ↗ Example call: extract an article from a URL: GET https://api.diffbot.com/v3/article?url={url}&token={token} ↗ Unit price: Extract: 1 credit per page; overage $0.001/credit (Startup), $0.0009/credit (Plus) ↗	Also offers Knowledge Graph and Crawl products with separate pricing; official client library docs listed as 'coming soon'.	—
Firecrawl	Scrape, crawl, and extract structured data from any website via a single REST API call.	API key `Authorization: Bearer <fc-key>`	official	free tier: 1,000 credits/mo, no card required	Base URL: https://api.firecrawl.dev Free plan: 1,000 credits, per month, no card ↗ Rate limits: Free: 2 concurrent requests; scrape/map/search 10/min; crawl and agent 2/min ↗ Async jobs: crawl returns a job id; poll GET /v2/crawl/{id} or use webhooks (crawl.page, crawl.completed) ↗ Example call: scrape a page to markdown: POST https://api.firecrawl.dev/v2/scrape ↗ Unit price: Scrape: 1 credit per page; Hobby plan 5,000 credits/month for $16/mo billed yearly ($19 monthly); free plan 1,000 credits ↗ MCP tools: 11: firecrawl_scrape, firecrawl_search, firecrawl_map, firecrawl_crawl, firecrawl_check_crawl_status, firecrawl_parse … ↗	Also offers OAuth-based remote MCP endpoint (mcp.firecrawl.dev/v2/mcp-oauth) and a keyless trial with daily limits.	SourceForge 5.0★/1 review (2026-09-24 16:55 UTC)
Hyperbrowser	Spin up cloud browsers and call scrape, crawl, and structured-extract endpoints via REST API or MCP server.	API key `x-api-key header`	official	free plan: starter credits, 1 concurrent browser, no card required	Base URL: https://api.hyperbrowser.ai MCP tools: 10: scrape_webpage, crawl_webpages, extract_structured_data, search_with_bing, browser_use_agent, openai_computer_use_agent … ↗	SDKs are compatible with Playwright, Puppeteer, and Selenium.	GitHub 792★ hyperbrowserai/mcp (2026-09-24 16:54 UTC)
Oxylabs	Fetch web pages, search-engine results, and e-commerce data at scale via REST API with built-in proxy/unblocking infrastructure.	API key `HTTP Basic Auth with API-user credentials (--user USERNAME:PASSWORD)`	official	Trial: Web Scraper API free trial with up to 2,000 results one-time, no card required	Base URL: https://realtime.oxylabs.io/v1 Free plan: up to 2,000 results (Web Scraper API), one-time, no card ↗ Example call: scrape a web page (realtime): POST https://realtime.oxylabs.io/v1/queries ↗ Unit price: Web Scraper API (Free Trial/Micro): $1.15 per 1,000 results for other sites without JS ($1.00 Google, $0.50 Amazon), $1.35 per 1,000 with JS rendering; failed requests not charged ↗ MCP tools: 10: universal_scraper, google_search_scraper, amazon_search_scraper, amazon_product_scraper, ai_scraper, ai_crawler … ↗	Enterprise-oriented pricing overall, but self-serve API-user creation and free trial are available via the dashboard. Official MCP is hosted at mcp.oxylabs.io; its GitHub repo was archived on 2026-09-16.	SourceForge 4.2★/1,211 reviews (2026-09-24 17:02 UTC)
Playwright MCP (Microsoft)	Official Microsoft MCP server giving agents structured browser control (click, type, navigate) via Playwright's accessibility tree.	none `no auth; runs as a local process via npx`	official	free, open source, no key	Free plan: no_key, no card ↗ MCP tools: 18: browser_navigate, browser_navigate_back, browser_click, browser_type, browser_press_key, browser_hover … ↗	Runs locally/self-hosted via `npx @playwright/mcp`; no cloud hosting, account, or API key required, but needs a local or self-managed browser runtime. Apache-2.0 licensed.	GitHub 37.5k★ microsoft/playwright-mcp (2026-09-24 16:54 UTC)
ScrapeGraphAI API	Use AI-driven prompts to scrape and extract structured data from webpages via REST API or Python/JS SDK.	API key `SGAI-APIKEY: <key> header`	official	Trial: 500 API credits one-time on the Free plan (10 requests/min), no card required	Base URL: https://v2-api.scrapegraphai.com/api Free plan: 500 API credits one-time (10 requests/min), one-time, no card ↗ Rate limits: Free plan: 10 requests/min, 1 concurrent crawl, 1 monitor; status polling not counted ↗ Async jobs: crawl is async: start returns an id, poll the crawl get-status endpoint ↗ Example call: extract structured data from a web page with a prompt: POST https://v2-api.scrapegraphai.com/api/extract ↗ Unit price: Extract: 5 credits per call (+5 with stealth); scrape to markdown 1 credit. Starter $20/mo for 10,000 credits (about $0.01 per extract); credit packs from $5 per 1,000 credits ↗ MCP tools: 20: scrape, extract, search, crawl_start, crawl_get, crawl_pages … ↗	Also offers the open-source scrapegraph-ai library with LangChain/LlamaIndex integrations.	—
ScraperAPI	Scrape any webpage with automatic proxy rotation, CAPTCHA handling, and JS rendering via a single REST API call.	API key `api_key query parameter`	official	Free: free plan with 1,000 API credits per month (max 5 concurrent connections); new accounts start with a 7-day trial of 5,000 one-time credits, no card required	Base URL: https://api.scraperapi.com Free plan: 1,000 API credits (after a 7-day 5,000-credit trial), max 5 concurrent connections, per month ↗ Rate limits: Free plan: max 5 concurrent connections ↗ Async jobs: optional async endpoint https://async.scraperapi.com returns a job to fetch later ↗ Example call: scrape a web page: GET https://api.scraperapi.com/?api_key={API_KEY}&url={URL} ↗ Unit price: 1 API credit per normal request; render=true or premium=true 10 credits; Google/Bing SERP 25 credits; only successful (200/404) requests are charged ↗	Widely used alternative to ScrapingBee/Zyte; official hosted MCP server at mcp.scraperapi.com/mcp.	—
Scrapfly	Scrape webpages, run cloud browsers, and capture screenshots via REST API or hosted MCP server with anti-bot bypass.	API key / OAuth `key= query param for REST API; API key or OAuth2 for MCP server`	official	1,000 free credits/mo, no card required, no time limit	Base URL: https://api.scrapfly.io Free plan: 1,000 API credits, no card ↗ Rate limits: Free plan: 5 concurrent requests; concurrency defined by subscription ↗ Async jobs: optional: pass webhook_name to queue the request and receive the result on a webhook ↗ Example call: scrape a web page: GET https://api.scrapfly.io/scrape?key={API_KEY}&url={url} ↗ Unit price: Basic scrape (datacenter proxy, no JS) = 1 credit; with JS rendering = 6 credits; residential proxy = 25 credits. Extra credits $5.00 per 10k on Discovery ($30/mo, 200k credits) ↗ MCP tools: 11: scraping_instruction_enhanced, web_get_page, web_scrape, screenshot, check_if_blocked, cloud_browser_open … ↗	MCP usage draws from the same credit balance as direct API calls.	—
ScrapingBee	Scrape any webpage with JS rendering, proxy rotation, and screenshot capture via a single REST API call.	API key `Authorization: Bearer <key> (legacy: api_key query param, deprecated)`	official	Trial: 1,000 free API credits one-time on signup, no card required; then Hobby plan $19/mo for 75,000 credits	Base URL: https://app.scrapingbee.com/api/v1 Free plan: 1,000 free API credits, one-time, no card ↗ Example call: scrape a web page: GET https://app.scrapingbee.com/api/v1?url={url} ↗ Unit price: 1 credit per request without JS rendering, 5 credits with JS rendering (default), 10/25 with premium proxy; Hobby plan $19/mo for 75,000 credits ↗ MCP tools: 14: get_page_text, get_page_html, extract_page_data, get_screenshot, get_file, fast_search … ↗	Postman collection provided in place of a published OpenAPI spec. Hosted MCP takes the API key as an api_key query parameter on the server URL.	—
Spider.cloud	Crawl and scrape websites at scale, returning JSON/XML/CSV, via REST API, CLI, or MCP server.	API key `Authorization: Bearer sk-<key>`	official	Free: keyless /scrape endpoint, rate-limited (quota not published); other endpoints return 402 without a key	Base URL: https://api.spider.cloud Free plan: keyless /scrape, rate-limited (quota not published), no card ↗ Example call: scrape one page to markdown: POST https://api.spider.cloud/scrape ↗ Unit price: Pay-as-you-go per request: $1 per GB of fetched content plus $0.0001 per CPU-minute; failed requests free (10,000 credits = $1) ↗ MCP tools: 22: spider_crawl, spider_scrape, spider_search, spider_links, spider_screenshot, spider_unblocker … ↗		—
Steel	Create managed cloud browser sessions with stealth proxies and CAPTCHA solving, controllable via REST API or Playwright/Puppeteer.	API key `steel-api-key header`	official	Trial: Launch plan is $0 + usage with $30 one-time usage credits valid 90 days (10 concurrent sessions, 60 requests/min); no Hobby plan	Base URL: https://api.steel.dev Free plan: $30 usage credits (Launch plan, valid 90 days), one-time, no card ↗ Rate limits: Launch plan: 60 requests/min, 10 concurrent sessions, 15 min max session ↗ Data policy: Launch plan: 7 days data retention ↗ Example call: create a cloud browser session: POST https://api.steel.dev/v1/sessions ↗ Unit price: Browser hours: $0.10/hour on Launch plan (CAPTCHA solves $3 per 1,000, proxy $10/GB) ↗ MCP tools: 16: steel_scrape, steel_screenshot, steel_pdf, steel_session_create, steel_session_release, steel_navigate … ↗	Open-source self-hostable browser engine (steel-browser) also available; official MCP server at steel-dev/steel-mcp-server.	—
Zyte API	Fetch and auto-extract web pages (HTML, browser rendering, product/article data) via one HTTP API with ban avoidance.	API key `HTTP Basic Auth: API key as username, blank password`	—	Trial: $5 free credit one-time on signup, valid 30 days, no subscription needed; then pay-as-you-go or a monthly commitment	Base URL: https://api.zyte.com/v1 Free plan: $5 free credit, one-time ↗ Rate limits: Per-API-key rate limit (429 /limits/over-user-limit); retry with exponential backoff ↗ Example call: fetch a web page (raw HTTP response body): POST https://api.zyte.com/v1/extract ↗ Unit price: Pay-as-you-go: HTTP requests $0.13-$1.27 per 1,000 and browser-rendered requests $1.01-$16.08 per 1,000, depending on the site's difficulty tier; lower with monthly commitments ↗	Part of Zyte's broader suite (Scrapy Cloud, Zyte IDE).	—

Web scraping and browser automation APIs and MCP servers for AI agents

Other categories