Firecrawl
APIfree tieropen source core
Use it for: Scraping or crawling pages into LLM-ready markdown or JSON with one call, plus an MCP server for Claude Code and Cursor.
Tradeoff: Credit-based usage; heavily protected sites can still need a real browser session.
Browserbase
hosted browsersusage-based
Use it for: Agents that must log in, click through flows or handle JavaScript-heavy apps, without running Chrome yourself.
Tradeoff: You write the browser automation; slower and heavier than a fetch-and-clean API.
Playwright (self-run)
open sourceself-hosted
Use it for: Full browser control on your machine or CI, free, with Microsoft's Playwright MCP for agents.
Tradeoff: You handle proxies, blocking, scaling and turning HTML into clean text.
Exa
search APIfree tier
Use it for: Semantic web search that returns relevant pages and their contents for research agents.
Tradeoff: A search engine, not a crawler: less suited to pulling every page of one site.
Tavily
search APIfree tier
Use it for: A search step built for agent loops, with extract and crawl endpoints and many framework integrations.
Tradeoff: Results are tuned for answers; for exact page structure use a scraper.
Apify
marketplacefree tierusage-based
Use it for: Prebuilt scrapers (Actors) for common sites, scheduling and storage when you need data at volume.
Tradeoff: Actor quality varies by author; costs depend on the Actor and compute used.
ScrapingBee
APIproxies
Use it for: Fetching pages through managed proxies and headless browsers when sites block plain requests.
Tradeoff: Returns HTML first; you still clean it into text for the model.
Crawl4AI
open sourceself-hostedPython
Use it for: A free, LLM-friendly crawler you run yourself, with markdown output and extraction strategies.
Tradeoff: You run and maintain it, including browsers and anti-bot handling.