crw

fastCRW is a web data engine for AI agents, written in Rust and run as a small binary or through a managed API. It offers five operations: Scrape (one URL to markdown, HTML, links, screenshots or schema JSON), Crawl (a bounded crawl of a site), Map (discover URLs without scraping every page), Search (web search with optional scraping of results) and Extract (structured fields from one or many URLs). The README says crawl and map follow `robots.txt` by default and asks users to respect website policies.

Install with `curl -fsSL https://fastcrw.com/install | sh`, which runs locally for free with no account. If you pass `CRW_API_KEY`, the same installer connects your cloud key and registers the MCP server with tools it detects, such as Claude Code, Cursor, Codex, Gemini CLI, OpenCode and Windsurf; set `CRW_NO_AGENTS=1` to skip that step. The managed API adds proxies, JS rendering and search, with 1,000 free credits advertised, while self-hosting leaves renderers, search, auth, proxies and capacity to you.

The README makes benchmark claims: on Firecrawl's public 1,000-URL dataset it says fastCRW recovered more ground truth than Crawl4AI and Firecrawl with about 14 MB idle RAM, and that it answers 90.0% of 600 AA-Omniscience questions correctly. Both are the vendor's own results, with methodology linked. Licensing is split: the engine and MCP server are AGPL-3.0 and the Python and TypeScript SDKs are MIT, with a separate embedding licence available. Last push was 2026-10-13. This review is based on the README and repository data; we have not run it ourselves.