AgentIndex icon
AgentIndex
ToolsMCP ServersHOTCategoriesTrendingNewBest ListsCompareGuides
Submit Tool
ToolsMCP ServersHOTCategoriesTrendingNewBest ListsCompareGuides
Home/
Browser Automation/
crw
crw logo

crw

Active·★ 1.3k·AGPL-3.0·Updated 2026-11-21
★ Essential★ Trending

fastCRW is a web data engine for AI agents, written in Rust and run as a small binary or through a managed API. It offers five operations: Scrape (one URL to markdown, HTML, links, screenshots or schema JSON), Crawl (a bounded crawl of a site), Map (discover URLs without scraping every page), Search (web search with optional scraping of results) and Extract (structured fields from one or many URLs). The README says crawl and map follow `robots.txt` by default and asks users to respect website policies.

Install with `curl -fsSL https://fastcrw.com/install | sh`, which runs locally for free with no account. If you pass `CRW_API_KEY`, the same installer connects your cloud key and registers the MCP server with tools it detects, such as Claude Code, Cursor, Codex, Gemini CLI, OpenCode and Windsurf; set `CRW_NO_AGENTS=1` to skip that step. The managed API adds proxies, JS rendering and search, with 1,000 free credits advertised, while self-hosting leaves renderers, search, auth, proxies and capacity to you.

The README makes benchmark claims: on Firecrawl's public 1,000-URL dataset it says fastCRW recovered more ground truth than Crawl4AI and Firecrawl with about 14 MB idle RAM, and that it answers 90.0% of 600 AA-Omniscience questions correctly. Both are the vendor's own results, with methodology linked. Licensing is split: the engine and MCP server are AGPL-3.0 and the Python and TypeScript SDKs are MIT, with a separate embedding licence available. Last push was 2026-10-13. This review is based on the README and repository data; we have not run it ourselves.

This write-up is based on the project's README and public repository data. We have not hand-tested the tool. Data as of 2026-11-19.

License
AGPL-3.0
Latest release
v0.46.2 · 2026-11-18
Last commit
2026-11-18
Contributors
17
Forks
102
Open issues and PRs
8
First published
2026-03-02
npm package
crw-sdk 0.46.2 · 1,637 downloads last week
PyPI package
crw 0.46.2 · 1,009 downloads last week

Source: GitHub, npm and PyPI public APIs.

#Web Scraping#Data Extraction#Content Transformation#Web Crawler#API#CLI#AI Agent Tooling#Rust
$ Install
$ curl -fsSL https://fastcrw.com/install | sh
↗ Visit site★ GitHub
01

Features

01Scrape one URL to markdown, HTML, links, screenshots, or schema JSON
02Crawl a bounded site and collect its pages
03Map and discover URLs without scraping every page
04Search the web and optionally scrape selected results
05Extract structured fields from one or many URLs
02

Why choose it

+Five operations in one engine: scrape, crawl, map, search and extract, as markdown or JSON
+Small Rust binary that runs locally for free, with a managed API option
+Installer can register its MCP server with Claude Code, Cursor, Codex, Gemini CLI and others
+Crawl and map follow robots.txt by default; SDKs are MIT
03

Trade-offs

!Engine and MCP server are AGPL-3.0, which may not suit closed-source products; embedding needs a separate licence
!Accuracy, latency and memory claims are the vendor's own benchmarks
!Managed proxies and JS rendering are part of the paid cloud, while self-hosting leaves you to set them up
!Scraping must respect site terms and law; installing via `curl | sh` means you should read the script first
04

Maintenance check

ActiveChecked: 2026-11-06
Last commit
2026-11-06
Commits, last 90 days
100+
Releases, last 90 days
18
Latest release
v0.42.1 (2026-11-06)
Open issues
7
Recent issues with at least one comment
10/10
Commits per week, last 12 weeks
Commits per week, last 12 weeks: 20, 27, 29, 44, 37, 13, 23, 5, 54, 15, 18, 78

Oldest week on the left; the latest four weeks are orange. Source: GitHub commit activity.

Status rule: archived on GitHub = Archived; last commit within 90 days = Active; 91 to 365 days = Slowing; older = Stale. All figures come from the GitHub API on the date shown.

Most discussed open issues

  • #660 `onlyMainContent` (default true) still emits filter UI, cart drawer, live-chat widget and cookie/browser-warning text5 comments · 2026-11-01
  • #658 Infinite-list / "Load more" collection pages return only the first page of results; `?page=N` returns identical content and there is no way 2 comments · 2026-11-01
  • #637 camoufox tier: reuse Cloudflare clearance across requests (per-host session) and allow camoufox-first ordering1 comments · 2026-10-16
  • #590 Add crw to awesome-ai-plugins?0 comments · 2026-09-29
  • #586 Proposal: optional fee-less per-request Nano (XNO) settlement rail beside credit billing0 comments · 2026-09-25

Issue titles are quoted from GitHub. Open the link to check whether a problem still applies to the version you use.

05

Compatibility

Rust
Runtime
Verified via docs
MacOS
OS
Verified via docs
Linux
OS
Verified via docs
Python
SDK
Verified via docs
Node.js
SDK
Verified via docs
AI Agents (MCP)
Integration
Verified via docs
06

Quick start

1
$ curl -fsSL https://fastcrw.com/install | sh
07

Use cases

↳Converting web pages into structured data (markdown/JSON) for further processing or storage.
↳Building AI agents that require real-time and structured information from the web.
↳Collecting data for market research, competitive analysis, or content aggregation.
↳Monitoring website changes or tracking specific data points across multiple pages.
↳Integrating web data extraction capabilities into custom applications via SDKs or REST API.
08

Alternatives

⚡Top VS:
VSwigolo★5.6k→VSfirecrawl-mcp-server★7.9k→VSaginxbrowser★56→VSneobrowser★100→
wigolo logo
wigolo★ 5.6k
The go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.
VSCompare→
firecrawl-mcp-server logo
firecrawl-mcp-server★ 7.9k
🔥 Official Firecrawl MCP Server - Adds powerful web scraping and search to Cursor, Claude and any other LLM clients.
VSCompare→
PyScrappy logo
PyScrappy★ 265
Python web scraping toolkit and MCP server that gives AI agents clean, structured web data from any URL or built-in scrapers.
aginxbrowser logo
aginxbrowser★ 56
The browser built for AI agents — fetch live pages as markdown, render JS/SPAs with built-in V8, take screenshots without Chromium, meta-search 5 engines, and drive interactive login sessions. One Rust binary, stealth TLS fingerprints, MCP native for Claude Code & Cursor. Headless browser alternative to Puppeteer/Playwright.
VSCompare→
neobrowser logo
neobrowser★ 100
MCP server that drives real Chrome with your real logged-in sessions — genuine fingerprint (passes bot.sannysoft), human-like input, bot-wall aware. 43 tools, single static Rust binary.
VSCompare→
browser-mcp logo
browser-mcp★ 66
Drive your real, logged-in Chrome from any AI agent (Claude Code, Cursor, VS Code) — works where headless dies. Reads emailed login codes from your Gmail, solves CAPTCHAs, 34 tools. MIT, local-only.
VSCompare→
anymd logo
anymd★ 1.1k
Any file → clean Markdown for AI agents: PDF, Word, PowerPoint, Excel, EPUB, HTML and web pages, images (OCR), audio and video (metadata, subtitles, transcripts). A fast Rust MCP server and CLI that runs on your machine. No API key.
VSCompare→
scrapfly-mcp logo
scrapfly-mcp★ 15
Official Scrapfly MCP server for Cursor, Claude Desktop, and any MCP-compatible client. Enterprise-grade web scraping, AI extraction, and anti-bot–aware data access as first-class tools.
See all alternatives →

Comments

Log in to leave a comment

No comments yet. Be the first!

On this page
01Features02Why choose it03Trade-offs04Maintenance check05Compatibility06Quick start07Use cases08Alternatives
fastcrw /
crw
↗

AGPL-3.0 Rust engine that turns URLs into markdown or JSON with search, scrape, map, crawl and extract, runnable locally or as a managed API

1.3k102AGPL-3.0
Last commit1d ago
StatusActive
CategoryBrowser Automation
Contributors17
Latest releasev0.46.2
Links
Documentation↗Discussion↗Issues↗Releases↗
Cloud Infrastructure
Ad

Deploy on DigitalOcean

Host your MCP servers, Docker containers, and AI agents with $200 free credit over 60 days.

Claim $200 Free Credit
© 2026 AgentIndex.app|Built by a 10-year iOS Developer.
Best by use caseAboutContactAdvertisePrivacy PolicyTerms of UseQYSGitHubBuy me a coffee ☕

Browse by Category

MCP ServersCode AssistantWorkflow AutomationRAG / Knowledge BaseMulti-AgentBrowser AutomationLLM InfraDev ToolingObservability

Not affiliated with Anthropic, OpenAI or Microsoft.