spider

Spider is a crawling engine written in Rust that puts concurrency first. It fetches over HTTP and only launches headless Chrome on pages that actually need JavaScript, so most crawls stay fast. Both paths stream, so pages are delivered through a subscription channel as they arrive rather than batching at the end, and the same API drives one async task or a distributed worker fleet without code changes.

Configuration has sensible defaults, so you set only what you need: concurrency limit, depth, delay, subdomains, user agent, stealth and robots.txt handling. For JavaScript-heavy sites you enable the chrome feature and call crawl_smart(), which tries HTTP first and escalates to Chrome per page. It ships as a Rust crate, a CLI (spider_cli), Node.js (@spider-rs/spider-rs) and Python (spider_rs) packages, and an MCP server (spider_mcp) for Claude, Cursor and similar clients.

The same engine powers the commercial Spider Cloud, which handles proxies, headless browsers and anti-bot churn behind the same API; a Smart mode routes through proxies first and escalates to a stealth scrape only on pages that fight back. The open-source crate is MIT and runs fully locally with no key, with 50+ runnable examples. Respect each site's robots.txt and terms when crawling at scale.