PyreCrawl
Active·★ 22·MIT·Updated 2026-09-21
PyreCrawl equips AI agents with robust web browsing capabilities, enabling self-hosted scraping, extraction, crawling, mapping, and searching without API keys or rate limits. It integrates via the Model Context Protocol (MCP) and features a smart auto-fallback ladder to ensure reliable web data access across diverse site protections.
#Web Scraping#AI Agent Tools#Browser Automation#Crawler#Content Extraction#Self-Hosted#Model Context Protocol#Cloudflare Bypass
01
Features
01Comprehensive Web Data Tools: Offers a suite of tools for single-page scraping, structured extraction, multi-page crawling with BFS depth, and site mapping.
02Intelligent Auto-Fallback: Employs a smart ladder that automatically tries HTTP, stealth browser (with Cloudflare solver), and deep processing to ensure successful data retrieval.
03Integrated Search & Document Processing: Features web search via DuckDuckGo, academic search (arXiv/Crossref), and the ability to convert PDFs/DOCX/PPTX to markdown, all without API keys.
04Persistent Sessions & Monitoring: Supports persistent browser sessions for complex workflows (e.g., logins) and robust change detection with diffing for web page elements.
02
Compatibility
Python
Runtime
Verified via docs
Model Context Protocol (MCP)
Standard Interface
Verified via docs
Claude, Cursor, VS Code, Codex, OpenCode, Hermes
Integrations
Verified via docs
Linux, macOS, Windows
Platform
Verified via docs
03
Quick start
1
$ uv tool install pyrecrawl
2
$ pyrecrawl setup
04
Use cases
↳Empowering AI Agents with Web Access: Enable AI models to interact with the entire web, from basic scraping to complex data extraction and navigation.
↳Automated Research & Information Gathering: Automatically collect and synthesize information from websites and academic sources for research projects, generating citation-rich evidence packs.
↳Monitoring Website Changes: Set up monitors to detect and track specific changes on competitor websites, product pages, or news sources.
↳Bypassing Anti-Bot Measures: Access and extract data from websites protected by Cloudflare or requiring full JavaScript rendering, leveraging stealth browser capabilities.
05
Alternatives
cloakbrowser-mcp★ 169
⚡ CloakBrowser MCP server for AI agents: Playwright-powered browsing, clean tool forwarding, Docker support, and multi-session HTTP transport.
wigolo★ 5.5k
The go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.
PyScrappy★ 263
Python web scraping toolkit and MCP server that gives AI agents clean, structured web data from any URL or built-in scrapers.
oya-browser★ 351
The browser control plane for AI agents. One API over Oya Cloud, Browserbase, Steel, Anchor, Browser Use and your own Chrome, with persistent personas, CAPTCHA and MFA handling, and live human takeover.
neobrowser★ 100
MCP server that drives real Chrome with your real logged-in sessions — genuine fingerprint (passes bot.sannysoft), human-like input, bot-wall aware. 43 tools, single static Rust binary.
aginxbrowser★ 41
The browser built for AI agents — fetch live pages as markdown, render JS/SPAs with built-in V8, take screenshots without Chromium, meta-search 5 engines, and drive interactive login sessions. One Rust binary, stealth TLS fingerprints, MCP native for Claude Code & Cursor. Headless browser alternative to Puppeteer/Playwright.
Comments
Log in to leave a comment
No comments yet. Be the first!