AgenticCrawler
Active·★ 15·MIT·Updated 2026-10-17
AgenticCrawler is a powerful browser automation agent implemented as a single Rust binary, capable of navigating, interacting, and extracting information from web pages. It extends beyond basic actions by integrating full DevTools observability, allowing agents to inspect network traffic, console errors, performance, and accessibility for comprehensive web interaction and debugging.
#Browser Automation#AI Agent#Web Scraping#DevTools#LLM Integration#Rust#Multi-Agent#Workflow Automation
01
Features
01One binary, zero runtimes: Self-contained Rust executable, no external runtimes (Python, Node, Docker) needed.
02Full DevTools Observability: Agents have first-class access to DevTools for network inspection, console logs, performance, and accessibility auditing.
03Smart Fetch Routing & Sub-Agent Parallelism: Intelligently switches between HTTP and headless browser for rendering, and supports concurrent multi-page crawls using sub-agents.
04Extensive LLM & MCP Integration: Compatible with 25+ LLM providers and functions as both an MCP client to use custom tools and an MCP server to expose its browser capabilities.
02
Compatibility
Rust
Runtime
Verified via docs
Node.js
Dependency
Verified via docs
Linux
OS
Verified via docs
macOS
OS
Verified via docs
Windows
OS
Verified via docs
03
Quick start
1
$ curl -fsSL https://raw.githubusercontent.com/Mingye-Lu/AgenticCrawler/main/install.sh | bash
04
Use cases
↳Scraping dynamic web content: Extract structured data (e.g., product catalogs, GitHub trending repos) from static or JavaScript-rendered pages, including pagination.
↳Automated form filling and submission: Programmatically fill out and submit web forms, confirming results.
↳Price monitoring and data extraction: Monitor specific data points (e.g., product prices) on single pages efficiently.
↳Parallel multi-page crawling: Speed up large-scale data collection by concurrently crawling multiple pages using sub-agents.
05
Alternatives
mcp-browser-use★ 62
FastAPI server implementing MCP protocol Browser automation via browser-use library.
invisible_playwright_mcp★ 873
Playwright MCP server undetected by anti-bots and captchas: AI agent browses the web on anti-detect stealth Firefox, Python, undetected browser automation, scraping, computer use.
neobrowser★ 100
MCP server that drives real Chrome with your real logged-in sessions — genuine fingerprint (passes bot.sannysoft), human-like input, bot-wall aware. 43 tools, single static Rust binary.
aginxbrowser★ 41
The browser built for AI agents — fetch live pages as markdown, render JS/SPAs with built-in V8, take screenshots without Chromium, meta-search 5 engines, and drive interactive login sessions. One Rust binary, stealth TLS fingerprints, MCP native for Claude Code & Cursor. Headless browser alternative to Puppeteer/Playwright.
browsentic★ 47
An agentic browser extension: an AI side panel in your own logged-in Chrome, Edge, Arc, Brave or even Firefox run by the agent you already use (Claude Code, Codex, Antigravity). Free, open source, no API key. Also an MCP server.
mcp-playwright★ 5.7k
Playwright Model Context Protocol Server - Tool to automate Browsers and APIs in Claude Desktop, Cline, Cursor IDE and More 🔌
Comments
Log in to leave a comment
No comments yet. Be the first!