anansi
Active·★ 118·Apache-2.0·Updated 2026-09-02
Anansi is a resilient web scraper that handles adversarial sites, survives redesigns, and improves over time. It features self-healing CSS selectors, automatic browser upgrade, TLS fingerprint mimicry, and adaptive rate limiting. It ships with an MCP server for LLM integration.
#adaptive-scraping#ai-agent#anti-bot#crawler#data-extraction#llm-tools#mcp#mcp-server
01
Features
01Self-healing parser with confidence-scored selectors and four healing strategies
02TLS/HTTP-2 fingerprint mimicry to bypass enterprise bot detection
03Automatic browser upgrade for JavaScript shells and Cloudflare challenges
04Incremental crawling with ETag, Last-Modified, and content hash comparison
05Structured data extraction from JSON-LD, Open Graph, and Microdata
02
Compatibility
Python
Python 3.8+
Verified via docs
03
Quick start
1
$ pip install "git+https://github.com/mdowis/anansi"
04
Use cases
↳Extract structured data from product pages with adaptive selectors
↳Concurrently crawl large websites with resilience and automatic recovery
↳Bypass anti-bot measures for authorized web scraping
05
Alternatives
x-twitter-scraper★ 217
X (Twitter) automation skill for AI coding agents. 60+ API endpoints, 20 MCP tools. Tweet search, user lookup, follower extraction, write actions (tweet, like, retweet, follow, DM), media download, account monitoring & trending topics. REST API, MCP server, HMAC webhooks. Works with Claude Code, Cursor, Codex, Copilot, Windsurf & 40+ agents.
apitap★ 130
The MCP server that turns any website into an API — no docs, no SDK, no browser. npm: @apitap/core
mcp-appstore★ 78
This is an MCP server that provides tools to LLMs for searching and analyzing apps from both Google Play Store and Apple App Store – perfect for ASO.
Comments
Log in to leave a comment
No comments yet. Be the first!