AgentIndex icon
AgentIndex
ToolsMCP ServersHOTCategoriesTrendingNewBest ListsCompareGuides
Submit Tool
ToolsMCP ServersHOTCategoriesTrendingNewBest ListsCompareGuides
Home/
Browser Automation/
Scrapling
Scrapling logo

Scrapling

Active·★ 103.9k·BSD-3-Clause·Updated 2026-11-17

Scrapling is an adaptive Python web scraping framework that scales from a single request to a full crawl. Its standout idea is adaptive parsing: elements you select can be saved and automatically relocated when a website's layout changes, so scrapers break less often. It combines fast HTTP fetching that can impersonate browser fingerprints, full browser automation for dynamic pages, and a Scrapy-like spider framework.

It suits developers and data teams collecting public web data, and AI builders who want agents to fetch and extract web content. The spider framework supports concurrency limits, per-domain throttling, automatic retry of blocked requests, pause and resume, streaming results and exports to JSON, CSV and XML, with optional robots.txt compliance. An MCP server lets AI assistants use Scrapling directly, and an interactive shell helps you develop selectors.

Getting started: requires Python 3.10+. Install with `pip install scrapling`; for browser-based fetchers install `scrapling[fetchers]` and run `scrapling install`, and for the MCP server install `scrapling[ai]`.

This write-up is based on the project's README and public repository data. We have not hand-tested the tool. Data as of 2026-11-14.

License
BSD-3-Clause
Latest release
v0.5.1 · 2026-11-04
Last commit
2026-11-14
Contributors
54
Forks
10,717
Open issues and PRs
21
First published
2024-10-13
PyPI package
scrapling 0.5.1 · 384,878 downloads last week

Source: GitHub, npm and PyPI public APIs.

#Web Scraping#Python#Crawler Framework#Anti-bot Bypass#Adaptive Parsing#Proxy Rotation#Headless Browser#CLI Tool
$ Install
$ pip install scrapling
↗ Visit site★ GitHub
01

Features

01Full-featured crawling framework with Scrapy-like API, concurrent requests, multi-session support, and pause/resume functionality.
02Advanced fetchers with anti-bot bypass, dynamic loading, persistent session management, and integrated proxy rotation.
03Adaptive scraping capabilities with smart element tracking and flexible selection methods, including AI integration via MCP server.
04High-performance and memory-efficient architecture, optimized for speed and battle-tested in production environments.
02

Why choose it

+Adaptive selectors that survive website redesigns reduce scraper maintenance.
+One framework covers fast HTTP requests, dynamic browser pages and large crawls.
+Production crawling features: throttling, retries, pause and resume, streaming and built-in exporters.
+MCP server for AI assistants; BSD-3-Clause licensed.
03

Trade-offs

!Its anti-bot bypass capabilities make it easy to scrape sites that do not want to be scraped; check each site's terms, robots.txt and applicable laws before collecting data.
!Browser-based fetchers need extra installation and much more memory and CPU than plain HTTP requests.
!The README is heavy with proxy-provider sponsorships; you do not need any of them to use the library.
05

Maintenance check

ActiveChecked: 2026-11-08
Last commit
2026-11-08
Commits, last 90 days
100+
Releases, last 90 days
4
Latest release
v0.5.1 (2026-11-04)
Open issues
16
Recent issues with at least one comment
6/9
Commits per week, last 12 weeks
Commits per week, last 12 weeks: 10, 3, 5, 11, 40, 7, 33, 44, 9, 10, 26, 1

Oldest week on the left; the latest four weeks are orange. Source: GitHub commit activity.

Status rule: archived on GitHub = Archived; last commit within 90 days = Active; 91 to 365 days = Slowing; older = Stale. All figures come from the GitHub API on the date shown.

Most discussed open issues

  • #563 <base href> ignored when resolving relative URLs1 comments · 2026-11-05
  • #558 Response.follow() inherits credentials when following links to a different origin0 comments · 2026-11-05
  • #557 cannot access online sites like coupang.0 comments · 2026-11-04

Issue titles are quoted from GitHub. Open the link to check whether a problem still applies to the version you use.

04

Who it fits

Pick it if
  • You need an ultra-fast, adaptive Python web scraping engine that handles modern anti-bot challenges (Cloudflare, fingerprinting) effortlessly.
  • You want flexible fetchers ranging from lightweight HTTP to full headless browsers with intelligent element extraction.
Look elsewhere if
  • You need a full-blown mobile app automation framework; mobile-mcp covers iOS and Android devices.
  • You need an enterprise distributed crawling queue with persistent database scheduling out of the box.
06

How it differs

mobile-mcp logovs mobile-mcp★ 10.7k

Scrapling is a high-speed Python web scraping engine built for modern websites and anti-bot bypass. mobile-mcp controls mobile OS screens and mobile applications via MCP.

Full comparison →
07

Compatibility

Python
Runtime
Verified via docs
Docker
Deployment
Verified via docs
08

Quick start

1
$ pip install scrapling
09

Use cases

↳Performing single web requests or launching full-scale, concurrent web crawls.
↳Adapting to website design changes by automatically relocating scraped elements.
↳Bypassing advanced anti-bot systems such as Cloudflare Turnstile.
↳Scaling up to multi-session crawls with proxy rotation and checkpoint-based persistence.
↳Leveraging AI for assisted web scraping and targeted data extraction.
10

Alternatives

⚡Top VS:
VSmobile-mcp★10.7k→VSlinkedin-mcp-server★4.2k→VSapify-mcp-server★16.7k→VSbrightdata-mcp★2.7k→
mobile-mcp logo
mobile-mcp★ 10.7k
Model Context Protocol Server for Mobile Automation and Scraping (iOS, Android, Emulators, Simulators and Real Devices)
VSCompare→
linkedin-mcp-server logo
linkedin-mcp-server★ 4.2k
This MCP server allows Claude and other AI assistants to access your LinkedIn. Scrape LinkedIn profiles, companies and jobs, and perform job searches.
VSCompare→
apify-mcp-server logo
apify-mcp-server★ 16.7k
The Apify MCP server enables your AI agents to extract data from social media, search engines, maps, e-commerce sites, or any other website using thousands of ready-made scrapers, crawlers, and automation tools available on the Apify Store.
VSCompare→
brightdata-mcp logo
brightdata-mcp★ 2.7k
A powerful Model Context Protocol (MCP) server that provides an all-in-one solution for public web access.
VSCompare→
Jobs_Applier_AI_Agent_AIHawk logo
Jobs_Applier_AI_Agent_AIHawk★ 32.6k
Give it your resume, get the jobs that fit, select them and apply. Runs in your terminal, with a browser you can watch.
VSCompare→
aginxbrowser logo
aginxbrowser★ 56
The browser built for AI agents — fetch live pages as markdown, render JS/SPAs with built-in V8, take screenshots without Chromium, meta-search 5 engines, and drive interactive login sessions. One Rust binary, stealth TLS fingerprints, MCP native for Claude Code & Cursor. Headless browser alternative to Puppeteer/Playwright.
VSCompare→
steel-mcp-server logo
steel-mcp-server★ 58
MCP Server for interacting with a Steel web browser
VSCompare→
mcp-browser-use logo
mcp-browser-use★ 63
FastAPI server implementing MCP protocol Browser automation via browser-use library.
VSCompare→
See all alternatives →

Featured in these lists

  • Best MCP Servers for Browser Automation

Comments

Log in to leave a comment

No comments yet. Be the first!

On this page
01Features02Why choose it03Trade-offs05Maintenance check04Who it fits06How it differs07Compatibility08Quick start09Use cases10Alternatives
D4Vinci /
Scrapling
↗

Adaptive Python web scraping framework with anti-bot fetchers

103.9k11kBSD-3-Clause
Last commit1d ago
StatusActive
CategoryBrowser Automation
Contributors54
Latest releasev0.5.1
Links
Documentation↗Discussion↗Issues↗Releases↗
Cloud Infrastructure
Ad

Deploy on DigitalOcean

Host your MCP servers, Docker containers, and AI agents with $200 free credit over 60 days.

Claim $200 Free Credit
© 2026 AgentIndex.app|Built by a 10-year iOS Developer.
Best by use caseAboutContactAdvertisePrivacy PolicyTerms of UseQYSGitHubBuy me a coffee ☕

Browse by Category

MCP ServersCode AssistantWorkflow AutomationRAG / Knowledge BaseMulti-AgentBrowser AutomationLLM InfraDev ToolingObservability

Not affiliated with Anthropic, OpenAI or Microsoft.