Mindket

Scrapling

Newversion 1 · updated
GitHub trending
Star99.9k

Overview

This Python web scraping framework combines fast HTTP requests, browser automation, and data parsing into one library. It includes an MCP server for AI agents to collect data, render JavaScript, and manage concurrent crawling tasks.

Scrapling is an open-source project on GitHub in the Developer category, listed by Mindket. It is free. This is version 1, last updated on . Install it from GitHub as its README says. Get Mindket’s setup notes on your Mindket account, as a zip.

At a glance

Runs in
  • Python scripts
  • Terminal environments
  • AI agents
Start with
git clone https://github.com/D4Vinci/Scrapling.git
cd Scrapling

About

What it is

Scrapling is a Python web scraping framework. It combines fast HTTP requests, stealth browsers, parsing tools, and concurrent crawling into a single library. You can scrape static pages, render JavaScript, interact with sites, and extract data from backend API calls.

Who it suits

This framework helps developers who need to scrape data or build web-connected AI agents.

Where to use it

  • Python scripts
  • Terminal environments
  • AI agents

How to set it up

  1. Install the package.
  2. Initialize your spider or fetcher instance in your code.
  3. Define your scraping logic.

Run these in a terminal:

# Copy the project to your computer
git clone https://github.com/D4Vinci/Scrapling.git
# Go into its folder
cd Scrapling

How to use it

You can use the fetcher class for HTTP requests or create spiders to manage concurrent crawls. The library includes templates for sitemap crawling, link following, and data extraction.

For newcomers and experts

If you are new to scraping, you can start by fetching a single page and printing its text. Experienced users can build spiders with concurrent crawling, automatic retries, and pause or resume functionality.

Good to know

  • Supports asynchronous execution for concurrency.
  • Handles browser TLS fingerprinting and HTTP/3.
  • Includes automatic retries and blocked request detection.
  • Exports data to JSON, JSONL, CSV, and XML formats.
  • Respects robots.txt directives.

README and files

README

Scrapling

This Python web scraping framework combines fast HTTP requests, browser automation, and data parsing into one library. It includes an MCP server for AI agents to collect data, render JavaScript, and manage concurrent crawling tasks.

Made by D4Vinci on GitHub. These notes are Minda's, written from its README; the project's own files and license are in its repository.

About

What it is: Scrapling is a Python web scraping framework. It combines fast HTTP requests, stealth browsers, parsing tools, and concurrent crawling into a single library. You can scrape static pages, render JavaScript, interact with sites, and extract data from backend API calls.

Who it suits: This framework helps developers who need to scrape data or build web-connected AI agents.

Where to use it:

  • Python scripts
  • Terminal environments
  • AI agents

How to set it up:

  1. Install the package.
  2. Initialize your spider or fetcher instance in your code.
  3. Define your scraping logic.

How to use it: You can use the fetcher class for HTTP requests or create spiders to manage concurrent crawls. The library includes templates for sitemap crawling, link following, and data extraction.

For newcomers and experts: If you are new to scraping, you can start by fetching a single page and printing its text. Experienced users can build spiders with concurrent crawling, automatic retries, and pause or resume functionality.

Good to know:

  • Supports asynchronous execution for concurrency.
  • Handles browser TLS fingerprinting and HTTP/3.
  • Includes automatic retries and blocked request detection.
  • Exports data to JSON, JSONL, CSV, and XML formats.
  • Respects robots.txt directives.

On GitHub, 4 November 2026

  • Stars: 94,938 (1,658 that day)
  • Forks: 9,956
  • Watching: 333
  • Open issues and pull requests: 6
  • Language: Python
  • License: BSD-3-Clause
  • Topics: ai, ai-scraping, automation, crawler, crawling, crawling-python, data, data-extraction, mcp, mcp-server, playwright, python
  • Trending: #2 repository of the week on GitHub trending (4 November 2026)

Links

Intro videos

Get the code

git clone https://github.com/D4Vinci/Scrapling.git

Then follow the setup in its README. Nothing in this kit runs by itself.

Files
README.md  3 KB

1 file, 3 KB, downloaded as a zip.

What’s new

  • Version 1 · 2026-11-04

Compare with similar

Listed GitHub projects in the same category that share the most topics with this one.

ProductStarsActivityLanguageLicenseLast change
Scrapling This page99,911ActivePythonBSD-3-Clause
Crawl4AIShares crawler · data-extraction · playwright87,016——Apache-2.0—
闲鱼智能监控系统Shares playwright · automation · ai18,425QuietPythonMIT
Chrome DevTools for agentsShares mcp-server · mcp55,482ActiveTypeScriptApache-2.0
agent-desktopShares automation · mcp19,705ActiveRustApache-2.0
BLA: Blind LLM AgentsShares mcp-server · mcp2,486ActiveTypeScriptMIT
Numbers as Mindket last read them, from 2026-11-06 to 2026-11-14. Activity is read from each project’s last change on GitHub: Active within 30 days, Quiet within 180, Stale after that.

Reviews

No ratings yet

Write a review

No reviews yet.

What people say

Posts and comments that name this project on Hacker News, Bluesky, X, Reddit, Facebook and YouTube, found by a weekly search and on Trendshift. Mindket doesn’t write or check them. Open one to read it where it was posted.

d4vinciScrapling: Fast, Adaptive Web Scraping for Python4 points · 1 replymidudev · Miguel Ángel DuránSi quieres hacer scraping y extracción de datos de webs, usa este framework open source. ✓ Peticiones que imita la huella de Chrome ✓ Navegador con JS y Captcha de Cloudflare ✓ MCP y Agent Skill para usar con IA → github.com/D4Vinci/Scra...13 likesnsokinShow HN: Replen – maps your repos to a knowledge graph to match open-sourceHi all. I built Replen as a way to improve my AI dev workflows. In essence, it automates what was a manual process for me and turns it from a reactive stance to a proactive one. That being reading/ scrolling lists and creators that showcased cool and trending repos. If i saw something I liked i’d…1 pointデイリーGitHubトレンド今日のGitHubトレンド D4Vinci/Scrapling Scraplingは、高速なHTTPリクエスト、ステルスブラウザ、強力なパース機能、並行クローリングを統合したPython用ウェブスクレイピングフレームワークです。 静的ページの抽出やJavaScriptのレンダリング、APIからのデータ収集に対応し、プロキシ回転や自動再試行などの機能を提供します。 Python、CLI、AIエージェントから利用可能で、単一ページからサイト全体まで効率的にデータを収集し、Markdownなどで書き出すことを目的としています。1 like · 1 replyd4vinciScrapling v0.2.99 – Easy, effortless Web Scraping With Python as it should be1 pointEdy | Indie BuilderToday's GitHub trending: AI doing the grunt work. - D4Vinci/Scrapling: Python scraping, one request to a full crawl (+1,658 today) - max-sixty/worktrunk: Git worktree CLI for running several AI coding agents (+438) - JustVugg/colibri: big MoE models on your own PC (+802) #buildinpublic #AI2 likesnullbioLaunch HN: Intuned (YC S22) – Build and run reliable browser automations as codeWhat is the advantage of your product over having Codex generate a script using something like https://github.com/D4Vinci/Scrapling?GitHub Trending📈 Today's GitHub Trending: 1. https://github.com/vercel-labs/agent-desktop - Rust, 2243 stars today 2. https://github.com/thedotmack/claude-mem - TypeScript, 1799 stars today 3. https://github.com/D4Vinci/Scrapling - Python, 858 stars today 4. https://github.com/hyprwm/Hyprland - C++, 508 stars […]1 like · 1 replyllamavoreAgentic Coding Slot Machines – Did We Just Summon a Genie Addiction? – Part 1Check the post from Simon Willison: https://til.simonwillison.net/claude-code/playwright-mcp-cla... You install the playwright mcp server and then it just launches chrome. Also check out https://browser-use.com/ https://github.com/browser-use/browser-use and https://github.com/D4Vinci/ScraplingGitHub Trending🎉 Celebrating! 🎉 (500+ new stars) 📦 D4Vinci / Scrapling ⭐ 92,148 (+892) 🗒 Python 🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: ...1 like · 1 replyGithub Repositorieshttps://github.com/D4Vinci/ScraplingGitTrending_BlueSky🔥 Scrapling: The Python Framework Transforming Web Scraping 📦 D4Vinci/Scrapling Scrapling is gaining traction as a powerful and flexible web scraping framework that simp... #github #opensource #Python 👉 https://www.gittrending.com/article/scrapling-the-python-framework-transforming-web-scraping

Searched

More from Mindket

See all from Mindket

Bonsai-27B-GGUF

lmstudio-community · Kit

A GGUF conversion of the 27B Bonsai model for LM Studio, with a 3.80 GB Q1_0 language file and a 0.93 GB BF16 vision projector. Tested for text, vision and par…

New

Runs DeepSeek-V4.1-Flash-1017 as a 2.9 bits-per-weight EXL3 quantization, a 196 GiB copy for people serving large models on their own hardware. Perplexity stay…

New

MusicGen Small

facebook · Kit

Generates short music clips from a text description or an audio prompt, in the small checkpoint of facebook's MusicGen. It suits developers and music hobbyists…

New

Similar projects

See all in Developer

Crawl4AI

unclecode · Kit

UncleCode’s open source web crawler and scraper for language models and AI agents. It loads a page in a browser and returns clean Markdown, or JSON in a shape…

New87k

TrendRadar

sansan0 · Kit

An open source news and public opinion monitor. It reads Chinese hot-search lists and your RSS feeds, keeps what matches your keywords or an AI score, and push…

New63.5k

闲鱼智能监控系统

Usagi-org · Kit

Monitors Xianyu product listings using Playwright and AI analysis to notify users of items matching specific criteria. It includes a web dashboard for task man…

New18.4k