MediaCrawler

MediaCrawler

Automated data collection tool that captures post content and comments from major social media platforms like Xiaohongshu, Douyin, and Bilibili. It supports keyword searches, profile scraping, and media downloads using browser automation.

Made by NanmiCoder on GitHub. These notes are Minda's, written from its README; the project's own files and license are in its repository.

About

What it is: MediaCrawler is a tool for collecting public information from social media platforms. It captures data by using a browser instance to maintain your login status, which avoids the need for complex security reversals.

Who it suits: It is for researchers and students who study social media data.

Where to use it: You can run this tool on your local machine using a Python environment or a web interface.

How to set it up:

1. Clone the project.
2. Install the package manager.
3. Install dependencies using the package manager.
4. Enable remote debugging in your Chrome browser settings.

uv sync
uv run playwright install

How to use it: You run the tool from the command line by selecting the platform and the data type.

uv run main.py --platform xhs --lt qrcode --type search

Example: You can collect post information and comments for a specific search keyword. The tool browses the platform with your session, gathers the data, and saves it to your local machine.

For newcomers and experts: If you are new to this, the tool acts as an automated assistant that mimics your actions on a web page to gather what you see. For those experienced with data collection, the tool uses a browser automation framework to keep your session active, which trades direct API access for easier maintenance as platform security evolves.

Good to know:

• Media download is not available for all platforms.
• The tool interacts with local files, your browser session, and the web.
• You must provide active consent during the initial login phase.

On GitHub, 5 November 2026

• Stars: 70,198
• Forks: 13,307
• Watching: 260
• Open issues and pull requests: 222
• Language: Python
• License: see the repository
• Trending: #2 repository of the day on GitHub trending (5 November 2026)

Links

• Code: https://github.com/NanmiCoder/MediaCrawler
• Website: https://nanmicoder.github.io/MediaCrawler/
• Releases: https://github.com/NanmiCoder/MediaCrawler/releases
• Issues: https://github.com/NanmiCoder/MediaCrawler/issues
• Star history: https://www.star-history.com/#NanmiCoder/MediaCrawler&Date

Get the code

git clone https://github.com/NanmiCoder/MediaCrawler.git

Then follow the setup in its README. Nothing in this kit runs by itself.