Crawl4AI

What it is

Crawl4AI is an open source web crawler and scraper for language models and AI agents. You give it an address and it loads the page in a headless browser. It hands back clean Markdown, or JSON in a shape you define. Its README says you can run it yourself, free forever. It comes as a Python library, a command called crwl and a Docker server with a REST API. The README also describes Crawl4AI Cloud, a hosted service with its own account and prices. This listing covers the parts you run yourself, and names the cloud only where the maker gives an agent-setup line for it.

1. You give it an address, or raw HTML
2. A headless browser loads the page
3. An optional filter trims menus, footers and boilerplate
4. You get Markdown, or JSON from a schema you define
5. Your RAG pipeline or agent reads the result

Who it suits

Developers who feed web pages to a model. That means RAG pipelines, agents that read the web and data jobs that need a page’s text and tables as clean Markdown. The library and the crwl command need Python 3.10 or newer and a terminal. The Docker server needs Docker instead. The Docker server suits a team that wants one crawler its apps call over HTTP. Skip it if you only want to read one page once, since copying the text by hand is quicker.

Where to use it

Crawl4AI runs where you put it, and the maker’s pages name these routes.

• A Python environment, as a library you import
• A terminal, through the crwl command
• A Docker container on AMD64 or ARM64, on your machine or a server
• Claude Code, through the Docker server’s MCP endpoint
• Crawl4AI Cloud, the hosted service, from Claude Code, Codex, Cursor, Windsurf, OpenCode, OpenClaw, Claude Desktop and ChatGPT over MCP

The README says it drives Chromium, Firefox and WebKit.

How to set it up

Get it saves a zip of Mindket’s setup notes in your downloads folder, with a file for the library, the command line, the Docker server, recipes and each agent. The main route is pip.

1. Install the package from PyPI.
2. Run crawl4ai-setup, which installs the browser once.
3. Run crawl4ai-doctor to check the installation.
4. Save the README’s first script as a file and run it with python.

pip install -U crawl4ai
crawl4ai-setup
crawl4ai-doctor

If the browser setup fails, the README gives a manual line:

python -m playwright install --with-deps chromium

The README’s first script prints a page as Markdown.

import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url="https://news.ycombinator.com")
        print(result.markdown)

asyncio.run(main())

How to use it

You can run Crawl4AI in three ways.

The crwl command picks its output with -o. Deep crawls use the same command.

crwl https://news.ycombinator.com -o markdown
crwl https://docs.crawl4ai.com --deep-crawl bfs --max-pages 10

The second line crawls the maker’s own docs breadth first and stops at 10 pages.

The Docker server needs a token. Without one it answers only inside its container, so your published port resets the connection even though Docker shows the container healthy. Make a token, start the container, and give it about 10 seconds.

export CRAWL4AI_API_TOKEN="$(openssl rand -hex 32)"
docker run -d -p 11235:11235 --name crawl4ai --shm-size=1g \
  -e CRAWL4AI_API_TOKEN="$CRAWL4AI_API_TOKEN" \
  unclecode/crawl4ai:latest

Then send the README’s test request.

curl -s http://localhost:11235/md \
  -H "Authorization: Bearer $CRAWL4AI_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://news.ycombinator.com"}' | jq -r .markdown

In Python, two classes hold the settings. BrowserConfig controls the browser. CrawlerRunConfig controls each crawl: caching, extraction, timeouts and hooks. The Quick Start says the default cache mode bypasses the cache, so you get fresh content.

Example

You want every quote from quotes.toscrape.com/scroll as JSON. That page loads more quotes as you scroll, so the README’s script sets scan_full_page to True and describes one quote block with a CSS schema.

schema = {
    "name": "Quotes",
    "baseSelector": "div.quote",
    "fields": [
        {"name": "text", "selector": "span.text", "type": "text"},
        {"name": "author", "selector": "small.author", "type": "text"},
        {"name": "tags", "selector": "a.tag", "type": "list", "fields": [{"name": "tag", "type": "text"}]},
    ],
}

The script passes JsonCssExtractionStrategy(schema) in the run config, then prints how many quotes it extracted and the first one as JSON. No model is called. The README’s LLM example uses LLMExtractionStrategy with a model name (openai/gpt-4o-mini) and a Pydantic schema on a pricing page, and the page text then goes to that model.

For newcomers and experts

If you are new. A crawler visits web pages, and a scraper reads data out of them. Raw HTML is full of menus, scripts and tags, which a model reads poorly. Markdown keeps the headings, lists, tables and code in plain text. RAG means a model answers from passages you hand it. A schema is a short description of the fields you want from a page, such as title, link and price. Start with the first script on one page, or with crwl and a single address, before you touch any settings.

For experts. The Markdown result carries a raw version and a fit version. The fit version passes through a content filter, and the README names PruningContentFilterLXML, BM25ContentFilter for a query and LLMContentFilter. Extraction has no-LLM strategies (JsonCssExtractionStrategy, JsonXPathExtractionStrategy and RegexExtractionStrategy) and LLMExtractionStrategy, which takes any provider LiteLLM supports.

A one-time call to generate_schema asks an LLM to write a reusable CSS schema, and later runs need no model. Deep crawls come as breadth-first, depth-first and best-first strategies, and the Deep Crawling page marks best-first as the recommended one. resume_state resumes a long crawl after a crash. AdaptiveCrawler stops when it has learned enough to answer your query, and arun_many runs many URLs with a memory-adaptive dispatcher.

The Docker server has been secure by default since 0.9.0. Every endpoint except GET /health needs a bearer token. A screenshot or PDF comes back in the response with an artifact id as well. The stored file downloads from /artifacts/<artifact_id> with your token, and artifacts expire. Hooks stay off until you set CRAWL4AI_HOOKS_ENABLED=true, and they are declarative JSON actions, not Python code.

The server’s MCP endpoints are /mcp/sse over Server-Sent Events and /mcp/ws over WebSocket. Their tools are md, html, screenshot, pdf, execute_js, crawl and ask.

Good to know

Crawl4AI runs on your own machine, and a few details are worth reading first.

• License: Apache-2.0, which GitHub names. The LICENSE file ends with an attribution requirement, so distributions, publications and public uses must carry the maker’s credit line. The README adds that any use must carry one of two credits, a badge or a text line
• Cloud: Crawl4AI Cloud is a separate hosted service with its own account, key and prices. The maker’s pages say your cloud key can call the billing routes, a top-up among them, so decide whether an agent should hold it
• Your data: a crawl connects to the addresses you give it. LLM extraction sends page text to the model you choose, unless that model runs on your machine. The README and the docs pages behind Mindket’s notes say nothing about usage data
• Crawling rules: the Deep Crawling page says to respect robots.txt and marks that as disabled by default. The undetected browser page says to respect each site’s terms of service and to use delays. Check a site’s rules before a large crawl
• Old pages: the installation page is titled 2023 Edition and shows an old Docker tag. The README and the Self-Hosting Guide are newer, so the notes follow them
• MCP and the token: the guide’s MCP line for your own server shows no token and does not say how the endpoints take one. Check the connection after you add it
• Removal: the docs give no uninstall steps. Mindket’s notes add pip uninstall crawl4ai for the package, and the guide gives docker stop crawl4ai and docker rm crawl4ai for the container