Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8m
# 🚀🤖 Crawl4AI: the open-source web crawler for LLMs and AI agents
Crawl4AI turns any website into clean, LLM-ready Markdown for RAG, AI agents and data pipelines. Run the open-source web crawler and scraper yourself, free forever, or use it hosted with one key: `/scrape` a page to Markdown, `/extract` it to JSON, `/search` the web, `/serp` for Google's results page as data, `/answer` a question with sources; MCP for your agent. Free: 1,000 credits every month, 10 a minute, 2 at a time. Paid: 120 a minute, 10 at a time, from $10 a month.
## Two ways to use Crawl4AI
### 🐍 Run it yourself: open source, forever
```bash
pip install -U crawl4ai
crawl4ai-setup # installs the browser, once
```
```python
import asyncio
from crawl4ai import AsyncWebCrawler
async def main():
async with AsyncWebCrawler() as crawler:
result = await crawler.arun(url="https://news.ycombinator.com")
print(result.markdown)
asyncio.run(main())
```
Docker server, CLI and every option: [Installation](#installation) · [docs.crawl4ai.com](https://docs.crawl4ai.com)
### ☁️ Or use the cloud: no browsers, no proxies
1. [](https://crawl4ai.com/?ref=readme)
Verify your email and 1,000 free credits are yours every month. No card.
2. Get any page as Markdown:
```bash
curl -s https://api.crawl4ai.com/scrape \
-H "Authorization: Bearer $CRAWL4AI_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://news.ycombinator.com"}' | jq -r .markdown
```
The same key works for `/extract`, `/search`, `/serp`, `/answer` and many URLs at once (`/scrape/batch`, `/scrape/jobs`). Pay as you go: [live prices](https://crawl4ai.com/docs?ref=readme#pricing).
3. Give it to your AI agent. Claude Code shown; [Codex, Cursor and OpenCode →](https://crawl4ai.com/docs?ref=readme#mcp)
```bash
claude mcp add --transport http crawl4ai https://api.crawl4ai.com/mcp \
--header "Authorization: Bearer $CRAWL4AI_KEY"
```
### Which one?
| | 🐍 Library | 🐳 Your own server | ☁️ Crawl4AI Cloud |
|---|---|---|---|
| **Runs the browsers** | you, in your Python process | you, in Docker on your machine | we do |
| **JS-heavy pages and bot walls** | your settings, your proxies | your settings, your proxies | handled for you, automatically |
| **Web search** | – | – | `/search`, `/serp` and `/answer` |
| **Price** | free, forever | free (your hosting) | 1,000 free credits a month; plans from $10 |
🤓 My Personal Story
I grew up on an Amstrad, thanks to my dad, and never stopped building. In grad school I specialized in NLP and built crawlers for research. That’s where I learned how much extraction matters.
In 2023, I needed web-to-Markdown. The “open source” option wanted an account, API token, and $16, and still under-delivered. I went turbo anger mode, built Crawl4AI in days, and it went viral. Now it’s the most-starred crawler on GitHub.
I made it open source for **availability**, anyone can use it without a gate. Now I’m building the platform for **affordability**, anyone can run serious crawls without breaking the bank. If that resonates, join in, send feedback, or just crawl something amazing.
That platform is live now: [Crawl4AI Cloud](https://crawl4ai.com/?ref=readme).
Why developers pick Crawl4AI
- **LLM-ready output**: smart Markdown with headings, tables, code and citation hints
- **Fast in practice**: async browser pool, caching, minimal hops
- **Full control**: sessions, proxies, cookies, user scripts, hooks
- **Adaptive intelligence**: learns site patterns, explores only what matters
- **Deploy anywhere**: no keys needed, CLI and Docker, or the hosted cloud
## ✨ Features
📝 Markdown generation
- 🧹 **Clean Markdown**: headings, lists, tables and code blocks, in a structure an LLM reads well.
- 🎯 **Fit Markdown**: filters remove menus, footers and boilerplate: `PruningContentFilterLXML`, `BM25ContentFilter` (for a query) and `LLMContentFilter`.
- 🔗 **Citations**: page links become a numbered reference list.
- 🛠️ **Your own strategy**: plug in a custom Markdown generator.
☁️ Same in the cloud: `POST /scrape` returns this Markdown, with no browser to run. [Docs →](https://crawl4ai.com/docs?ref=readme#scrape)
📊 Structured data extraction
- 🔎 **CSS and XPath schemas**: fast extraction with no LLM (`JsonCssExtractionStrategy`, `JsonXPathExtractionStrategy`, `RegexExtractionStrategy`).
- 🪄 **Schema generator**: describe what you want once; `generate_schema` writes a reusable schema.
- 🤖 **LLM extraction**: any LLM provider, open-source or hosted, into a typed JSON schema (`LLMExtractionStrategy`).
- 🧱 **Chunking**: topic, regex and sentence chunking for long pages.
- 🌌 **Cosine similarity**: find the chunks that match a query (`CosineStrategy`).
☁️ Same in the cloud: `POST /extract`, with no LLM key of your own. [Docs →](https://crawl4ai.com/docs?ref=readme#extract)
🌐 Browser control
- 🖥️ **Your own browser**: persistent profiles with saved logins, cookies and settings.
- 🔄 **Remote browsers**: connect over the Chrome DevTools Protocol (CDP).
- 🔒 **Sessions**: keep a browser state across multi-step crawls.
- 🧩 **Proxies**: with authentication and rotation.
- 🕶️ **Stealth mode**: `enable_stealth`, and an undetected-browser adapter for sites that detect automation.
- ⚙️ **Full control**: headers, cookies, user agents, viewport.
- 🌍 **Chromium, Firefox and WebKit**.
🔎 Crawling and scraping
- 🕸️ **Deep crawl**: BFS, DFS and best-first strategies, with crash recovery (`resume_state`) for long crawls.
- 🧠 **Adaptive crawling**: `AdaptiveCrawler` stops when it has learned enough to answer your query.
- 🌱 **URL discovery**: `AsyncUrlSeeder` (sitemaps, Common Crawl) and `DomainMapper`; `prefetch=True` finds URLs 5 to 10 times faster.
- 🚀 **Dynamic pages**: run JavaScript, wait for elements, scroll the full page (`scan_full_page`) for infinite scroll and lazy images.
- 📸 **Screenshots and PDFs** of any page.
- 🖼️ **Media and links**: images, audio, video, `srcset`, internal and external links, iframes, metadata.
- 📂 **Raw HTML and local files**: `raw:` and `file://`.
- 🛠️ **Hooks** at every step of a crawl.
- 💾 **Caching** to skip repeated fetches.
- ⚡ **Many URLs at once**: `arun_many` with a memory-adaptive dispatcher.
☁️ Same in the cloud: up to 50 URLs in one streamed call, or 10,000 in a background job. [Docs →](https://crawl4ai.com/docs?ref=readme#batch)
🐳 Self-hosting (Docker)
- 🔐 **Secure by default**: every endpoint needs your `CRAWL4AI_API_TOKEN`.
- 🧰 **REST API**: `/md`, `/html`, `/crawl`, `/crawl/stream`, `/screenshot`, `/pdf`, `/execute_js`.
- 🤖 **MCP**: connect Claude Code and other agents to your own server.
- 📊 **Monitoring dashboard and playground**, a browser pool with pre-warmed pages.
- 🏗️ **AMD64 and ARM64** images.
☁️ Rather not run a server? The cloud is the same idea, hosted. [Get a key →](https://crawl4ai.com/?ref=readme)
☁️ What the cloud adds
- 🔍 **Web search API**: `GET /search`, browser-free, ranked and cleaned. [Docs →](https://crawl4ai.com/docs?ref=readme#search)
- 📊 **SERP API**: `GET /serp`, Google's results page as data: true ranks, every block, localised to a place. Beta. [Docs →](https://crawl4ai.com/docs?ref=readme#serp)
- 💬 **Answers**: `GET /answer` gives a direct answer to a question (experimental). [Docs →](https://crawl4ai.com/docs?ref=readme#answer)
- 🧪 **Extraction without your own LLM key**: `POST /extract`. [Docs →](https://crawl4ai.com/docs?ref=readme#extract)
- 🧗 **JS-heavy pages and bot walls**: handled automatically; you never pick an engine. [Docs →](https://crawl4ai.com/docs?ref=readme#scrape)
- 🤝 **MCP for your agent**: one line in Claude Code, Codex, Cursor or OpenCode. [Docs →](https://crawl4ai.com/docs?ref=readme#mcp)
## 🛠️ Installation
🐍 pip
```bash
pip install -U crawl4ai
crawl4ai-setup # installs and sets up the browser
crawl4ai-doctor # checks the installation
```
If the browser setup fails, install it by hand:
```bash
python -m playwright install --with-deps chromium
```
Pre-release versions: `pip install crawl4ai --pre`
**Development install**, for contributors:
```bash
git clone https://github.com/unclecode/crawl4ai.git
cd crawl4ai
pip install -e ".[all]" # or: pip install -e . (the core only)
```
🐳 Docker server
The server needs a token. Without one it answers only inside its container.
```bash
export CRAWL4AI_API_TOKEN="$(openssl rand -hex 32)"
docker run -d -p 11235:11235 --name crawl4ai --shm-size=1g \
-e CRAWL4AI_API_TOKEN="$CRAWL4AI_API_TOKEN" \
unclecode/crawl4ai:latest
```
Test it (allow about 10 seconds for the start):
```bash
curl -s http://localhost:11235/md \
-H "Authorization: Bearer $CRAWL4AI_API_TOKEN" \
-H "Content-Type: application/json" \
-d '{"url": "https://news.ycombinator.com"}' | jq -r .markdown
```
The dashboard is at `http://localhost:11235/dashboard`, the playground at `http://localhost:11235/playground`. LLM keys, MCP and every setting: [Self-hosting guide](https://docs.crawl4ai.com/core/self-hosting/).
⌨️ Command line (`crwl`)
```bash
# A page as Markdown
crwl https://news.ycombinator.com -o markdown
# Deep crawl, breadth first, at most 10 pages
crwl https://docs.crawl4ai.com --deep-crawl bfs --max-pages 10
# Ask a question about a page (needs an LLM key: crwl config)
crwl https://www.example.com/products -q "Extract all product prices"
```
## 🔬 Advanced usage examples
More in [docs/examples](https://github.com/unclecode/crawl4ai/tree/main/docs/examples).
📝 Clean and fit Markdown
```python
import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
from crawl4ai.content_filter_strategy import PruningContentFilterLXML
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator
async def main():
run_config = CrawlerRunConfig(
cache_mode=CacheMode.BYPASS,
markdown_generator=DefaultMarkdownGenerator(
content_filter=PruningContentFilterLXML(threshold=0.48, threshold_type="fixed", min_word_threshold=0)
),
)
async with AsyncWebCrawler(config=BrowserConfig(headless=True)) as crawler:
result = await crawler.arun(url="https://en.wikipedia.org/wiki/Web_crawler", config=run_config)
print(len(result.markdown.raw_markdown), "characters of raw Markdown")
print(len(result.markdown.fit_markdown), "characters after the filter")
asyncio.run(main())
```
🖥️ A JavaScript page and structured data, without an LLM
```python
import asyncio, json
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode, JsonCssExtractionStrategy
schema = {
"name": "Quotes",
"baseSelector": "div.quote",
"fields": [
{"name": "text", "selector": "span.text", "type": "text"},
{"name": "author", "selector": "small.author", "type": "text"},
{"name": "tags", "selector": "a.tag", "type": "list", "fields": [{"name": "tag", "type": "text"}]},
],
}
async def main():
run_config = CrawlerRunConfig(
extraction_strategy=JsonCssExtractionStrategy(schema),
scan_full_page=True, # scroll to the end, so the page loads every quote
scroll_delay=0.5,
cache_mode=CacheMode.BYPASS,
)
async with AsyncWebCrawler(config=BrowserConfig(headless=True)) as crawler:
result = await crawler.arun(url="https://quotes.toscrape.com/scroll", config=run_config)
quotes = json.loads(result.extracted_content)
print(f"Extracted {len(quotes)} quotes")
print(json.dumps(quotes[0], indent=2))
asyncio.run(main())
```
📚 Structured data with an LLM
```python
import os, asyncio
from pydantic import BaseModel, Field
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode, LLMConfig, LLMExtractionStrategy
class ModelFee(BaseModel):
model_name: str = Field(..., description="Name of the model.")
input_fee: str = Field(..., description="Fee for input tokens.")
output_fee: str = Field(..., description="Fee for output tokens.")
async def main():
run_config = CrawlerRunConfig(
cache_mode=CacheMode.BYPASS,
extraction_strategy=LLMExtractionStrategy(
# any provider LiteLLM supports, e.g. "ollama/llama3.3" with api_token="no-token"
llm_config=LLMConfig(provider="openai/gpt-4o-mini", api_token=os.getenv("OPENAI_API_KEY")),
schema=ModelFee.model_json_schema(),
extraction_type="schema",
instruction="Extract every model name with its input and output token fee.",
),
)
async with AsyncWebCrawler() as crawler:
result = await crawler.arun(url="https://openai.com/api/pricing/", config=run_config)
print(result.extracted_content)
asyncio.run(main())
```
🤖 Your own browser with a saved profile
```python
import os, asyncio
from pathlib import Path
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
async def main():
user_data_dir = os.path.join(Path.home(), ".crawl4ai", "browser_profile")
os.makedirs(user_data_dir, exist_ok=True)
browser_config = BrowserConfig(headless=True, user_data_dir=user_data_dir, use_persistent_context=True)
run_config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS, magic=True)
async with AsyncWebCrawler(config=browser_config) as crawler:
result = await crawler.arun(url="ADDRESS_OF_A_CHALLENGING_WEBSITE", config=run_config)
print(result.success, len(result.markdown))
asyncio.run(main())
```
## 📖 Documentation
- Library docs, guides and API reference: [docs.crawl4ai.com](https://docs.crawl4ai.com/)
- Cloud docs: [crawl4ai.com/docs](https://crawl4ai.com/docs?ref=readme)
- Release notes: [releases](https://github.com/unclecode/crawl4ai/releases) · Roadmap: [ROADMAP.md](https://github.com/unclecode/crawl4ai/blob/main/ROADMAP.md)
## 🤝 Contributing
We welcome contributions from the open-source community. Check out our [contribution guidelines](https://github.com/unclecode/crawl4ai/blob/main/CONTRIBUTORS.md) for more information.
## 📄 License & Attribution
This project is licensed under the Apache License 2.0, attribution is recommended via the badges below. See the [Apache 2.0 License](https://github.com/unclecode/crawl4ai/blob/main/LICENSE) file for details.
### Attribution Requirements
When using Crawl4AI, you must include one of the following attribution methods:
📈 1. Badge Attribution (Recommended)
Add one of these badges to your README, documentation, or website:
| Theme | Badge |
|-------|-------|
| **Disco Theme (Animated)** | |
| **Night Theme (Dark with Neon)** | |
| **Dark Theme (Classic)** | |
| **Light Theme (Classic)** | |
HTML code for adding the badges:
```html
|
| **Night Theme (Dark with Neon)** | 
|
| **Dark Theme (Classic)** | 
|
| **Light Theme (Classic)** | 
|
HTML code for adding the badges:
```html
