Crawl4AI Scraper Kit
LLM-ready web data, on your machine.
Agents, RAG pipelines, and research need web data. The easy path is a scraping SaaS (per-page credit) or a hosted crawler (per-token spend) - both bill per request and see your target URLs. This kit runs Crawl4AI, the open-source, LLM-friendly crawler, as a local token-gated API. docker compose up and you own it.
The kit
- 📄 The Guide (20-page PDF) — architecture, 5-step launch, the 3 scripts, extraction schema patterns, rate-limit/retry, troubleshooting
- ⚙️ docker-compose.yml — the official unclecode/crawl4ai image, token-gated, loopback-bound
- ⚙️ scrape_to_markdown.py — one URL to clean Markdown (fit / raw / bm25)
- ⚙️ batch_crawl.py — a URL list to per-page Markdown + JSON
- ⚙️ extraction_schema_example.py — a CSS schema to clean JSON rows
- ⚙️ .env.example — token, port, host — fill yours
- 📊 XLSX (2 tabs) — file index + config reference
- 🖨️ Cheatsheet (2-page PDF) — 1-page launch ref
Why local instead of a scraping SaaS
- No per-page credits — crawl until it works
- No rate-limit wall tied to a paid plan
- No target URLs leaving your machine
- Markdown output is ready input for your RAG / agents
Hardware needs (honest)
- 4GB RAM — works, single URLs, light pages
- 8GB RAM — comfortable (recommended)
- 16GB+ RAM — big batches, heavy JS, parallel jobs
- GPU — NOT required. This is CPU-bound. A GPU only helps if you add your own local LLM downstream (not included)
Private by design
The API binds to 127.0.0.1 and every route except /health needs your Bearer token. Pages you fetch go from the container to the target site and back - never to a third party. The same container Operator Co runs for its research pipeline.
Who this is for
Freelancers/agencies/developers who want LLM-ready web data for RAG, agents, or research without paying a scraping SaaS or burning API credits on a hosted crawler. You'll run Docker (free).
Who it's NOT for
People who want a hosted scraping SaaS or a no-code page. This is a local API plus scripts - Docker required.
Why trust this?
Every script in this kit ran against a live Crawl4AI container before packaging - same version (unclecode/crawl4ai:0.9.2) and the same token-gated setup this kit ships.