Firecrawl Local Skill
Self-hosted Firecrawl integration using the v1 REST API. Tests connectivity first, executes scrape/crawl/map, handles async crawl polling automatically.
Setup (one-time)
mkdir -p ~/.openclaw/skills/firecrawl-local
cp run.sh ~/.openclaw/skills/firecrawl-local/run.sh
chmod +x ~/.openclaw/skills/firecrawl-local/run.sh
The script lives at scripts/run.sh in this skill folder — copy it into place as above.
Prerequisites: curl, jq installed. Firecrawl running at localhost:3002.
Optional env vars:
export FIRECRAWL_LOCAL_URL="http://localhost:3002" # default
export FIRECRAWL_API_KEY="fc-your-key" # only needed if auth enabled
Commands
Default — scrape a single page (URL only, no subcommand needed)
firecrawl-local https://docs.example.com/api
Scrape — explicit, with format options
firecrawl-local scrape https://docs.example.com/api
firecrawl-local scrape https://docs.example.com/api --formats markdown,html
Map — discover all URLs on a site
firecrawl-local map https://docs.example.com
firecrawl-local map https://docs.example.com --limit 200
Crawl — bulk extract multiple pages (async, auto-polled)
firecrawl-local crawl https://docs.example.com
firecrawl-local crawl https://docs.example.com --limit 30 --max-depth 2
firecrawl-local crawl https://docs.example.com --include /docs --exclude /blog
Agent Instructions
When to use each command
| Goal | Command |
|---|---|
| Get content from one URL (quickest) | firecrawl-local <url> |
| Discover what pages exist | map |
| Get content from one URL with format control | scrape |
| Ingest an entire docs site | crawl |
| RAG pipeline ingestion | map → targeted scrape or crawl |
Optimal workflows
Documentation RAG pipeline:
1. map https://docs.example.com → get full URL list
2. scrape <specific key pages> → targeted extraction
3. Pass markdown to embedding pipeline
Full site ingestion:
1. crawl https://docs.example.com --limit 50 --max-depth 3
2. Results auto-polled and returned as JSON array of {url, markdown}
Parameters
| Flag | Applies to | Description |
|---|---|---|
--limit N |
map, crawl | Max pages (default: 50 for crawl, 500 for map) |
--max-depth N |
crawl | How deep to follow links (default: 2) |
--include /path |
crawl | Only crawl URLs matching this path prefix |
--exclude /path |
crawl | Skip URLs matching this path prefix |
--formats list |
scrape | Comma-separated: markdown, html, rawHtml, links |
Reading the output
- scrape: Returns
{success, data: {markdown, html, metadata}} - map: Returns
{success, links: [...]} - crawl: Returns
{success, data: [{url, markdown, metadata}, ...]}← after polling completes
Failure signals and fixes
| Error | Cause | Fix |
|---|---|---|
Local Firecrawl unavailable |
Service not running | Start Firecrawl, check port 3002 |
success: false |
Bad URL or blocked | Check URL is reachable, try --formats html |
Empty markdown field |
JS-rendered page | Firecrawl handles most JS — check if site blocks bots |
| Crawl times out | Site is large | Reduce --limit or --max-depth |
Script reference
See scripts/run.sh for the full implementation. Key design decisions:
- Health check uses
/healthendpoint with 3s timeout - Auth header only sent when
FIRECRAWL_API_KEYis set - Crawl polling retries every 5s up to 60 attempts (5 minutes)
- All parameters are passed via
jqto prevent shell injection in JSON