Tavily Crawl

核心功能为批量爬取网站多页内容，支持通过深度、数量限制、路径筛选精准控制爬取范围，还可按语义指令提取内容分块。典型场景包括下载文档/知识库用于离线访问，或为大模型提取上下文。需配置Tavily API key，通过命令行调用脚本，输入目标网址及可选参数，输出本地Markdown文件或JSON数据，适配离线分析与AI喂料需求。

matthew77

热度 73.73

最近更新 2026-05-17 13:06:26

分类：自动化依赖：代码执行 / 第三方 API应用场景：自动化脚本 / 文件转换 / 研究

安装技能

复制下面的话术发给你的 Agent，它会通过 secure-skills 检索并安装该技能。

执行命令安装此 skill： npx secure-skills-cli add liang-tavily-crawl

Metadata

name: tavily-crawl
description: Crawl any website and save pages as local markdown files. Ideal for downloading documentation, knowledge bases, or web content for offline access or analysis.
homepage: https://tavily.com
metadata: {"openclaw":{"emoji":"🕷️","requires":{"bins":["node"],"env":["TAVILY_API_KEY"]},"primaryEnv":"TAVILY_API_KEY"}}

Tavily Crawl

Crawl websites to extract content from multiple pages. Ideal for documentation, knowledge bases, and site-wide content extraction.

Authentication

Get your API key at https://tavily.com and add to your OpenClaw config:

{
  "skills": {
    "entries": {
      "tavily-crawl": {
        "enabled": true,
        "apiKey": "tvly-YOUR_API_KEY_HERE"
      }
    }
  }
}

Or set in environment variable:

export TAVILY_API_KEY="tvly-YOUR_API_KEY_HERE"

Quick Start

Using the Script

node {baseDir}/scripts/crawl.mjs "https://docs.example.com"
node {baseDir}/scripts/crawl.mjs "https://docs.example.com" --output ./docs
node {baseDir}/scripts/crawl.mjs "https://example.com" --depth 2 --limit 50

Examples

# Basic crawl
node {baseDir}/scripts/crawl.mjs "https://docs.example.com"

# Deeper crawl with limits
node {baseDir}/scripts/crawl.mjs "https://docs.example.com" --depth 2 --limit 50

# Save to files
node {baseDir}/scripts/crawl.mjs "https://docs.example.com" --depth 2 --output ./docs

# Focused crawl with path filters
node {baseDir}/scripts/crawl.mjs "https://example.com" --depth 2 \
  --select "/docs/.*" --exclude "/blog/.*"

# With semantic instructions
node {baseDir}/scripts/crawl.mjs "https://docs.example.com" \
  --instructions "Find API documentation" --chunks 3

Options

Option	Description	Default
`--depth <n>`	Crawl depth (1-5)	1
`--breadth <n>`	Links per page	20
`--limit <n>`	Total pages cap	50
`--output <dir>`	Save pages to directory	-
`--instructions <text>`	Natural language guidance	-
`--chunks <n>`	Chunks per page (1-5, requires instructions)	-
`--depth-mode <mode>`	Extract depth: `basic` or `advanced`	`basic`
`--select <pattern>`	Regex pattern to include	-
`--exclude <pattern>`	Regex pattern to exclude	-
`--timeout <sec>`	Max wait time (10-150 seconds)	150
`--json`	Output raw JSON	false

Depth vs Performance

Depth	Typical Pages	Time
1	10-50	Seconds
2	50-500	Minutes
3	500-5000	Many minutes

Start with --depth 1 and increase only if needed.

Crawl for Context vs Data Collection

For agentic use (feeding results into context): Always use --instructions + --chunks. This returns only relevant chunks instead of full pages, preventing context window explosion.

For data collection (saving to files): Omit --chunks to get full page content.

Tips

Always use --chunks for agentic workflows - prevents context explosion when feeding results to LLMs
Omit --chunks only for data collection - when saving full pages to files
Start conservative (--depth 1, --limit 20) and scale up
Use path patterns to focus on relevant sections
Always set a --limit to prevent runaway crawls