Tavily Crawl

通过 Node 脚本调用 Tavily 爬取网站，支持深度、页数上限、路径正则过滤、自然语言指令等参数，可输出完整页面或分块内容，并保存为本地 Markdown 文件；需配置 TAVILY_API_KEY。

matthew77

热度 73.77

最近更新 2026-05-17 13:06:26

分类：搜索依赖：代码执行 / 第三方 API / 读写文件应用场景：研究 / 数据处理 / 工作流

安装技能

复制下面的话术发给你的 Agent，它会通过 secure-skills 检索并安装该技能。

执行命令安装此 skill： npx secure-skills-cli add liang-tavily-crawl

Metadata

name: tavily-crawl
description: Crawl any website and save pages as local markdown files. Ideal for downloading documentation, knowledge bases, or web content for offline access or analysis.
homepage: https://tavily.com
metadata: {"openclaw":{"emoji":"🕷️","requires":{"bins":["node"],"env":["TAVILY_API_KEY"]},"primaryEnv":"TAVILY_API_KEY"}}

Tavily Crawl

Crawl websites to extract content from multiple pages. Ideal for documentation, knowledge bases, and site-wide content extraction.

Authentication

Get your API key at https://tavily.com and add to your OpenClaw config:

{
  "skills": {
    "entries": {
      "tavily-crawl": {
        "enabled": true,
        "apiKey": "tvly-YOUR_API_KEY_HERE"
      }
    }
  }
}

Or set in environment variable:

export TAVILY_API_KEY="tvly-YOUR_API_KEY_HERE"

Quick Start

Using the Script

node {baseDir}/scripts/crawl.mjs "https://docs.example.com"
node {baseDir}/scripts/crawl.mjs "https://docs.example.com" --output ./docs
node {baseDir}/scripts/crawl.mjs "https://example.com" --depth 2 --limit 50

Examples

# Basic crawl
node {baseDir}/scripts/crawl.mjs "https://docs.example.com"

# Deeper crawl with limits
node {baseDir}/scripts/crawl.mjs "https://docs.example.com" --depth 2 --limit 50

# Save to files
node {baseDir}/scripts/crawl.mjs "https://docs.example.com" --depth 2 --output ./docs

# Focused crawl with path filters
node {baseDir}/scripts/crawl.mjs "https://example.com" --depth 2 \
  --select "/docs/.*" --exclude "/blog/.*"

# With semantic instructions
node {baseDir}/scripts/crawl.mjs "https://docs.example.com" \
  --instructions "Find API documentation" --chunks 3

Options

Option	Description	Default
`--depth <n>`	Crawl depth (1-5)	1
`--breadth <n>`	Links per page	20
`--limit <n>`	Total pages cap	50
`--output <dir>`	Save pages to directory	-
`--instructions <text>`	Natural language guidance	-
`--chunks <n>`	Chunks per page (1-5, requires instructions)	-
`--depth-mode <mode>`	Extract depth: `basic` or `advanced`	`basic`
`--select <pattern>`	Regex pattern to include	-
`--exclude <pattern>`	Regex pattern to exclude	-
`--timeout <sec>`	Max wait time (10-150 seconds)	150
`--json`	Output raw JSON	false

Depth vs Performance

Depth	Typical Pages	Time
1	10-50	Seconds
2	50-500	Minutes
3	500-5000	Many minutes

Start with --depth 1 and increase only if needed.

Crawl for Context vs Data Collection

For agentic use (feeding results into context): Always use --instructions + --chunks. This returns only relevant chunks instead of full pages, preventing context window explosion.

For data collection (saving to files): Omit --chunks to get full page content.

Tips

Always use --chunks for agentic workflows - prevents context explosion when feeding results to LLMs
Omit --chunks only for data collection - when saving full pages to files
Start conservative (--depth 1, --limit 20) and scale up
Use path patterns to focus on relevant sections
Always set a --limit to prevent runaway crawls