Web Crawl

该 Skill 提供网页内容提取的多种模式（text、markdown、links、structured、full），可抓取单个或多个 URL 的内容，并通过检索与抓取组合完成主题研究。实现为 Python 脚本（web_crawl.py、research.py），按材料中的示例以 python3 方式调用。

nowhitestar

热度 83.75

最近更新 2026-05-11 17:33:49

分类：搜索依赖：代码执行应用场景：研究 / 数据处理

安装技能

复制下面的话术发给你的 Agent，它会通过 secure-skills 检索并安装该技能。

执行命令安装此 skill： npx secure-skills-cli add web-crawl

Metadata

name: web-crawl
version: 1.0.0
description: Advanced web crawling and content extraction tool with multiple extraction modes
activation: {"keywords":["crawl","抓取","提取网页","研究","深度研究","research","analyze website"],"tags":["web","research","crawling"]}

Web Crawl Skill

Advanced web content extraction with multiple modes and intelligent content detection.

When to Use

Use this skill when:

User asks to "研究" / "深度研究" a topic
User wants to "抓取" / "提取" content from websites
Need to analyze multiple web pages systematically
Current web_fetch output is insufficient

Extraction Modes

Mode	Use Case
`text`	Clean plain text
`markdown`	Formatted Markdown (recommended)
`links`	Extract all links
`structured`	JSON metadata + content
`full`	Markdown + links combined

Tools Available

web_crawl - Extract content from a single URL
parallel_crawl - Extract from multiple URLs in parallel
research_topic - Multi-step research with search + crawl

Example Usage

User: "研究一下 OpenManus-Max 项目"
→ Use research_topic tool with query="OpenManus-Max GitHub features"