360 Secure SkillHub
360 安全技能中心

Crawl By Desearch

这是Desearch提供的网页爬取工具，核心功能是抓取指定URL的网页内容，默认返回干净纯文本，也可指定返回原始HTML。典型场景包括读取技术文档、获取网页源码做分析等。需先获取API密钥并配置环境变量，通过命令行调用，支持格式参数切换，密钥无效或余额不足会返回对应错误。

okradze
okradze
热度 86.65
最近更新 2026-05-11 17:26:14
分类：搜索依赖：第三方 API应用场景：研究

安装技能

复制下面的话术发给你的 Agent，它会通过 secure-skills 检索并安装该技能。

执行命令安装此 skill： npx secure-skills-cli add desearch-crawl
Metadata
name
desearch-crawl
description
Crawl/scrape and extract content from any webpage URL. Returns the page content as clean text or raw HTML. Use this when you need to read the full contents of a specific web page.
metadata
{"clawdbot":{"emoji":"🕷️","homepage":"https://desearch.ai","requires":{"env":["DESEARCH_API_KEY"]}}}

Crawl Webpage By Desearch

Extract content from any webpage URL. Returns clean text or raw HTML.

Quick Start

  1. Get an API key from https://console.desearch.ai
  2. Set environment variable: export DESEARCH_API_KEY='your-key-here'

Usage

# Crawl a webpage (returns clean text by default)
scripts/desearch.py crawl "https://en.wikipedia.org/wiki/Artificial_intelligence"

# Get raw HTML
scripts/desearch.py crawl "https://example.com" --crawl-format html

Options

Option Description
--crawl-format Output content format: text (default) or html

Examples

Read a documentation page

scripts/desearch.py crawl "https://docs.python.org/3/tutorial/index.html"

Get raw HTML for analysis

scripts/desearch.py crawl "https://example.com/page" --crawl-format html

Response

Example (format=text, truncated, default)

Artificial intelligence (AI) is the capability of computational systems to perform tasks that typically require human intelligence, such as learning, reasoning, problem-solving, perception, and decision-making...

Example (format=html, truncated)

<!DOCTYPE html>
<html>
  <head><title>Artificial intelligence - Wikipedia</title></head>
  <body>
    <p>Artificial intelligence (AI) is the capability of computational systems...</p>
  </body>
</html>

Notes

  • Response is plain text or raw HTML — not JSON.
  • Default format is text. Use --crawl-format html only when you need to inspect page structure.
  • Prefer text format to avoid bloating the agent context with markup.

Errors

Status 401, Unauthorized (e.g., missing/invalid API key)

{
  "detail": "Invalid or missing API key"
}

Status 402, Payment Required (e.g., balance depleted)

{
  "detail": "Insufficient balance, please add funds to your account to continue using the service."
}

Resources