360 Secure SkillHub
360 安全技能中心

Web Crawling API

该 Skill 借助 Just Serp API 抓取用户给定的网页，提供三个只读 GET 操作：返回原始 HTML、渲染后 HTML 以及提取正文的干净 Markdown。调用需提供目标 URL 与 JUST_SERP_API_KEY，由 node 脚本发起请求并返回后端响应，适用于内容提取、页面检查与抓取准备。

justserpapi
justserpapi
热度 54.81
最近更新 2026-06-09 21:10:09
分类：搜索依赖：代码执行 / 第三方 API应用场景：研究 / 数据处理

安装技能

复制下面的话术发给你的 Agent，它会通过 secure-skills 检索并安装该技能。

执行命令安装此 skill： npx secure-skills-cli add justserpapi-web
Metadata
name
Web Crawling API
description
Fetch raw HTML, rendered HTML, or clean Markdown from public webpages through Just Serp API.
author
Just Serp API
homepage
https://justserpapi.com/?utm_source=clawhub.ai&utm_medium=referral&utm_campaign=justserpapi_web&utm_content=project_link
metadata
{"openclaw":{"homepage":"https://justserpapi.com/?utm_source=clawhub.ai&utm_medium=referral&utm_campaign=justserpapi_web&utm_content=project_link","primaryEnv":"JUST_SERP_API_KEY","requires":{"bins":["node"],"env":["JUST_SERP_API_KEY"]},"skillKey":"justserpapi_web"}}

Web Crawling

Use this skill when the user needs webpage retrieval rather than a search engine result. It fits crawling, scraping preparation, readable content extraction, and page structure inspection for a known URL.

When To Use It

  • The user already has a target webpage URL and wants its raw HTML, rendered HTML, or cleaned Markdown.
  • The task is about content extraction, page inspection, scrape preparation, or converting a page into LLM-friendly text.
  • The user can provide a direct url to crawl.
  • The user needs page content from the source URL itself, not Google search results about that URL.

Representative Operations

  • html: Crawl Webpage (HTML) — Retrieve the raw HTML response for a page.
  • renderedHtml: Crawl Webpage (Rendered HTML) — Retrieve DOM output after rendering for JavaScript-heavy pages.
  • markdown: Crawl Webpage (Markdown) — Extract the main readable content as clean Markdown for summarization or downstream processing.

Request Pattern

  • 3 read-only GET operations are available in this skill.
  • All operations require a direct url query parameter.
  • No operation in this skill requires a request body.
  • Choose renderedHtml for dynamic pages, html for raw source, and markdown for readable content extraction.

How To Work

  1. Read generated/operations.md before choosing an endpoint.
  2. Start with one of these operations when it matches the user's request: html, renderedHtml, markdown.
  3. Pick the smallest matching operation instead of guessing.
  4. Ask the user for any missing required parameter. Do not invent values.
  5. Call the helper with:
node {baseDir}/bin/run.mjs --operation "<operation-id>" --api-key "$JUST_SERP_API_KEY" --params-json '{"key":"value"}'

Environment

  • Required: JUST_SERP_API_KEY
  • This skill uses JUST_SERP_API_KEY only for authenticated Just Serp API requests.
  • Keep JUST_SERP_API_KEY private. Do not paste it into chat messages, screenshots, or logs.
  • Project site: Just Serp API.
  • Authentication details: Just Serp API Docs.

Output Rules

  • Start with what was fetched: raw HTML, rendered HTML, or cleaned Markdown.
  • Echo the target URL so the crawl scope is explicit.
  • For markdown, surface the extracted readable content or key sections before raw JSON.
  • For HTML-oriented requests, mention whether the user asked for source HTML or rendered output.
  • If the backend errors, include the backend payload and the exact operation ID.