Smart Web Scraper

提供 extract/tables/links/structure/crawl 五个命令，通过本地 Python 脚本抓取静态网页，支持 CSS 选择器筛选、表格自动识别、分页爬取，并可输出 text/JSON/CSV/Markdown 或保存到文件，默认限速并遵守 robots.txt。

mariusfit

热度 89.50

最近更新 2026-05-11 17:27:00

分类：数据依赖：代码执行 / 读写文件应用场景：数据处理 / 自动化脚本 / 工作流

安装技能

复制下面的话术发给你的 Agent，它会通过 secure-skills 检索并安装该技能。

执行命令安装此 skill： npx secure-skills-cli add smart-web-scraper

Metadata

name: smart-web-scraper
description: Extract structured data from any web page. Supports CSS selectors, auto-detection of tables and lists, JSON/CSV output formats. Use when asked to scrape a website, extract data from a page, pull product info, gather contact details, or collect listings from a URL.

Smart Web Scraper

Extract structured data from web pages into clean JSON or CSV.

Quick Start

# Scrape a page, extract all text content
uv run --with beautifulsoup4 --with lxml python scripts/scraper.py extract "https://example.com"

# Extract specific elements with CSS selector
uv run --with beautifulsoup4 --with lxml python scripts/scraper.py extract "https://example.com/products" -s ".product-card"

# Auto-detect and extract tables
uv run --with beautifulsoup4 --with lxml python scripts/scraper.py tables "https://example.com/pricing"

# Extract all links from a page
uv run --with beautifulsoup4 --with lxml python scripts/scraper.py links "https://example.com"

# Extract structured data (title, meta, headings, links)
uv run --with beautifulsoup4 --with lxml python scripts/scraper.py structure "https://example.com"

# Output as JSON
uv run --with beautifulsoup4 --with lxml python scripts/scraper.py extract "https://example.com" -s ".item" -f json

# Output as CSV
uv run --with beautifulsoup4 --with lxml python scripts/scraper.py extract "https://example.com" -s "table tr" -f csv

# Save to file
uv run --with beautifulsoup4 --with lxml python scripts/scraper.py extract "https://example.com" -s ".product" -f json -o products.json

# Multi-page scrape (follow pagination)
uv run --with beautifulsoup4 --with lxml python scripts/scraper.py crawl "https://example.com/page/1" --pages 5 -s ".article"

Commands

Command	Args	Description
`extract`	`<url> [-s selector] [-f format] [-o file]`	Extract content, optionally filtered by CSS selector
`tables`	`<url> [-f format] [-o file]`	Auto-detect and extract all HTML tables
`links`	`<url> [--external] [--internal]`	Extract all links (href + text)
`structure`	`<url>`	Extract page structure: title, meta, headings, images, links
`crawl`	`<url> --pages N [-s selector] [-f format] [-o file]`	Follow pagination links, extract from multiple pages

Output Formats

Format	Flag	Description
Text	`-f text`	Plain text (default)
JSON	`-f json`	Structured JSON array
CSV	`-f csv`	Comma-separated values
Markdown	`-f md`	Markdown-formatted

Examples

Extract product listings

uv run --with beautifulsoup4 --with lxml python scripts/scraper.py extract "https://shop.example.com" -s ".product" -f json

Output:

[
  {"text": "Widget Pro - $29.99", "tag": "div", "class": "product"},
  {"text": "Widget Max - $49.99", "tag": "div", "class": "product"}
]

Extract pricing table

uv run --with beautifulsoup4 --with lxml python scripts/scraper.py tables "https://example.com/pricing" -f csv

Get all external links

uv run --with beautifulsoup4 --with lxml python scripts/scraper.py links "https://example.com" --external

Rate Limiting

Default: 1 request per second (respectful crawling)
Override with --delay 0.5 (seconds between requests)
Respects robots.txt by default (override with --ignore-robots)

Notes

Requires beautifulsoup4 and lxml (auto-installed by uv run --with)
Uses a standard browser User-Agent to avoid blocks
Handles redirects, encoding detection, and error pages gracefully
No JavaScript rendering (use for static HTML pages)