Crawlee Web Scraper

该 Skill 提供两个 Python 脚本：crawlee_fetch.py 接收单个 URL 或 URL 列表文件，抓取后输出含状态码、抓取时间、长度与正文的 JSON，可选提取纯文本；crawlee_http.py 作为库辅助函数，先用 requests 请求，遇 403/429/503 时回退到 Crawlee。

bryantegomoh

热度 83.78

最近更新 2026-05-18 11:14:53

分类：搜索依赖：代码执行 / 读写文件 / 第三方 API应用场景：研究 / 数据处理 / 自动化脚本 / 工作流

安装技能

复制下面的话术发给你的 Agent，它会通过 secure-skills 检索并安装该技能。

执行命令安装此 skill： npx secure-skills-cli add crawlee-web-scraper

Metadata

name: crawlee-web-scraper
description: Resilient web scraper with bot-detection evasion using the Crawlee library. Use when web_fetch is blocked by rate limits or bot detection. Supports single URLs, bulk file input, and automatic fallback from requests to Crawlee on 403/429 responses.

crawlee-web-scraper

Drop-in replacement for web_fetch when sites block automated requests. Crawlee handles session management, retry logic, and bot-detection evasion automatically.

Scripts

crawlee_fetch.py — main scraper; accepts a single URL or a file of URLs; returns JSON
crawlee_http.py — library helper; tries requests first, falls back to Crawlee on 403/429/503

Usage

# Single URL, return HTML preview
python3 scripts/crawlee_fetch.py --url "https://example.com"

# Single URL, extract text (strips HTML tags)
python3 scripts/crawlee_fetch.py --url "https://example.com" --extract-text

# Bulk scrape from file
python3 scripts/crawlee_fetch.py --urls-file urls.txt --output results.json

Library usage

from crawlee_http import fetch_with_fallback

resp = fetch_with_fallback("https://example.com")
print(resp.status_code, resp.text[:500])

Output

JSON array with one object per URL:

[
  {
    "url": "https://example.com",
    "status": 200,
    "fetched_at": "2026-01-01T00:00:00Z",
    "length": 12345,
    "text": "Page content..."
  }
]

Installation

pip install crawlee requests

When to use

web_fetch returns 403 / 429 / empty
Bulk scraping 10+ URLs
Sites using Cloudflare or similar bot protection