Phoenix Scraper

这是一款高韧性网页爬虫，具备三层自动故障转移机制：优先用Brave Search API（免费快速），失败则切换到Bright Data代理（支持JS渲染、分区路由），仍失败则启用Playwright无头浏览器执行全JS。适用于求职板、社交媒体、新闻等反爬严格或动态站点。输入目标URL及可选参数，输出结构化结果字典，无异常抛出，全失败时返回错误信息。

stevojarvisai-star

热度 40.90

最近更新 2026-05-11 17:42:28

分类：搜索依赖：代码执行 / 第三方 API / 联网浏览搜索应用场景：数据处理 / 自动化脚本 / 社交媒体 / 研究

安装技能

复制下面的话术发给你的 Agent，它会通过 secure-skills 检索并安装该技能。

执行命令安装此 skill： npx secure-skills-cli add phoenix-scraper

Metadata

name: phoenix-scraper
description: Resilient multi-layer web scraper with automatic failover. Use when scraping web content that may be JS-rendered, behind bot protection, or on sites that block standard HTTP requests. Supports three-tier failover chain: Brave Search API, then Bright Data Web Unlocker (with optional browser render mode), then Playwright headless browser. Use for job boards (LinkedIn, Glassdoor, Reed, Indeed, CWJobs, TotalJobs), social media monitoring (X/Twitter via API v2), news sites, and any page requiring JS rendering. Includes zone routing logic for Bright Data (web_unlocker vs job_search_scraper zones).

Phoenix Scraper

Resilient three-tier failover scraper. Never returns empty — if one method fails, the next activates automatically.

Failover Chain

Tier 1: Brave Search API (fast, free tier, 2k req/month)
    ↓ (on block/empty/timeout)
Tier 2: Bright Data Web Unlocker (residential proxy, JS-render optional)
    ↓ (on block/429/timeout)
Tier 3: Playwright headless browser (full JS execution)

Quick Start

from scripts.phoenix_scraper import scrape

# Basic fetch
result = scrape("https://example.com/page")

# With JS rendering (for SPA/dynamic sites)
result = scrape("https://example.com/page", render_js=True)

# With specific Bright Data zone
result = scrape("https://linkedin.com/jobs/...", zone="job_search_scraper")

Zone Routing

Use Case	Zone
Job boards (LinkedIn, Glassdoor, Reed, Indeed)	`job_search_scraper`
Social media, news, general web	`web_unlocker`
X.com / Twitter	Use X API v2 (see references/x-api.md)

Bright Data render_js

Set render_js=True for JS-heavy sites (CWJobs, TotalJobs, ContractorUK). Adds "render": True (boolean) to payload and uses 60s timeout.

Critical: Use boolean True, not string "html" — Bright Data validation rejects strings.

Bright Data Premium Domains (Cost Note)

LinkedIn, Glassdoor, and other heavily-protected job boards may be classified as Premium Domains in your Bright Data zone (updated quarterly). API call syntax is identical — but cost per request is higher. Check your zone's Premium Domains list if costs spike unexpectedly.

Playwright Stealth (2026 Enhancement)

For Tier 3, consider installing playwright-stealth to patch headless browser fingerprints — reduces detection on Cloudflare/advanced bot-protected sites:

pip install playwright-stealth

# Optional enhancement in phoenix_scraper.py Tier 3:
from playwright_stealth import stealth_sync
stealth_sync(page)

The base Playwright tier works without this, but stealth patching significantly improves success rates on heavily protected sites (Coupang, Naver, etc.) as of 2026.

URL Formatting

CWJobs/TotalJobs: use hyphen-slugs — finance-systems-consultant NOT finance+systems+consultant
Glassdoor: https://www.glassdoor.co.uk/Job/united-kingdom-{slug}-jobs-SRCH_IL.0,14_IN2_KO15,{end}.htm

Environment Variables

BRIGHT_DATA_API_KEY=<key>          # Bright Data API key
BRIGHT_DATA_ZONE=job_search_scraper # Default zone (override per-call)
BRAVE_API_KEY=<key>                # Brave Search API key
X_BEARER_TOKEN=<token>             # X API v2 bearer token (for X.com)

X.com Monitoring

For X/Twitter, use X API v2 (not scraping). See references/x-api.md for endpoint details and rate limits.

Error Handling

All tiers log failures before escalating. On total failure, returns {"success": False, "html": "", "method": "all_failed", "error": "<reason>"}.

Never raises exceptions — always returns a result dict.