360 Secure SkillHub
360 安全技能中心

Apify Scraper

通过 Python 脚本调用 Apify 平台的云端 Actor，抓取 Twitter/X 线程、Reddit 帖子与评论、LinkedIn 资料、YouTube 评论及 Google SERP 等被反爬机制拦截的站点内容，脚本轮询至任务完成后以数据集形式返回结果。需要 APIFY_API_KEY，消耗 Apify 账户额度。

nissan
nissan
热度 60.90
最近更新 2026-05-11 17:37:34
分类：搜索依赖：代码执行 / 第三方 API应用场景：研究 / 社交媒体 / 数据处理 / 自动化脚本

安装技能

复制下面的话术发给你的 Agent，它会通过 secure-skills 检索并安装该技能。

执行命令安装此 skill： npx secure-skills-cli add apify-scraper
Metadata
name
apify-scraper
version
1.0.0
description
Scrape content from sites that block bots — Twitter/X threads, Reddit, LinkedIn, YouTube comments, Google SERP, and more. Use when standard web_fetch is blocked and you need full thread/comment data. Requires APIFY_API_KEY.
metadata
{"openclaw":{"emoji":"🕷️","requires":{"bins":["python3"],"env":["APIFY_API_KEY"]},"primaryEnv":"APIFY_API_KEY","network":{"outbound":true,"reason":"Calls Apify API (api.apify.com) to run cloud actors that scrape external sites. Actors run server-side on Apify infrastructure."},"security_notes":"All scraping is performed via the user's own Apify account using their API key. Actors run on Apify's infrastructure — no data is sent to third parties beyond Apify (the user's chosen scraping platform). LinkedIn scraping is ToS-sensitive; use sparingly for research purposes only."}}

Apify Scraper Skill

Use this skill when you need to scrape content from sites that block bots — Twitter/X threads, Reddit, LinkedIn, YouTube comments, Google SERP, Amazon, Product Hunt, etc.

When to Use

  • A Twitter/X URL is shared and you need the full thread (not just the first tweet)
  • You need Reddit thread content without the expensive API
  • LinkedIn company/profile data is needed
  • YouTube comments or video metadata beyond what the API gives
  • Google search results programmatically
  • Any site that blocks standard web_fetch

Do NOT use for: sites accessible via normal web_fetch or web_search. Apify costs credits — use it only when needed.

Setup

  • API Key: op://OpenClaw/Apify API Credentials/credential (also in gateway plist as APIFY_API_KEY)
  • Dashboard: https://console.apify.com (account: redditech)
  • Plan: FREE ($5/mo credit)
  • Script: python3 scripts/apify-run.py <actor> <input_json>

Running an Actor

python3 /Users/loki/.openclaw/workspace/scripts/apify-run.py \
  "apidojo/tweet-scraper" \
  '{"twitterHandles": ["solanamobile"], "maxItems": 50}'

Key Actors

Twitter/X

  • apidojo/tweet-scraper — $0.40/1K tweets. Full thread support via conversationIds. Advanced search syntax. 
    {"conversationIds": ["2034675043033375103"], "maxItems": 50}
    or by handle: 
    {"twitterHandles": ["solanamobile"], "maxItems": 20}

Reddit

  • trudax/reddit-scraper-lite — Free tier friendly. Fetch threads + comments. 
    {"startUrls": [{"url": "https://reddit.com/r/solana/comments/..."}], "maxItems": 100}

YouTube

  • streamers/youtube-scraper — Comments + metadata. 
    {"startUrls": [{"url": "https://youtube.com/watch?v=..."}], "maxComments": 200}

Google SERP

  • apify/google-search-scraper — Search results as structured data. 
    {"queries": "solana mobile grants", "maxPagesPerQuery": 1}

LinkedIn

  • anchor/linkedin-profile-scraper — ⚠️ ToS risk. Use sparingly for research only.

Pricing Reference

  • 1 CU = 1 GB RAM × 1 hour
  • Free tier: $5/mo (~16.7 CU)
  • Tweet scraping: 0.035–0.04 CU/1K tweets ($0.01/1K on free tier)
  • Some actors charge flat per-result: $0.25–$0.40/1K tweets
  • Check usage: https://console.apify.com/billing

Notes

  • Results are returned as a dataset — the script polls until complete
  • Timeout: 5 minutes default (most actors finish in 30–60s)
  • If an actor breaks (community-maintained), check Apify Store for alternatives
  • MCP integration pending — Apify MCP server exists but openclaw.json doesn't support mcpServers key yet (schema validation rejects it). Use this script approach instead.