Metadata

name: free-web-search-js
description: Web search & content fetching via Playwright + HTTP — zero API keys. Searches Bing CN (domestic) or DDG (international), auto-fetches top page contents. For real-time search, fact-checking, news, tutorials, documentation lookup.
version: 29.1.0
license: MIT
author: free-web-search-js
trigger_keywords: ["search","find","look up","搜索","查一下","找一下"]
tools: [{"name":"search","description":"Search + auto-fetch; Bing CN via Playwright, DDG via HTTP","script":"scripts/search.js","parameters":{"query":{"type":"string","description":"Search query","required":true},"max":{"type":"integer","description":"Max results 1-30, default 10","required":false},"region":{"type":"string","description":"Region: auto/cn/intl, defaults to auto (IP detection)","required":false}}},{"name":"fetch","description":"Fetch page content by URL, HTTP first with headed fallback","script":"scripts/fetch.js","parameters":{"urls":{"type":"string","description":"URLs to fetch, space-separated","required":true},"max-len":{"type":"integer","description":"Max chars per page, default 12000","required":false}}}]

free-web-search-js

Pure search — no rewriting, no filtering, noise handled by the AI. Uses real browser automation (Playwright) for Bing CN anti-bot, with HTTP fallback for international queries. Zero API keys required.

Quick start

# Search (auto-fetches top 3 page contents)
node scripts/search.js "silver price"
node scripts/search.js "how to deploy docker" --max=5
node scripts/search.js "xxx" --region=cn
node scripts/search.js "xxx" --fetch=5         # fetch top 5
node scripts/search.js "xxx" --no-fetch        # search only

# Fetch specific URLs
node scripts/fetch.js "https://example.com/page1" "https://example.com/page2"

Architecture

Domestic (CN IP):
  Playwright headless → Bing CN → home page for cookie → search box submit
  → 0 results? fallback to DDG HTML
  → auto-fetch top N page contents

International:
  Pure HTTP → DDG HTML parse
  → fetch fails? fallback to Playwright request
  → DDG empty? fallback to Bing
  → auto-fetch top N page contents

Install

All dependencies are installed upfront by the setup script. The skill never runs npm install or pip install at runtime. This avoids audit flags and supply-chain risks.

Prerequisites

Dependency	Purpose	Size
Node.js >= 18	Runtime	—
cheerio	HTML parsing	small
commander	CLI arg parsing	small
iconv-lite	GBK charset conversion	small
playwright	Browser automation	~50 MB
Chromium (Playwright)	Browser engine for Bing search + headed fetch	~150 MB

One-command setup

# Linux / macOS
bash scripts/setup.sh

# Windows
powershell -File scripts/setup.ps1

The scripts auto-detect region and use npmmirror.com mirrors inside China for faster downloads.

Manual install

cd skills/free-web-search-js
npm install
npx playwright install chromium    # ~150 MB

# Verify
node scripts/check-env.js

If you already have Chrome/Chromium installed:

export CHROMIUM_EXECUTABLE_PATH="/path/to/chrome"
node scripts/check-env.js

check-env.js will confirm whether the provided executable is usable.

Environment variables

Variable	Purpose
`CHROMIUM_EXECUTABLE_PATH`	Path to system Chrome/Chromium, skips 150 MB download
`PLAYWRIGHT_DOWNLOAD_HOST`	Mirror for Chromium download (auto-set by setup script)
`BROWSER_DAEMON_IDLE_MS`	Daemon idle timeout in ms (default 600000 = 10 min)

Search engines

Engine	Method	Region	Notes
Bing CN	Playwright search box submit	Domestic	Visits home page first for cookie, then types into search box
DDG	Playwright search box submit	International	Full browser automation

Strategy

Region	Search	Fetch
Domestic	Bing CN, fallback DDG	Auto-fetch top 3
International	DDG HTML, fallback Playwright request; DDG empty→Bing	Auto-fetch top 3

IP detection

On each search, three probes run in parallel — first responder wins:

Probe	Service	Logic
1	`myip.ipip.net` / `cip.cc`	Reachable from China
2	`ipinfo.io` / `ipapi.co`	International
3	`cn.bing.com`	Reachable → likely CN
Fallback	—	Defaults to domestic

Override with --region=cn or --region=intl if your proxy causes misdetection.

Fetch mode

After search, auto-fetches the top N results (default 3).

Tier	Method	Speed	Notes
1	Lightweight HTTP + cheerio	Fast	No browser launch
2	Playwright headed	Slower	Full JS rendering

Tier 1 enhancements:

JSON API response detection with structured content extraction
JSON-LD (<script type="application/ld+json">) articleBody/description
__NEXT_DATA__ embedded data extraction
Meta tag fallback (og:description / description)
Auto GBK charset detection & conversion

Deduplication

Smart dedup: domain + path stem (ignores www/m subdomains, tracking params, trailing slash, .html suffix). Bing redirect URLs (bing.com/ck/) auto-decoded.

Browser daemon (optional, ~70% faster)

Avoid 1.5s+ browser launch overhead by reusing a persistent Chromium instance:

node scripts/browser-daemon.js            # Start (run with & or as background job)
node scripts/browser-daemon.js --status   # Show status
node scripts/browser-daemon.js --stop     # Stop

The daemon auto-exits after 10 minutes of inactivity (configurable via BROWSER_DAEMON_IDLE_MS). Activity is tracked via a .browser-heartbeat file that search.js and fetch.js touch on each connection. The daemon checks the heartbeat every 60 seconds.

Verification & troubleshooting

cd skills/free-web-search-js
node scripts/check-env.js
node scripts/search.js "OpenClaw" --max 2 --region cn --fetch 0
node scripts/search.js "OpenClaw" --max 2 --region intl --fetch 0

Common issues:

node_modules not found → run npm install
Executable doesn't exist ... → run bash scripts/setup.sh or set CHROMIUM_EXECUTABLE_PATH
browserType.launch ... closed → headless+no-sandbox is default; run node scripts/check-env.js to diagnose
0 search results → try --region=cn / --region=intl, or --fetch 0 to test search alone

Privacy & network

Searches connect to Bing CN / DDG HTML / Sogou directly from the skill's runtime — no intermediate proxy
The daemon heartbeat file (.browser-heartbeat) and endpoint file (.browser-endpoint) are written to the skill root directory
No analytics, no telemetry, no third-party logging beyond the search engine's own server logs
Headed warm-up fetches: Cookie warm-up for Bing uses a headed browser context to obtain session cookies from the search engine home page; no cookies are persisted to disk between sessions

Known limitations

First domestic search slower: Chromium cold start takes 3–6s; daemon avoids this
Bing CN instant answers: Weather/calculator cards don't use li.b_algo, return 0 results
Sogou HTTP unstable: No cookie, easily blocked by anti-crawl
JS-required pages: HTTP tier returns empty for SPA pages — headed tier catches them
Region-specific sites: Some domestic sites time out from international IPs
Proxy interference: IP detection may misclassify behind proxies — use --region to override
DDG blocked in China: DDG HTML is not reachable from mainland China; domestic strategy avoids it

Free Web Search Js

安装技能