Web Search Plus

Stop choosing search providers. Let the skill do it for you.

One command searches across 11 search providers, and a companion command extracts page text from URLs. Since 5.0 the skill runs the same engine as Web Search Plus 5.0.1: automatic routing picks the first provider by query type and moves on quickly when a provider fails, is slow or returns nothing.

This skill is source-only: ranked links and extracted page text, not model-written answers. The native OpenClaw plugin is web-search-plus-plugin-v2.

🔐 Data Handling & Privacy

Read this before searching with sensitive queries.

• Search queries and extraction URLs are sent to third-party providers. Every search sends your query text to the provider that serves it (Serper, Brave, Tavily, Linkup, Querit, Exa, Firecrawl, SerpBase, Keenable, You.com, or your SearXNG instance). When a provider fails, is slow or returns nothing, the query also goes to the next provider in line. Every extraction sends the target URL to the chosen extraction provider (Tavily, Exa, Linkup, Firecrawl, You.com, Keenable, Serper), whose servers fetch the page. Each provider's own privacy policy and retention rules apply.
• For sensitive work, select the provider explicitly (--provider <name>) instead of relying on automatic routing, so you control exactly which third party receives the query. An explicit provider does not fall back unless you add --allow-fallback. Self-hosted SearXNG keeps queries on infrastructure you control.
• Do not submit internal or private URLs for extraction. URLs you extract are forwarded to external services. The skill blocks private, loopback and link-local targets and cloud metadata endpoints by default (see Security below).
• Local caching is on by default. Search and extraction answers (including the query text), the full text of long extracted pages (kept up to 7 days), provider state (v3/state.sqlite3: circuit breakers, budgets, latency samples; provider_stats.json, provider_health.json) and routing receipts without query text are stored under the cache directory (.cache by default, WSP_CACHE_DIR to relocate). Directories are created 0700, files 0600.
  • Bypass the answer cache for one call: --no-cache
  • Bypass it for every call: WSP_DISABLE_CACHE=1 (provider state is still kept)
  • Wipe cached answers: python3 scripts/search.py --clear-cache (inspect with --cache-stats)
• API keys are never logged or cached; error text is sanitized before it reaches the cache or stderr.

🎯 Triggers

To avoid auto-activating on everyday requests, the manifest registers only narrowly scoped trigger phrases:

• web search plus
• wsp search
• search the web for
• multi-provider web search
• extract url content
• extract content from url

Generic words like "search", "find", "look up", or "research" intentionally do not trigger this skill.

✨ What Makes This Different?

• Just search: no need to think about which provider to use
• Routing by query type: Brave for general, news, local and community queries, Exa for docs and academic queries, Serper for security and shopping queries
• Fast fallback: if the first provider fails, answers with nothing, or is slower than usual (at least 2.5 s), the next one starts and the first useful answer wins
• 11 providers, 1 interface: general web, research, semantic discovery, privacy-first, prepaid-credits, and extraction-capable providers together
• URL extraction included: markdown/HTML page text with fallback across seven providers; one bad URL no longer sinks the batch
• Research mode: several providers at once, results merged by rank, top sources extracted with short query-ranked passages
• Works with just 1 credential: start with any single provider, add more later

🚀 Quick Start

# Interactive setup (recommended for first run)
python3 scripts/setup.py

# Or manually
cp .env.example .env
python3 scripts/search.py -q "latest OpenClaw release"
python3 scripts/extract.py --url https://example.com

With only one key, Brave is the best single choice for automatic routing; Exa and Serper cover the other query types.

🔑 Providers

Search providers

• Brave: independent web index; first choice for general, news, local and community queries
• Serper: Google-style results; first choice for security and shopping queries
• Exa: semantic search with source URLs; first choice for docs and academic queries, also --similar-url
• Tavily: research-oriented search; fourth in the fallback chain
• Querit: multilingual and international search; explicit-only unless allowed in auto_routing.auto_allow
• Linkup: source-grounded search
• Firecrawl: search with scrape-ready metadata
• You.com: current-web snippets
• SearXNG: private/self-hosted search; no API key, just an instance URL
• SerpBase: low-cost Google SERP with prepaid credits; explicit/fallback-only (--provider serpbase)
• Keenable: independent web index; keyed via KEENABLE_API_KEY or keyless against the opt-in shared public tier (WSP_KEENABLE_ALLOW_PUBLIC=1, no SLA, warning in result metadata). Last in the fallback chain

Extraction providers

scripts/extract.py falls back in this order:

1. Tavily
2. Exa
3. Linkup
4. Firecrawl
5. You.com
6. Keenable
7. Serper (webpage scraper via scrape.serper.dev)

🧠 Routing at a Glance

Automatic routing gives each query one of eight types and picks the first provider from that:

academic, docs   → Exa
security, shopping → Serper
everything else  → Brave   (general, news, local, community)

A provider without a key is skipped. Fallback then follows auto_routing.provider_priority, by default:

brave → serper → exa → tavily → you → firecrawl → linkup → searxng → keenable

SerpBase and Querit are in the list too but are only used when named with --provider or allowed with "auto_routing": {"auto_allow": {"querit": true}}. To use your own order for every query instead, set "auto_routing": {"order": "custom", "provider_priority": [...]} in config.json.

Examples:

python3 scripts/search.py -q "weather in Vienna today"                 # local → Brave
python3 scripts/search.py -q "python asyncio TaskGroup documentation"  # docs → Exa
python3 scripts/search.py -q "iPhone 16 Pro Max price"                 # shopping → Serper
python3 scripts/search.py -p firecrawl -q "YC startups web scraping"   # explicit provider
python3 scripts/search.py -p serpbase -q "best laptop 2026"            # explicit SerpBase

Show the decision without searching:

python3 scripts/search.py --explain-routing -q "your query"

Queries starting with a dash can use --query=-foo or -q -- "-foo". When -n is omitted, defaults.max_results supplies the count (fallback 5); counts are clamped to 1–20. Long queries are shortened at a word boundary when a provider has a length limit (Brave: 600 characters / 75 words), and the output says so.

--cache-ttl SECONDS sets the answer-cache lifetime; recency, date filters and news/security queries (300 s) can only lower it. --no-cache bypasses the cache. Set WSP_HTTP_KEEPALIVE=0 to disable connection reuse; requests through proxies use the standard transport.

Freshness, news vertical & locale

python3 scripts/search.py -q "AI regulation" --freshness week
# providers with native date filters receive the mapped value; others run the
# normal search and report freshness.applied=false in metadata

python3 scripts/search.py -p serper --type news -q "quantum computing"
# Serper serves the Google news vertical natively; other providers report
# search_type.applied=false

python3 scripts/search.py -q "beste Kaffeehäuser Wien"
# location hints in the query set the country (at); a clearly German query is
# searched in German. Resolved values appear in metadata.locale.

Without a configured language, a clearly detected query language is sent to the provider; unclear queries go out as English. Set a fixed language with "defaults": {"locale": {"language": "en"}} in config.json or WSP_LOCALE_LANGUAGE=en.

Domain filters and result quality

--include-domains / --exclude-domains work on every search provider (Brave, Serper, SerpBase and You.com get site: / -site: operators). Known mirror and spam domains are removed, and one domain fills at most two of the top results; explicit domain filters bypass both. For docs, security and news queries the site of the named vendor or project ranks first.

📖 Extraction Examples

python3 scripts/extract.py --url https://example.com
python3 scripts/extract.py --url https://docs.linkup.so --provider linkup
python3 scripts/extract.py --url https://example.com --url https://example.org --include-images
python3 scripts/extract.py --url https://example.com --format html --include-raw-html
python3 scripts/extract.py --url https://example.com --provider serper
python3 scripts/extract.py --url https://example.com --extract-char-limit 30000
python3 scripts/extract.py --url https://example.com --spans --spans-query "pricing"

Each URL gets its own result: a blocked or failing URL gets an error entry and the others are still extracted, retried on the next provider if needed, in the order you gave them. Long pages return a head/tail window plus a footer with the real page length (default 15,000 characters per page; WSP_EXTRACT_CHAR_LIMIT or --extract-char-limit). When the engine kept the full text, the footer names the file. Inline base64 images become [IMAGE: alt] placeholders. Extraction answers are cached for an hour (--no-cache to bypass).

🔬 Research Mode & Quality Reports

Research mode queries up to three providers at once, merges their results by rank (a page found by several providers ranks higher and lists them in providers), then extracts the top sources and adds short query-ranked passages:

python3 scripts/search.py --mode research -q "EU AI Act obligations for foundation models"
python3 scripts/search.py --mode research -q "..." --research-providers tavily linkup exa
python3 scripts/search.py --mode research -q "..." --research-extract-count 2 --research-time-budget 30

Quality reports add routing and result diagnostics, including authority signals:

python3 scripts/search.py -q "official anthropic claude release notes" --quality-report

⚙️ Configuration Notes

• .env.example documents supported env vars; .env is read from the skill directory only
• config.example.json shows the 5.0 layout; config.json is your local config
• 4.x config.json files keep working: a top-level locale block, provider country/language keys, the old default provider order and the old SerpBase endpoint are translated when the config is loaded
• SerpBase is explicit/fallback-only by default
• Keenable is last in search and extraction fallback; enable keyless use with WSP_KEENABLE_ALLOW_PUBLIC=1 (or KEENABLE_ALLOW_PUBLIC=1, or "keenable": {"allow_public": true})
• Locale defaults: "defaults": {"locale": {"country": "...", "language": "..."}} in config.json (or WSP_LOCALE_COUNTRY/WSP_LOCALE_LANGUAGE); --country/--language always win
• Rate limits: Retry-After waits are capped at 30 s; a longer one skips the retry and leaves the provider blocked (at most one hour). Missing keys never block a provider
• Extra commands: python3 scripts/search.py doctor (offline configuration check), python3 scripts/search.py --extract-urls URL ...

🔒 Security

URL SSRF protection (extraction, --similar-url):

• Only http / https URLs are accepted; URLs that different URL parsers read differently are rejected
• Hostnames are resolved and blocked if they point at private, loopback, link-local or reserved ranges; error messages do not reveal the resolved address
• Cloud metadata endpoints (169.254.169.254, metadata.google.internal) are always blocked
• Opt out for trusted private networks with --allow-private-urls or WSP_ALLOW_PRIVATE_URLS=1 (off by default; metadata endpoints stay blocked)

SearXNG SSRF protection:

• Enforces http / https only
• A SearXNG URL on a private or LAN address needs SEARXNG_ALLOW_PRIVATE=1; without it SearXNG is skipped and other providers keep working

Local data:

• Cache directory 0700, files 0600
• API keys are never written to the cache or logs
• The skill starts no subprocesses and opens no listening sockets

Declared permissions (see package.json → clawhub.permissions):

• Outbound network access limited to the listed provider API hosts (plus any user-configured SearXNG instance)
• Environment reads limited to provider *_API_KEY vars, SEARXNG_*, KEENABLE_ALLOW_PUBLIC, WEB_SEARCH_PLUS_CONFIG and WSP_* settings
• Filesystem writes limited to the cache directory (and config.json from the setup wizard)

✅ Verification

python3 -m unittest discover -s tests -p 'test_*.py'
python3 scripts/search.py --explain-routing -q "find credible sources for climate change impacts"
python3 scripts/search.py doctor
python3 scripts/extract.py --url https://example.com --provider auto --compact