hfpclawer Paper Search & Download

Part of the Exo suite — literature (hfpclawer) → experiments (expflow-pde) → proofs (omega-architect). Three independent CLIs that meet through files and CLI calls, never imports. Entry skill: exo-suite-linkage (wiring, cost tiers low → medium → high, degradation ladder). Install: uv tool install hfpclawer · uv tool install expflow-pde · uv tool install "omega-architect @ git+https://github.com/diamond2nv/omega-architect@v0.2.3"

🔒 Sanitization: This skill ships in the public repo. Never embed private LAN IPs, real person names, or machine codenames in examples — use <placeholder> / Jane Doe / dev@example.com. Real values live in the repo's gitignored .hermes/internal-guide.md. See repo AGENTS.md "Public-Release Sanitization".

A multi-source academic paper pipe: search across arXiv / HuggingFace Papers / OpenReview / PapersWithCode, deduplicate by title, download PDFs, convert to Markdown, and optionally sync to a wiki.

Who this is for: Researchers who want a daily "new papers on my topic" feed without manually checking multiple websites.

Overview

Typical workflow in one command:

hfpclawer search           # Discover new papers across sources
   └── ranked by relevance to your keywords
hfpclawer download         # Download PDFs for matched papers  
   └── 8 concurrent streams
hfpclawer convert --to-wiki # PDF → readable Markdown + wiki sync

Direct fetch of a known arXiv ID (v0.16.11+)

hfpclawer fetch <arxiv_id> downloads one paper by ID with a built-in network escape ladder — TCP quick-probe → QUIC/HTTP-3 → browser-hint. Use it whenever arXiv TCP/443 is RST-reset (common on CN networks); QUIC over UDP/443 survives the reset and is the primary escape channel.

hfpclawer fetch 2609.02737                    # PDF, auto transport
hfpclawer fetch 2608.06013 -k source -t quic  # TeX bundle, force QUIC

Robustness built in: bounded-memory streaming (512KB flush to .part), resume on interruption (fresh .part continues via Range; >24h stale .part reclaimed), and a sha256 integrity anchor printed on every completed fetch and appended to data/download_audit.jsonl.

Channel-verification trick: fetch the same arXiv ID twice — once from the official arXiv endpoint and once from a mirror (e.g. AlphaXiv; its real PDFs live at pdfs.assets.alphaxiv.org) — and compare sha256. Byte-identical hashes verify the QUIC channel end-to-end (cross-channel MITM detection).

Or run the full pipeline at once:

hfpclawer full --max-pages 3 --to-wiki

Multi-Source Registry (v0.18+)

Beyond the default HF/arXiv path, three biomedical adapters are registered — europepmc, biorxiv, medrxiv. They run only when enabled:

search:
  enabled: [hf_cli, arxiv_api, europepmc, biorxiv, medrxiv]

hfpclawer source-list                               # which adapters exist
hfpclawer source-search europepmc "CRISPR screen"   # query one adapter directly

• Registration is not enablement — an adapter in the registry does nothing until its key is in search.enabled, so adding one cannot change existing behaviour.
• Keep personal query lists and real names in the gitignored config.local.yaml; search.biomed_queries: [] in the tracked file is the declared placeholder slot.
• Europe PMC returns no abstract unless resultType=core is set (the adapter sets it); bioRxiv/medRxiv expose a date-range API only, so keyword filtering happens at ingest.
• Transient failures retry through anti_crawl.max_retries / retry_http_codes / retry_delay_base.

Prerequisites

Python ≥ 3.10. Pick the install that matches how you work — uv is recommended, because the CLI then lives in its own environment (no conflicts with your project's dependencies):

# 1) Recommended — uv tool: isolated CLI install, `hfpclawer` on your PATH
uv tool install hfpclawer
hfpclawer init                    # writes config.yaml

# 2) Try it without installing anything (ephemeral, one-off runs)
# pin the version — `uvx`/`uv tool run` reuse an installed tool env (may run an older
# release), and an unpinned launch is a supply-chain (rug-pull) risk
uvx "hfpclawer==0.19.0" --help

# 3) Inside an existing project / venv (uv-managed)
uv pip install hfpclawer

# 4) No uv yet — pip and pipx both work
pip install hfpclawer             # or: python -m pip install hfpclawer
pipx install hfpclawer            # CLI-style install, functionally like `uv tool`

Optional extras — the core install stays deliberately small:

⚠️ nlp extra + PyPI (checked against the published 0.19.0 metadata): PyPI strips the direct-URL spaCy model, so that extra installs spacy only — fetch the model yourself with python -m spacy download en_core_web_sm. The loader falls back configured → en_core_web_md → en_core_web_sm and logs one actionable hint when none is present: entity enrichment degrades, nothing else breaks.

Where to find it: repo https://github.com/diamond2nv/hfpapers-crawler · PyPI https://pypi.org/project/hfpclawer/ · registry: clawhub inspect <slug>

Edit config.yaml with your search interests (see Configuration section below).

Quick Start

1. First-time Setup

# Create default config
hfpclawer init

# Edit the config to match your research interests
vim config.yaml
# → Change: search.queries, keywords.include_high, keywords.exclude

2. One-Shot Full Pipeline (daily use)

# Discover → Download → Convert → Wiki sync in one command
hfpclawer full

# Limit pages for a quick check
hfpclawer full --max-pages 3 --to-wiki

3. Step-by-Step (for debugging)

# Step 1: Search across all sources
hfpclawer search --max-pages 5

# Step 2: Download PDFs for matched papers
hfpclawer download

# Step 3: Convert PDFs to Markdown
hfpclawer convert

# Step 4: Sync to wiki directory
hfpclawer convert --to-wiki

4. Monitor New Papers Regularly

# Check what papers have been downloaded
hfpclawer list

# Show paper store statistics
hfpclawer store stats

# Start the real-time download monitor
hfpclawer monitor start

Configuration

The config file config.yaml controls what papers are searched and downloaded:

search:
  max_per_dim: 50           # Papers per search query per source
  queries:
    - query: "neural operator"
      category: neural-operator
    - query: "physics-informed"
      category: physics-informed
    - query: "PDE solver deep learning"
      category: pde-solver

keywords:
  include_high:              # Papers must match these (OR)
    - "neural operator"
    - "pde"
    - "deep learning"
  include_low:               # Optional bonus keywords
    - "fourier"
    - "self-attention"
  exclude:                   # Exclude these topics
    - "quantum"
    - "llm"

classification:
  threshold_pass: 30         # Relevance score threshold (0-100)
  title_similarity_min: 0.40 # Dedup threshold

paths:
  data_dir: "data"           # SQLite DB location
  pdf_dir: "pdfs"            # Downloaded PDFs
  md_dir: "mds"              # Converted Markdown files

Available Commands

Daily Routine Examples

Morning — Check What's New

# Quick scan (3 pages per query, ~50 papers)
hfpclawer search --max-pages 3

# View results
hfpclawer store stats

Afternoon — Download & Read

# Download all new papers
hfpclawer download

# Convert to readable markdown
hfpclawer convert

# Read the best one
cat mds/2010.08895.md | head -80

Weekly — Full Pipeline

# Full sweep with wiki sync
hfpclawer full --max-pages 10 --to-wiki

# Validate references in newly added papers
hfpclawer audit verify "Key cited paper" --source openalex

Data Storage

hfpclawer uses three tiers:

The paper store tracks:

• arXiv ID, title, authors, abstract
• Source of discovery (HF / arXiv / OpenReview)
• Download status, conversion status
• Wikified path (if synced)
• Cross-verification with Crossref (DOI validation)

Search · Breadth · Recommend (搜广推, v0.18+)

Three families, each usable on its own — combined they form the discovery loop:

# Recommend papers for this repo (fuses config queries + repo/user profile, keeps suspect out)
hfpclawer recommend --limit 10
hfpclawer recommend --path ~/my-project        # read the profile from another repo

# Show the profile that drives it (repo block + private user profile)
hfpclawer profile            # repo: hfpclawer: YAML block in the nearest AGENTS.md
hfpclawer profile --user     # private ~/.hfpclawer/profile.yaml

# Grow the positive-example pool, then train the learned re-ranker (local lightgbm)
hfpclawer pool sync-favorited                  # Zotero favorites → positives
hfpclawer pool ingest-verified                 # human-approved papers
hfpclawer pool export --out data/train.jsonl
hfpclawer rank train --audit data/expand-audit.jsonl --out data/rank_model.txt

Why it is auditable: every recommendation row carries its provenance (which layer, which query, verification state). Suspect papers are gated out; verified ones are preferred.

Zotero Integration (read · write · ingest)

Talks to Zotero Desktop's local server on port 23119 — the local server needs no cloud account; the zotero pyzotero extra is still required for these paths. Zotero must be running.

hfpclawer zotero check                        # verify connectivity first
hfpclawer zotero list --limit 10 --tag hfpclawer
hfpclawer zotero search "neural operator" --limit 5
hfpclawer zotero get ABC123
hfpclawer zotero push <arxiv-id>              # paper_store → Zotero
hfpclawer zotero ingest ABC123                # Zotero PDF → store + wiki + annotations

Override ZOTERO_API_URL only if Zotero listens elsewhere. Real Zotero user ids are private — keep them in gitignored config, never in tracked files.

Hermes Agent Environment

hfpclawer is built to run inside Hermes Agent (and OpenCode) as a first-class tool — the agent discovers the skill, and every command below is callable without leaving the session.

1. Install the skill — place this folder under ~/.hermes/skills/research/<slug>/ (or install from ClawHub: clawhub inspect <slug> --file SKILL.md); Hermes loads it automatically and skill_view(name='<slug>') returns this file.

2. Register the MCP server so the agent calls the CLI as tools:

# ~/.hermes/config.yaml
mcp:
  servers:
    hfpclawer:
      command: "hfpclawer"
      args: ["mcp"]        # stdio mode — Hermes native MCP client

For OpenCode / debugging use HTTP mode: hfpclawer mcp --mode http --port 8765.

3. Environment variables — every one is optional; the pipeline runs with none set:

4. Cost profile — the mechanical layer (search / dedup / verify / audit / Zotero) needs low-cost by design — not zero-cost. The core path (search / dedup / verify / audit / Zotero) issues no LLM call and needs no API key, so the marginal cost per run is small; it is still not zero — bandwidth, disk and CPU are spent, and rate-limited upstreams (arXiv / OpenAlex / Semantic Scholar) can throttle or expect a key at volume. LLM features (sniff, abstract triage) are opt-in and metered where they run; rank trains locally with lightgbm. Describe the cost as low, and keep the LLM steps explicit — do not advertise the stack as zero-token.

5. Keep private data private — real author lists, ORCIDs and Zotero ids belong in config.local.yaml (gitignored) or ~/.hfpclawer/profile.yaml, never in tracked files. The repo-side profile (the hfpclawer: block in a project AGENTS.md) is deliberately public-safe: neutral academic keywords only.

Common Pitfalls

1. pip install needs to be in the right venv. If hfpclawer command is not found, check the active Python environment.
2. HuggingFace CLI rate limits. Too many queries per minute will trigger 429s. Reduce max_per_dim to 10 if this happens.
3. Scrapy spiders need scrapy extra installed. If you see ModuleNotFoundError: scrapy, run pip install hfpclawer[scrapy].
4. PDF conversion needs pymupdf4llm. Run pip install hfpclawer[pdf] if hfpclawer convert complains about missing pymupdf4llm.
5. Wiki sync defaults to raw/papers/. If you do not have a wiki directory, skip --to-wiki and read from mds/ directly.
6. First run creates a config.yaml. Edit it before running hfpclawer full, otherwise the default queries may not match your research area.

Verification Checklist

• hfpclawer init creates a valid config.yaml
• hfpclawer search --dry-run validates config without network calls
• hfpclawer search --max-pages 3 returns real papers
• hfpclawer download downloads PDFs correctly
• hfpclawer convert produces readable Markdown
• hfpclawer store stats shows non-zero counts
• hfpclawer store verify --arxiv-id 2010.08895 cross-checks via Crossref