N-gram SERP Intent Clusterer
AGENT ROLE: Autonomous keyword-clustering agent. Group keywords by SERP-result overlap (the only ground-truth signal for "same intent"), label intent, and emit the JSON in references/output.schema.json. Cluster by evidence, not by string similarity alone.
OBJECTIVE
Partition a keyword set into clusters where Google returns substantially the same URLs (⇒ one page can rank for all), assign each cluster a dominant intent, and name the canonical target URL.
INPUTS
keywords(REQUIRED string[]).overlap_threshold(OPTIONAL, default 3): min shared top-10 URLs for two keywords to be linked.top_n(OPTIONAL, default 10): SERP depth to compare.location/hl(OPTIONAL): default "United States"/"en".
AUTHENTICATION (SERP API)
- REQUIRE env
SERP_API_KEY. IF unset THEN STOPerror.code="AUTH_MISSING_API_KEY". - Endpoint (SerpApi shape):
GET https://serpapi.com/search.json?engine=google&q={kw}&num={top_n}&api_key={key}. - Record
serp_providerin output; the fetch layer is swappable for DataForSEO.
EXPECTED TOOL CALLS
- Run
scripts/cluster_intent.py --keywords keywords.json --overlap 3. - One SERP fetch per keyword; extract the ordered
organic_results[].linkset.
PROCEDURE (deterministic)
STEP 1 — FETCH the top-top_n organic URLs for every keyword (normalize to registrable-domain + path).
STEP 2 — BUILD an undirected graph: node = keyword; edge (a,b) IF |urls(a) ∩ urls(b)| >= overlap_threshold.
STEP 3 — CLUSTER = connected components (single-linkage on SERP overlap).
STEP 4 — INTENT per cluster: compute n-grams over the cluster's keywords + SERP titles; map modifier n-grams to intent via references-free rules — buy|price|cheap|deal|coupon|for sale→transactional; best|top|review|vs|alternative→commercial; how|what|why|guide|tutorial|ideas→informational; brand/domain tokens→navigational. Dominant = highest-weighted.
STEP 5 — CANONICAL TARGET: the URL appearing in the most keywords' SERPs within the cluster (modal URL); IF the user's own domain ranks, prefer it and note owns_target=true.
STEP 6 — EMIT clusters sorted by size desc; singletons returned as size-1 clusters.
RATE LIMITS & ERROR HANDLING
- SERP APIs bill per search. On
429/quota THEN backoff2^attempt(max 5); after that STOPerror.code="RATE_LIMITED"withpartial= keywords fetched. Never silently drop. - On
5xx/timeout retry ≤3 then mark that keywordstatus="serp_error"and exclude it from clustering (report inskipped). - Concurrency ≤ 3.
MISSING / INSUFFICIENT DATA
- IF a keyword returns < 3 organic URLs THEN it cannot be reliably clustered → return as singleton with
low_confidence=true. - Never merge two keywords below
overlap_thresholdeven if strings look similar (string similarity ≠ intent).
OUTPUT
One JSON object per references/output.schema.json. No prose.
FILES
scripts/cluster_intent.py— SERP fetch, overlap graph, connected-component clustering, intent labeling.references/output.schema.json— output contract.
