Ahrefs-GSC Backlink Reconciler
AGENT ROLE: Autonomous link-reconciliation agent. Fetch Ahrefs referring domains, ingest the GSC top-linking-sites export, optionally add Semrush, collapse every host to its registrable domain, and reconcile into one authoritative set with source attribution. Emit the JSON in references/output.schema.json. Stateful across runs via authoritative_domains.
OBJECTIVE
Produce a single deduped referring-domain list that is honest about where each domain was seen (Ahrefs, GSC, Semrush), quantify how much each tool uniquely contributes, and surface the domains each tool misses plus the run-over-run gains and losses — so link audits stop trusting any one vendor's index.
INPUTS
target(REQUIRED): root domain to reconcile (e.g.example.com).gsc-links-csv(OPTIONAL but recommended): the Search Console "Top linking sites" CSV export. Absent → Ahrefs is returned without reconciliation.use-semrush(OPTIONAL flag): also pull Semrush referring domains (needsSEMRUSH_API_KEY).previous(OPTIONAL): prior run's output (itsauthoritative_domainsdrives gained/lost). Absent → baseline.max-domains(OPTIONAL, default 5000): cost guard on rows requested per vendor.verify-min-dr(OPTIONAL, default 30): Ahrefs-only domains at/above this DR are flagged for manual verification.
AUTHENTICATION (Ahrefs API v3; optional Semrush)
- REQUIRE env
AHREFS_API_TOKEN. IF unset THEN STOPerror.code="AUTH_MISSING_AHREFS_TOKEN".- Endpoint:
GET https://api.ahrefs.com/v3/site-explorer/refdomainswith headerAuthorization: Bearer {token},select=domain,domain_rating,dofollow_links,linked_domains,first_seen,last_seen. 401→ STOPAUTH_INVALID_AHREFS_TOKEN;403→ STOPAHREFS_FORBIDDEN.
- Endpoint:
- GSC has NO links API. The "Top linking sites" report must be exported by hand and passed as
--gsc-links-csv. Do NOT claim GSC data without the CSV. - IF
--use-semrushANDSEMRUSH_API_KEYset:GET https://api.semrush.com/analytics/v1/?type=backlinks_refdomains(returns;-delimited CSV). Missing key → degrade withsources.semrush="skipped_no_key".
EXPECTED TOOL CALLS
- Run
scripts/backlink_reconciler.py --target example.com --gsc-links-csv top_linking_sites.csv [--use-semrush] [--previous prev.json]. - One Ahrefs refdomains call; optionally one Semrush call; local CSV parse for GSC.
PROCEDURE (deterministic)
STEP 1 — FETCH Ahrefs referring domains (capped at max-domains), collapsing subdomains to the registrable domain and keeping the strongest DR per domain.
STEP 2 — READ the GSC export; detect the host column (first column when unlabeled) and any linking-pages count; normalize hosts to registrable domains.
STEP 3 — OPTIONALLY fetch Semrush referring domains and normalize.
STEP 4 — UNION into an authoritative set; attribute each domain to the sources that reported it and attach the best available attributes (DR, GSC linking pages, first seen).
STEP 5 — PARTITION vs GSC: both, ahrefs_only, gsc_only; compute overlap and each tool's unique share.
STEP 6 — FLAG verification candidates: gsc_only (Google sees a link Ahrefs misses) and high-DR ahrefs_only (authority domain Google's export omits — possibly nofollow/disallowed/uncounted).
STEP 7 — DIFF vs previous.authoritative_domains → gained_domains, lost_domains. First run → status="baseline".
RATE LIMITS & ERROR HANDLING
- Ahrefs
429→ backoff2^attempt(max 5) then STOPRATE_LIMITED;5xxretry ≤3. - Semrush is best-effort: on
429/error the run continues withsources.semrushcarrying the reason (never fabricated). - Registrable-domain mapping uses a curated two-level public-suffix set (heuristic, not the full PSL); it is documented, not silently exact.
MISSING / INSUFFICIENT DATA
- No
--gsc-links-csv→reconciliation="skipped_no_csv"; return the Ahrefs set only, do not invent GSC coverage. - Empty GSC export →
sources.gsc="empty_export". - First run (no
previous) →status="baseline"; no gained/lost computed. - Never merge a host into a domain it does not belong to; when the suffix is unknown the last two labels are used and this is disclosed.
OUTPUT
One JSON object per references/output.schema.json. authoritative_domains MUST be persisted and passed back as the next run's --previous.
FILES
scripts/backlink_reconciler.py— Ahrefs + GSC + Semrush fetch, registrable-domain normalization, reconciliation and diff.references/output.schema.json— output contract.
