360 Secure SkillHub
360 安全技能中心

llms-txt-check

通过脚本对目标网站做只读检查：判断 /llms.txt 是否存在、格式是否符合 llmstxt.org、体积与链接是否可用，并查看 robots.txt 是否允许 GPTBot、ClaudeBot、PerplexityBot 等 AI 爬虫访问，最后输出问题清单与修复建议，不写入或修改任何文件。

未知作者
热度 0.00
最近更新 2026-10-18 21:39:41
分类：营销依赖：代码执行应用场景：报告撰写 / 研究

安装技能

复制下面的话术发给你的 Agent，它会通过 secure-skills 检索并安装该技能。

执行命令安装此 skill： npx secure-skills-cli add seoskills.sh/llms-txt-check
Metadata
name
llms-txt-check
description
Check whether a website has a valid /llms.txt (llmstxt.org format, working links, size) and whether robots.txt lets AI crawlers such as GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended in. Use when the user asks about llms.txt, AI crawler access, blocking or allowing AI bots, or whether ChatGPT/Claude/Perplexity can read their site. Read-only.
license
MIT

llms.txt and AI crawler access check

A read-only check of two files: /llms.txt and /robots.txt. It reports problems and does not write or change anything.

When to use

  • "Do I have a valid llms.txt?", "is my site blocking ChatGPT / Claude / Perplexity?", "can AI crawlers read my site?"

How to run

python3 scripts/llms_check.py example.com
python3 scripts/llms_check.py https://docs.example.com --max-links 40

Needs outbound HTTPS. Makes about 25 requests at most (it checks up to --max-links links from llms.txt, with a short pause between them).

What it checks

  • /llms.txt returns 200 and is text, not an HTML soft-404 page.
  • Format per llmstxt.org: one # H1, a > summary, ## sections, Markdown links.
  • Size under ~50 KB, so an agent can load it.
  • Up to N links resolve (HEAD, falling back to GET).
  • /llms-full.txt presence (optional).
  • robots.txt access to / for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended and CCBot.

Read the result honestly

AI search crawlers rarely fetch llms.txt today; coding agents and IDE assistants do. For appearing in AI answers, robots.txt access and server-rendered content matter more. Say this to the user rather than overselling llms.txt.

Example

python3 scripts/llms_check.py plausible.io → llms.txt missing (404), all 8 AI crawlers allowed, sitemap declared → "Publish /llms.txt" as the only fix.

Want the files written for you?