Enter a website to check its llms.txt and llms-full.txt against the llms.txt proposal and test the links they list. See which AI crawlers your robots.txt allows, and whether key pages show their text without JavaScript. llms.txt is an optional, emerging convention. Free, no account needed.
llms.txt is a proposal, not a standard, and no AI company is required to read it. The checks that matter most today are simpler: whether AI crawlers are allowed in, and whether your pages have text without JavaScript.
Whether /llms.txt and /llms-full.txt exist and are real text files rather than a web page or error page, plus the file's H1 title, blockquote summary, sections and how many links it lists.
The links in your llms.txt and the status each one answers with. Broken links send AI tools to dead ends; links that redirect should point at their final address.
For each AI crawler, whether robots.txt allows it across the site and for the URL you entered, and whether your file names it or it falls under the general * rules. Google-Extended and Applebot-Extended are control tokens, not crawlers.
How many words the URL you entered, your home page and up to two pages from llms.txt show before any script runs. Many AI crawlers don't run JavaScript, so a page that only fills in with scripts looks empty to them.
We check the site's root for /llms.txt and /llms-full.txt, so any page on the site works.
Against the proposal: one H1 with the site's name, an optional blockquote summary, optional notes, then H2 sections listing links as [title](url): notes.
Up to 20 links from llms.txt are requested to confirm they load. A larger file shows how many were checked.
robots.txt is tested for 14 AI crawlers and control tokens, and key pages are read without JavaScript to count the words they show.
The TaskGlaze site audit checks for llms.txt and flags pages whose content only appears with JavaScript, across your whole site. See the site audit.
AI assistants and AI search read the web much like other crawlers. The basics matter far more than any single file.
Training crawlers such as GPTBot, ClaudeBot and CCBot collect pages to train models. Search and user crawlers such as OAI-SearchBot and PerplexityBot fetch pages to answer questions, often with a link back. You can allow one kind and block the other in robots.txt.
Many AI crawlers don't run JavaScript. Server-side rendering or static generation makes sure your words are in the page they download.
Start the file with "# Your site name", add a one-paragraph summary as a "> " blockquote, then H2 sections listing your most useful pages as "- [Title](https://…): what it covers".
The proposal suggests linking to Markdown versions of pages (page.md) where you have them, because they are easier for language models to read than full HTML.
Put secondary links under "## Optional". The proposal treats that section as safe to skip when a tool has limited room.
Point to your key pages, not every page. Put full text in llms-full.txt or linked Markdown files.
Update llms.txt when pages move. A broken or redirecting link is a dead end for any tool that follows it.
Clear headings, plain explanations and structured data help AI systems quote you accurately, with or without llms.txt.
Each name below is matched against User-agent lines in robots.txt. Crawlers not named fall under the * group.
| User-agent | Run by | What it does |
|---|---|---|
| GPTBot | OpenAI | Collects pages for model training |
| OAI-SearchBot | OpenAI | Indexes pages for search answers |
| ChatGPT-User | OpenAI | Fetches a page when a user asks about it |
| ClaudeBot | Anthropic | Collects pages for model training |
| Claude-SearchBot | Anthropic | Indexes pages for search answers |
| Claude-User | Anthropic | Fetches a page when a user asks about it |
| Google-Extended | Control token for AI training; doesn't affect Search | |
| PerplexityBot | Perplexity | Indexes pages for answers |
| Perplexity-User | Perplexity | Fetches a page when a user asks about it |
| CCBot | Common Crawl | Open web archive used to train many models |
| Applebot-Extended | Apple | Control token for AI training |
| Bytespider, Amazonbot, Meta-ExternalAgent | ByteDance, Amazon, Meta | Crawlers used for AI products and training |
A catch-all route returns your home page or a styled 404 for /llms.txt with status 200. Tools get HTML instead of the text file.
The proposal's one required element is the "# Site name" first line.
Bullet points that describe pages without linking them give tools nowhere to go.
A broad Disallow under * also blocks AI crawlers, and some security plugins add blanket AI blocks. Check that the rules match what you intended.
Blocking OAI-SearchBot or PerplexityBot can keep you out of AI answers that cite and link sources.
A single-page app can look empty to crawlers that don't run scripts, whatever llms.txt says.
It is optional and new. A missing file costs nothing; broken pages and blocked crawlers cost far more.
Pages move and the file still points to old URLs. Re-check it after site changes.
llms.txt, AI crawlers and rendering, explained honestly.
llms.txt is a proposed plain-text Markdown file at the root of a website, /llms.txt, that gives AI tools a short summary of the site and links to its most useful pages. It is an optional, emerging convention, not an official standard.
Support varies and is still developing. Some tools and documentation platforms use it, and no AI company is required to. Treat it as a helpful extra, not a ranking factor.
No. A missing llms.txt isn't an error. Allowing the crawlers you want and serving your text in the HTML matter more for how AI tools see your site.
A first line "# Site name", then an optional "> " summary, optional notes, and H2 sections such as "## Docs" listing links as "- [Title](https://example.com/page.md): short description". A section called "## Optional" holds links that can be skipped.
An optional companion file with the full text of the pages llms.txt links to, in one document, so a tool can read everything in a single request.
Add a group to robots.txt, for example "User-agent: GPTBot" followed by "Disallow: /" to block it or "Allow: /" to allow it. The report shows the result for each crawler.
They are control tokens, not separate crawlers. Disallowing them tells Google or Apple not to use your content for AI training, without affecting normal search crawling.
It is a business decision. Blocking training crawlers keeps your content out of future training data; blocking search and user crawlers can also keep you out of AI answers that link to sources. The report shows the current rules without judging them.
Many AI crawlers download the HTML and don't run scripts. If your text only appears after JavaScript runs, those crawlers see an almost empty page.
No. robots.txt is a request that well-behaved crawlers follow. To enforce a block you need server or firewall rules.
Three per tool per day without an account, or fifteen per tool per day when you are signed in to your TaskGlaze account. Each free tool has its own count.
We check every link against public lists of phishing and scam sites before loading it, and refuse ones that look unsafe. A refused link doesn't use one of your daily checks.
More free checks from TaskGlaze, with the same daily allowance for each tool.
The TaskGlaze site audit checks for llms.txt and flags pages that depend on JavaScript, alongside every other technical check.