See exactly what your robots.txt tells each crawler — which paths are blocked, which rule blocked them, and whether the file is even readable. Then build a correct one. Free, no account.
Pick the pieces that match your site and combine them. Everything is built from fixed templates in your browser — no AI is guessing at your URL structure, and nothing you type here is sent anywhere.
Used for the header comment and the Sitemap: line.
# Generated with TaskGlaze's free robots.txt tool — https://www.taskglaze.com/free-tools/robots-txt-tester User-agent: * Disallow:
Save this as robots.txt in the root of your site, so it answers at https://www.example.com/robots.txt. A robots.txt in a subdirectory is ignored.
This runs entirely in your browser against the text above: syntax, unsupported directives, and rules that contradict each other. It can't see your site, so it can't tell you whether a path you blocked actually matters — re-run the live check at the top once the file is published for that.
robots.txt is the first thing a crawler asks for, and it is the bluntest instrument in technical SEO. A single stray Disallow: / left over from a staging deploy will keep every page of a live site out of search, and nothing on the page itself will look wrong.
It is also widely misunderstood. Disallow controls crawling, not indexing — a blocked URL that other sites link to can still appear in results, stripped of its title and description. And because Google has to read a page to see a noindex, blocking a page in robots.txt is the one reliable way to stop that noindex ever taking effect.
This tester reads the file the way a crawler does: it resolves www against non-www first, follows redirects, parses each group separately, and answers per user-agent rather than pretending every crawler sees the same rules.
Type your site's address — example.com is enough. We resolve the www and non-www versions and follow redirects to find the file a crawler would actually read.
Open “Test specific paths” and paste up to ten URLs, then pick the crawler to test as. Rules are matched per crawler, so Googlebot and Bingbot can get different answers from the same file.
You get the file's rules grouped by crawler, an allowed-or-blocked answer for every path you listed, every sitemap the file declares, and the raw text of the file itself.
Pick from the templates below, combine them, preview the result, and download a robots.txt you can drop straight into your site root.
Most robots.txt damage comes from a handful of misunderstandings. These are them.
No — and this is the single most common robots.txt mistake. Disallow stops a crawler reading the page; it does not remove it from the index. If other sites link to a blocked URL, Google can still list it, just with no description. To actually remove a page you have to let it be crawled and serve a noindex meta tag or X-Robots-Tag header on it, because Google has to read the page to see the noindex.
No. Googlebot ignores Crawl-delay entirely — if Google is crawling your site too hard, change the crawl rate in Search Console or return a 503 while you're overloaded. Bing and Yandex do honour Crawl-delay, so the directive isn't useless, it just does nothing for the crawler most people are worried about.
Google treats a 5xx response on robots.txt as a signal to stop crawling the site entirely, on the assumption the file might be temporarily broken rather than genuinely absent. That's why this tool reports a failed fetch as its own state and never as “everything is allowed” — a server error on that one file can quietly cost you your whole crawl.
Not in itself. A 404 on /robots.txt is a perfectly valid answer and means every crawler may read every page. It's worth publishing one anyway, if only to point crawlers at your sitemap — that's the cheapest crawl-discovery win available.
A crawler uses exactly one group: the one whose User-agent line names it most specifically, falling back to the * group only if nothing names it. Inside that group, Google matches the most specific path rule — the longest matching pattern — and on a genuine tie between an Allow and a Disallow of equal length, Allow wins.
No. It builds the file from fixed, hand-written templates entirely in your browser, and every caveat shown next to a template is stated rather than generated. Nothing you type into the generator is sent to a server.
Three per tool per day, with no account and nothing to sign up for. The counter resets automatically — the page tells you how long you have to wait. Inside TaskGlaze the same checks run as part of the full site audit, with no daily cap.
TaskGlaze audits robots.txt, sitemaps, canonicals, redirects and on-page signals across your whole site — on a schedule, with no daily cap.