robots.txt tester
See which bots may crawl a URL on your site. Free, no signup.
What this tool checks
Tests a URL against your robots.txt for Googlebot, Bingbot, AI crawlers and any user agent you add, and shows the rules that apply.
How to use it
- Enter a domain or the full URL of a page.
- Click “Check now” and wait a few seconds.
- Review the results and fix anything marked fail or warn.
How robots.txt rules work
Groups and precedence
Rules are grouped by User-agent. A crawler follows the most specific group that names it, falling back to *. Within a group, the longest matching Allow or Disallow path wins.
What robots.txt cannot do
It does not remove pages from search results and it is not a security control — anyone can read it. Use noindex to keep pages out of results and authentication to protect private content.
robots.txt tester: FAQ
Is the robots.txt tester free?
Yes — free, with no signup. Fair-use limits apply per hour.
Does robots.txt stop pages from being indexed?
No — it stops crawling. A blocked page can still be indexed from links. Use a noindex tag to keep a page out of search results.
How do I test robots.txt for Googlebot?
Enter the full URL of a page; the tester shows whether Googlebot, Bingbot, AI crawlers and any user agent you add may fetch it, and which rule applies.
Should robots.txt link to my sitemap?
Yes — add a Sitemap: line with the full URL of your XML sitemap so every crawler can find it.
More free tools
AI crawler access checker
Can ChatGPT, Claude, Perplexity and Gemini read your site?
llms.txt generator
Create an llms.txt file from your site in seconds.
llms.txt validator
Check that your /llms.txt exists and is well-formed.
Meta tags & Google snippet preview
Check titles, descriptions, Open Graph and preview your search result.
Schema markup checker
Find and check the structured data (JSON-LD) on a page.
Schema generator
Generate Organization, LocalBusiness or FAQ JSON-LD.
Security headers grade
Grade HTTPS and security headers (HSTS, CSP, X-Frame-Options…).
Redirect checker
Follow every redirect hop and its status code.
XML sitemap validator
Find your sitemap, count URLs and spot broken entries.
Broken link checker
Find broken links on any page.
Keyword density checker
See which words and phrases a page uses most.
See what’s holding your site back
Run a free audit in minutes: every SEO, AI-visibility, accessibility and security issue, ranked by impact, with the fix for your platform.