Free tool

robots.txt tester

See which bots may crawl a URL on your site. Free, no signup.

What this tool checks

Tests a URL against your robots.txt for Googlebot, Bingbot, AI crawlers and any user agent you add, and shows the rules that apply.

How to use it

  1. Enter a domain or the full URL of a page.
  2. Click “Check now” and wait a few seconds.
  3. Review the results and fix anything marked fail or warn.
In depth

How robots.txt rules work

Groups and precedence

Rules are grouped by User-agent. A crawler follows the most specific group that names it, falling back to *. Within a group, the longest matching Allow or Disallow path wins.

What robots.txt cannot do

It does not remove pages from search results and it is not a security control — anyone can read it. Use noindex to keep pages out of results and authentication to protect private content.

FAQ

robots.txt tester: FAQ

Is the robots.txt tester free?

Yes — free, with no signup. Fair-use limits apply per hour.

Does robots.txt stop pages from being indexed?

No — it stops crawling. A blocked page can still be indexed from links. Use a noindex tag to keep a page out of search results.

How do I test robots.txt for Googlebot?

Enter the full URL of a page; the tester shows whether Googlebot, Bingbot, AI crawlers and any user agent you add may fetch it, and which rule applies.

Should robots.txt link to my sitemap?

Yes — add a Sitemap: line with the full URL of your XML sitemap so every crawler can find it.

Free 14-day trial · no card

See what’s holding your site back

Run a free audit in minutes: every SEO, AI-visibility, accessibility and security issue, ranked by impact, with the fix for your platform.