How to Fix a Missing robots.txt File (Step-by-Step)

Our audit requests https://yourdomain/robots.txt and flags the site when that URL doesn't return a robots.txt file, usually because of a 404, a 5xx error or a redirect loop. By the end of this guide you'll have a plain-text robots.txt at the root of your domain that allows crawling and points search engines and AI crawlers to your XML sitemap.

By Updated ⏱ 10 minutes🎯 Beginner● High impactTechnical
Illustration: robots.txt present fix guide
TECHNICAL · FIX GUIDE robots.txt present IMPACTEFFORT TIME10 minutes LEVELBeginner

Why it matters

Every major crawler, including Googlebot, Bingbot, GPTBot and ClaudeBot, requests /robots.txt before crawling your site. A 404 is treated as "crawl everything", so the cost is that you lose the easiest place to declare your sitemap and to keep crawlers out of cart, search and admin URLs. A 5xx or timeout is worse: Google may pause crawling of the whole site until robots.txt responds again, which can delay indexing of new and updated pages.

✕Beforehttps://example.com/robots.txt → 404Not Found (or 500 / redirect to thehomepage)✓Afterhttps://example.com/robots.txt → 200OK, text/plain, containing User-agent,Allow and Sitemap lines

What you need

Access to your web server root, CMS admin, or code repository
The URL of your XML sitemap (e.g. https://example.com/sitemap.xml)
A browser or terminal with curl to test the result

Step-by-step fix

1

Confirm what /robots.txt currently returns

Request the file directly and look at the status code, not just what the browser shows. A soft 404 (a 200 HTML error page) or a redirect to the homepage also counts as missing. Check both the www and non-www hosts, plus http and https, because each host needs its own robots.txt.

text
curl -I https://example.com/robots.txt
curl -I https://www.example.com/robots.txt
Pro tip: If you see a 5xx or a timeout, fix that first. Server errors on robots.txt can make Google slow down crawling the entire site.
2

Find your XML sitemap URL

Open the likely sitemap locations and note the one that loads. Common paths are:

  • /sitemap.xml (Shopify, Wix, many static generators)
  • /sitemap_index.xml (Yoast SEO, Rank Math)
  • /wp-sitemap.xml (WordPress core)

Use the full absolute URL with https in robots.txt.

Pro tip: If you have several sitemaps, either list each one on its own Sitemap line or reference the sitemap index file.
3

Write the robots.txt content

Start with a permissive file that allows all crawlers and declares the sitemap. Only add Disallow rules for URLs that should never be crawled, such as internal search results or checkout. Never add Disallow: / on a live site.

text
User-agent: *
Allow: /
Disallow: /search
Disallow: /cart

Sitemap: https://example.com/sitemap.xml
Pro tip: Don't block CSS, JS or image folders. Google needs them to render your pages.
4

Upload it to the domain root

The file must sit at exactly /robots.txt on each host. Subfolders like /blog/robots.txt are ignored. Name it in lowercase, save it as UTF-8 plain text, and upload it to the web root (often public_html/, /var/www/html/ or your framework's public/ folder).

text
scp robots.txt user@example.com:/var/www/html/robots.txt
Pro tip: Keep the file under 500 KiB. Google ignores anything past that limit.
5

Make sure the server serves it correctly

txt hits a 404 handler. Add an explicit rule so the static file is served with Content-Type: text/plain` and no redirect. On Nginx, add a location block.

On Apache, exclude it from rewrite rules.

nginx
location = /robots.txt {
    try_files $uri =404;
    default_type text/plain;
    access_log off;
}
Pro tip: For Apache, put `RewriteRule ^robots\.txt$ - [L]` above your front-controller rewrite in .htaccess.
6

Purge caches and verify the live file

Clear your CDN and page cache so a cached 404 isn't served. Then request the file again. You should get 200 and content-type: text/plain, and the body should be your directives.

txt** and request a recrawl. Then re-run the audit.

text
curl -sS -D - https://example.com/robots.txt -o -

Platform-specific notes

WordPress

WordPress generates a virtual robots.txt only when no physical file exists and permalinks work. If yours 404s, a security plugin or server rule is usually intercepting it. Yoast SEO lets you create or edit the file under **Yoast SEO → Tools → File editor**. Rank Math uses **Rank Math → General Settings → Edit robots.txt**. A physical robots.txt in the web root overrides both.

Shopify

Shopify serves /robots.txt automatically with your sitemap included. You can't upload a file. To customize it, go to **Online Store → Themes → … → Edit code**, add the template `robots.txt.liquid`, and keep the default `{{ robots.default_groups }}` output so you don't drop Shopify's rules.

Next.js

With the App Router, create `app/robots.ts` that exports a function returning `{ rules: { userAgent: '*', allow: '/' }, sitemap: 'https://example.com/sitemap.xml' }`. Next.js serves it at /robots.txt. Alternatively, place a static `robots.txt` in `/public`. Don't use both, or they conflict.

Wix / Squarespace / Webflow

These platforms serve robots.txt automatically. On Wix, edit it under **SEO → Robots.txt Editor**. On Webflow, use **Site settings → SEO → Indexing → Robots.txt**, and note that it only applies after you publish. On Squarespace the file can't be edited. If it's missing, check that the site isn't password-protected or set to hide from search.

Cloudflare

Check **Security → Bots** and **AI Crawl Control**. Cloudflare can serve a managed robots.txt or prepend AI-bot rules. Also confirm no WAF rule or redirect rule is challenging or redirecting /robots.txt, because crawlers can't pass JavaScript challenges.

Check that it worked

✅ How to verify

  • Run `curl -I https://example.com/robots.txt` and confirm `HTTP/2 200` with `content-type: text/plain`, and no 301/302 to another URL.
  • Open Google Search Console → Settings → robots.txt and check that the file shows as Fetched with no errors for each property/host.
  • Use URL Inspection on a key page and confirm "Crawl allowed? Yes".
  • Re-run the audit and confirm the check reports "robots.txt found."

⚠️ Common mistakes

  • Leaving `Disallow: /` from a staging site in the production robots.txt, which blocks the whole site from crawling.
  • Using a relative sitemap path such as `Sitemap: /sitemap.xml`. The Sitemap directive needs a full absolute URL.
  • Fixing robots.txt on www but not on the non-www or a subdomain. Each host is checked separately.
  • Expecting robots.txt to remove pages from Google. Blocked URLs can still be indexed without content, so use `noindex` for that instead.

Is this still an issue on your site? Run a free audit — ZORO checks this and 294 other SEO, AI-visibility, accessibility and security signals, with fixes for your platform.

Check my site free

FAQ

Is a robots.txt file required for SEO?

It isn't strictly required, since a 404 is treated as permission to crawl everything. It's still best practice: it's where crawlers discover your sitemap, and it lets you keep low-value URLs out of the crawl. A server error on robots.txt, unlike a 404, can actively stop Google from crawling.

Where do I put the robots.txt file?

In the root of each host, so it loads at https://example.com/robots.txt. Files in subdirectories are ignored, and every subdomain needs its own file.

Should I block AI crawlers like GPTBot in robots.txt?

That's a business decision. If you want your content cited in AI answers, allow them, which the default `User-agent: *` / `Allow: /` does. To opt out, add a group such as `User-agent: GPTBot` followed by `Disallow: /`. Note that blocking Google-Extended doesn't affect regular Google Search.

How long until Google sees my new robots.txt?

Google usually caches robots.txt for up to 24 hours. You can speed it up by requesting a recrawl in Search Console under Settings → robots.txt.