How to Fix a Missing robots.txt File (Step-by-Step)
Our audit requests https://yourdomain/robots.txt and flags the site when that URL doesn't return a robots.txt file, usually because of a 404, a 5xx error or a redirect loop. By the end of this guide you'll have a plain-text robots.txt at the root of your domain that allows crawling and points search engines and AI crawlers to your XML sitemap.

Why it matters
Every major crawler, including Googlebot, Bingbot, GPTBot and ClaudeBot, requests /robots.txt before crawling your site. A 404 is treated as "crawl everything", so the cost is that you lose the easiest place to declare your sitemap and to keep crawlers out of cart, search and admin URLs. A 5xx or timeout is worse: Google may pause crawling of the whole site until robots.txt responds again, which can delay indexing of new and updated pages.
What you need
Step-by-step fix
Confirm what /robots.txt currently returns
Request the file directly and look at the status code, not just what the browser shows. A soft 404 (a 200 HTML error page) or a redirect to the homepage also counts as missing. Check both the www and non-www hosts, plus http and https, because each host needs its own robots.txt.
curl -I https://example.com/robots.txt curl -I https://www.example.com/robots.txt
Find your XML sitemap URL
Open the likely sitemap locations and note the one that loads. Common paths are:
/sitemap.xml(Shopify, Wix, many static generators)/sitemap_index.xml(Yoast SEO, Rank Math)/wp-sitemap.xml(WordPress core)
Use the full absolute URL with https in robots.txt.
Write the robots.txt content
Start with a permissive file that allows all crawlers and declares the sitemap. Only add Disallow rules for URLs that should never be crawled, such as internal search results or checkout. Never add Disallow: / on a live site.
User-agent: * Allow: / Disallow: /search Disallow: /cart Sitemap: https://example.com/sitemap.xml
Upload it to the domain root
The file must sit at exactly /robots.txt on each host. Subfolders like /blog/robots.txt are ignored. Name it in lowercase, save it as UTF-8 plain text, and upload it to the web root (often public_html/, /var/www/html/ or your framework's public/ folder).
scp robots.txt user@example.com:/var/www/html/robots.txt
Make sure the server serves it correctly
txt hits a 404 handler. Add an explicit rule so the static file is served with Content-Type: text/plain` and no redirect. On Nginx, add a location block.
On Apache, exclude it from rewrite rules.
location = /robots.txt {
try_files $uri =404;
default_type text/plain;
access_log off;
}Purge caches and verify the live file
Clear your CDN and page cache so a cached 404 isn't served. Then request the file again. You should get 200 and content-type: text/plain, and the body should be your directives.
txt** and request a recrawl. Then re-run the audit.
curl -sS -D - https://example.com/robots.txt -o -
Platform-specific notes
WordPress
WordPress generates a virtual robots.txt only when no physical file exists and permalinks work. If yours 404s, a security plugin or server rule is usually intercepting it. Yoast SEO lets you create or edit the file under **Yoast SEO → Tools → File editor**. Rank Math uses **Rank Math → General Settings → Edit robots.txt**. A physical robots.txt in the web root overrides both.
Shopify
Shopify serves /robots.txt automatically with your sitemap included. You can't upload a file. To customize it, go to **Online Store → Themes → … → Edit code**, add the template `robots.txt.liquid`, and keep the default `{{ robots.default_groups }}` output so you don't drop Shopify's rules.
Next.js
With the App Router, create `app/robots.ts` that exports a function returning `{ rules: { userAgent: '*', allow: '/' }, sitemap: 'https://example.com/sitemap.xml' }`. Next.js serves it at /robots.txt. Alternatively, place a static `robots.txt` in `/public`. Don't use both, or they conflict.
Wix / Squarespace / Webflow
These platforms serve robots.txt automatically. On Wix, edit it under **SEO → Robots.txt Editor**. On Webflow, use **Site settings → SEO → Indexing → Robots.txt**, and note that it only applies after you publish. On Squarespace the file can't be edited. If it's missing, check that the site isn't password-protected or set to hide from search.
Cloudflare
Check **Security → Bots** and **AI Crawl Control**. Cloudflare can serve a managed robots.txt or prepend AI-bot rules. Also confirm no WAF rule or redirect rule is challenging or redirecting /robots.txt, because crawlers can't pass JavaScript challenges.
Check that it worked
✅ How to verify
- Run `curl -I https://example.com/robots.txt` and confirm `HTTP/2 200` with `content-type: text/plain`, and no 301/302 to another URL.
- Open Google Search Console → Settings → robots.txt and check that the file shows as Fetched with no errors for each property/host.
- Use URL Inspection on a key page and confirm "Crawl allowed? Yes".
- Re-run the audit and confirm the check reports "robots.txt found."
⚠️ Common mistakes
- Leaving `Disallow: /` from a staging site in the production robots.txt, which blocks the whole site from crawling.
- Using a relative sitemap path such as `Sitemap: /sitemap.xml`. The Sitemap directive needs a full absolute URL.
- Fixing robots.txt on www but not on the non-www or a subdomain. Each host is checked separately.
- Expecting robots.txt to remove pages from Google. Blocked URLs can still be indexed without content, so use `noindex` for that instead.
Is this still an issue on your site? Run a free audit — ZORO checks this and 294 other SEO, AI-visibility, accessibility and security signals, with fixes for your platform.
Check my site freeFAQ
Is a robots.txt file required for SEO?
It isn't strictly required, since a 404 is treated as permission to crawl everything. It's still best practice: it's where crawlers discover your sitemap, and it lets you keep low-value URLs out of the crawl. A server error on robots.txt, unlike a 404, can actively stop Google from crawling.
Where do I put the robots.txt file?
In the root of each host, so it loads at https://example.com/robots.txt. Files in subdirectories are ignored, and every subdomain needs its own file.
Should I block AI crawlers like GPTBot in robots.txt?
That's a business decision. If you want your content cited in AI answers, allow them, which the default `User-agent: *` / `Allow: /` does. To opt out, add a group such as `User-agent: GPTBot` followed by `Disallow: /`. Note that blocking Google-Extended doesn't affect regular Google Search.
How long until Google sees my new robots.txt?
Google usually caches robots.txt for up to 24 hours. You can speed it up by requesting a recrawl in Search Console under Settings → robots.txt.
How to Fix robots.txt Blocking CSS, JS and Key Pages
How to Fix www vs non-www Duplicate Hostnames with 301s
How to Fix a Missing XML Sitemap (and Get It Found)
How to Fix a Missing HSTS Header (Strict-Transport-Security)
How to Fix Non-200 URLs in Your XML Sitemap
How to Fix SSL Certificate Expiry and Trust Errors