How to Fix AI Crawler 5xx Errors in Your Server Logs
Your site returned a 5xx server error (500, 502, 503 or 504) when requested with an AI crawler user agent such as GPTBot, ClaudeBot or PerplexityBot. By the end of this guide you will have reproduced the error, traced it to the layer causing it (rate limiter, firewall, CDN or application), fixed it, and confirmed that AI crawlers now get a 200 or a deliberate block code instead of a server error.

Why it matters
AI crawlers treat a 5xx as a server problem, not a policy decision. They retry, slow their crawl rate, and keep stale or missing copies of your pages, so ChatGPT search, Claude and Perplexity are less likely to cite current content from you. Repeated 5xx responses also waste server resources on retries and hide real outages in your monitoring.
txt rule communicates that clearly; a 5xx just looks like your site is broken.
What you need
Step-by-step fix
Reproduce the error with AI user agents
Request your homepage and one or two key pages with each AI crawler user agent and print only the status code. Compare against a normal browser user agent. Note which bots and which URLs return 5xx; this tells you whether the problem is UA-specific or affects everyone.
for ua in "GPTBot/1.1; +https://openai.com/gptbot" "OAI-SearchBot/1.0; +https://openai.com/searchbot" "ChatGPT-User/1.0; +https://openai.com/bot" "ClaudeBot/1.0; +claudebot@anthropic.com" "PerplexityBot/1.0; +https://perplexity.ai/perplexitybot" "CCBot/2.0; +https://commoncrawl.org/faq/"; do
code=$(curl -s -o /dev/null -w "%{http_code}" -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; $ua)" https://example.com/)
echo "$code $ua"
done
curl -s -o /dev/null -w "%{http_code} browser\n" -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/130.0 Safari/537.36" https://example.com/Find real AI crawler 5xx hits in your access logs
A one-off test can miss intermittent errors, so check what real crawlers received. The command below works on the default Nginx/Apache combined log format, where field 9 is the status code. It counts 5xx responses per status and URL for the main AI bots.
grep -E "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|PerplexityBot|Perplexity-User|CCBot|Bytespider|Applebot" /var/log/nginx/access.log* \
| awk '$9 ~ /^5/ {print $9, $7}' | sort | uniq -c | sort -rn | head -30Identify which layer returns the 5xx
Match the failing requests by timestamp to your error log (for example /var/log/nginx/error.log or your PHP-FPM log). Typical causes:
- 503 from a rate limiter: Nginx limit_req returns 503 by default when bots crawl fast.
- 503/5xx from a WAF or bot shield: some hosts and security plugins serve a 503 challenge page to non-browser agents.
- 500 from the application: code that crashes when cookies, Accept-Language or JavaScript are missing.
- 502/504 from timeouts: slow server-side rendering on uncached pages bots request.
grep -E "limiting requests|upstream timed out|PHP Fatal" /var/log/nginx/error.log | tail -50
Return 429 instead of 503 from rate limits
If a rate limiter is the cause, raise the limit for legitimate crawling and change the rejection code to 429 Too Many Requests, which crawlers understand as "slow down" rather than "server failure". In Nginx, set limit_req_status. Give bursts enough room for a crawler fetching a page plus its assets.
limit_req_zone $binary_remote_addr zone=perip:10m rate=10r/s;
server {
location / {
limit_req zone=perip burst=40 nodelay;
limit_req_status 429;
}
}Fix firewall and bot-protection rules
Open your WAF or bot-management settings and find rules matching AI user agents or "unknown bots". For bots you want to allow, add an exception (skip/allow rule) for verified crawlers. txt), never a 5xx challenge page.
Also check country blocks and JavaScript challenges, since crawlers cannot solve them.
# robots.txt - deliberate, clear opt-out instead of a server error User-agent: CCBot Disallow: / User-agent: GPTBot Allow: /
Fix application errors and timeouts
If the error log shows a 500 or a timeout, reproduce it with a bare request: no cookies, no Accept-Language, no JavaScript. Fix code that assumes a session, a geo cookie or a browser-only header exists, and make sure pages are served from cache so a crawler hitting an uncached URL does not trigger a slow render. Raise upstream timeouts only after fixing the slow query or render.
curl -sv -H "Accept-Language:" -H "Cookie:" -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.1; +https://openai.com/gptbot)" https://example.com/products/ 2>&1 | head -40
Re-test and monitor
Run the curl loop from step 1 again; every bot you allow should return 200 and every bot you block should return 403. Then re-run the log query from step 2 after 24-72 hours to confirm real crawlers are no longer receiving 5xx. Finally, run the audit again so the check passes.
Platform-specific notes
Cloudflare
Check Security > Events filtered by user agent to see which rule fired. Review Security > Bots and AI Crawl Control settings plus custom rules under Security > WAF. Blocks there normally return 403; 5xx usually comes from your origin (Cloudflare 520-526 codes mean the origin failed or refused the connection). Allow verified bots with a custom rule using cf.client.bot or the verified bot category.
WordPress
Security plugins such as Wordfence (Firewall > Rate Limiting), Solid Security or host-level bot protection on managed hosts can throttle or block crawlers. Set rate-limit actions to throttle rather than block, add AI crawlers to allowlists if desired, and make sure a page cache (WP Rocket, LiteSpeed Cache or host cache) serves bots so PHP is not hit on every request.
Shopify
You cannot change server or rate-limit settings. 5xx for AI bots is rare and usually comes from a third-party app proxy or a theme app embed timing out. Disable recently added apps to isolate it, use robots.txt.liquid for intentional blocks, and contact Shopify Support with the timestamps and user agents if errors continue.
Next.js on Vercel
504s often mean a server-rendered route exceeded the function timeout on an uncached request. Use static generation or ISR (revalidate) for content pages. Check Vercel Firewall and Bot Protection rules under the project's Firewall tab, and middleware that redirects or throws when cookies or geo headers are missing.
Apache
Modules like mod_evasive or mod_security can trigger on fast crawling. Check the mod_security audit log for the rule ID, add a targeted exception (SecRuleRemoveById scoped to the affected location) rather than disabling the module, and raise DOSPageCount/DOSSiteCount thresholds if mod_evasive is involved.
Check that it worked
✅ How to verify
- Run the curl loop with GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot user agents and confirm each returns 200 (or 403 for bots you intentionally block), never 5xx.
- Re-run the access log query after 24-72 hours and confirm zero 5xx lines for AI crawler user agents.
- Check your CDN or firewall event log for the same period to confirm no rules are serving challenge or error pages to allowed bots.
- Re-run the audit and confirm the AI crawler errors check passes.
⚠️ Common mistakes
- Using a 503 or challenge page to block AI bots instead of robots.txt or a 403, which makes crawlers retry and treats your site as unstable.
- Allowing all traffic with an AI user agent string without verifying IP ranges, which lets scrapers spoofing GPTBot bypass your protections.
- Testing only the homepage, while 5xx errors happen on uncached product, search or paginated URLs that crawlers actually fetch.
- Raising server timeouts to hide a slow page instead of caching it, which keeps the server under load when crawlers arrive in bursts.
Is this still an issue on your site? Run a free audit — ZORO checks this and 294 other SEO, AI-visibility, accessibility and security signals, with fixes for your platform.
Check my site freeFAQ
Should I return 403 or 503 to block AI crawlers?
Use robots.txt as the primary signal and a 403 if you also enforce it at the server or CDN. A 503 means "temporarily unavailable", so crawlers keep retrying and may treat your whole site as unreliable.
Why does GPTBot get a 503 when my site works in a browser?
Usually a rate limiter (Nginx limit_req returns 503 by default), a WAF or host bot shield serving a challenge page, or application code that fails without cookies or browser headers. Compare response headers between a browser request and a GPTBot request to see which layer answered.
Do 5xx errors for AI bots affect whether my site appears in ChatGPT or Perplexity answers?
Yes. If OAI-SearchBot or PerplexityBot cannot fetch a page, those tools cannot index or cite its current content, and repeated errors reduce how often they come back to crawl.
How do I tell a real AI crawler from a fake one in my logs?
Check the request IP against the ranges each provider publishes (OpenAI, Anthropic and Perplexity all list theirs), or use your CDN's verified bot flag. A matching user agent from an unlisted IP is a spoof.
How to Fix Key Content Only in PDFs: Publish HTML Versions
How to Get a Google Knowledge Graph Entity for Your Brand
How to Fix Conversational Long-Tail Coverage for AI Search
How to Add Customer Q&A Sections to Product Pages
How to Fix Brand Narrative Consistency for AI Search
How to Fix Missing or Vague Meta Descriptions for AI Answers