How to Check Googlebot Activity in Your Server Logs
This check reports Not Applicable until someone reviews raw server access logs, because a page-level crawler cannot see which bots actually request your URLs. By the end of this guide you will have exported your access logs, separated genuine Googlebot traffic from fakes, measured which sections Google crawls and which it ignores, and fixed the biggest sources of wasted crawling.

Why it matters
Server logs are the only first-hand record of what Googlebot actually fetches. If your product, service or article sections get few bot hits while filter URLs, old redirects or 404s get most of them, new and updated pages are discovered and refreshed slowly, which delays rankings. The same logs show whether AI crawlers such as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot can reach your content, which affects whether it appears in AI answers.
Logs also expose scrapers that spoof the Googlebot user agent to bypass rate limits. The impact is low for small sites but grows with site size.
What you need
Step-by-step fix
Export 14-30 days of raw access logs
Get the unfiltered access log, not an analytics report; JavaScript-based analytics never records bots. Typical locations:
- Nginx:
/var/log/nginx/access.logplus rotatedaccess.log.*.gz - Apache:
/var/log/apache2/access.log(Debian/Ubuntu) or/var/log/httpd/access_log(RHEL) - cPanel: Metrics > Raw Access, download the archive per domain
If a CDN sits in front of your server, the origin log may miss cached hits, so export from the CDN instead. Combine the files into one working file.
mkdir ~/logreview && cd ~/logreview cp /var/log/nginx/access.log . zcat /var/log/nginx/access.log.*.gz > older.log cat older.log access.log > all.log wc -l all.log
Filter requests by bot user agent
Pull every line whose user agent claims to be Googlebot into its own file, then do the same for the AI crawlers you care about. Googlebot Smartphone should dominate; Google indexes mobile-first, so a large desktop share is unusual. Count each bot to see the overall mix.
grep -i 'googlebot' all.log > googlebot.log grep -ci 'googlebot' all.log grep -i 'googlebot' all.log | grep -ci 'android' # Googlebot Smartphone grep -Eio 'GPTBot|OAI-SearchBot|ClaudeBot|PerplexityBot|bingbot|Applebot' all.log | sort | uniq -c | sort -rn
Verify the hits are real Googlebot
Anyone can fake the Googlebot user agent. com**, and a forward lookup of that host must return the same IP. json`.
log before analysing.
awk '{print $1}' googlebot.log | sort | uniq -c | sort -rn | head -20
host 66.249.66.1
# 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
host crawl-66-249-66-1.googlebot.com
# crawl-66-249-66-1.googlebot.com has address 66.249.66.1
grep -v '203.0.113.45' googlebot.log > googlebot-verified.logBreak crawls down by section and status code
In the default combined log format, field 7 is the requested path and field 9 is the HTTP status. Count hits per top-level directory, per status code, and for URLs with query strings. This shows where Google spends its crawling and how much of it lands on errors or redirects.
# Hits per top-level section
awk '{split($7,a,"/"); print "/"a[2]}' googlebot-verified.log | cut -d'?' -f1 | sort | uniq -c | sort -rn | head -20
# Status code mix
awk '{print $9}' googlebot-verified.log | sort | uniq -c | sort -rn
# Parameter URLs
awk '$7 ~ /\?/' googlebot-verified.log | wc -l
awk '$7 ~ /\?/ {split($7,a,"?"); print a[2]}' googlebot-verified.log | cut -d'=' -f1 | sort | uniq -c | sort -rn | head
# Most-crawled 404s
awk '$9==404 {print $7}' googlebot-verified.log | sort | uniq -c | sort -rn | head -20Compare crawl share against your priority sections
Put the section counts next to your list of important sections and look for these patterns:
- A priority section such as /products/ gets a small share, or key URLs never appear at all
- Parameter, internal-search or calendar URLs take a large share
- More than 5-10% of hits return 301/302 or 404/410
- 5xx responses or slow responses, which make Google crawl less
List the worst offenders with example URLs; these drive the fixes in the next step.
Cut crawl waste and strengthen priority paths
Fix what the logs revealed:
- Block infinite URL spaces (internal search, sort and filter parameters) in robots.txt
- Update internal links that point to redirected URLs so they link to the final URL directly
- Return 410 or redirect old 404s that still get crawled, and remove them from sitemaps
- Link priority sections from the main navigation and list them in your XML sitemap with accurate
lastmoddates - Fix recurring 5xx errors and slow endpoints
Only block URLs you do not want indexed; robots.txt stops crawling, not indexing of already-linked URLs.
User-agent: * Disallow: /search Disallow: /*?sort= Disallow: /*?filter= Disallow: /*&sessionid= Sitemap: https://www.example.com/sitemap.xml
Re-run the analysis and cross-check Search Console
Wait 2-4 weeks, export fresh logs and repeat steps 2-5 with the same commands so the numbers are comparable. Then open Google Search Console > Settings > Crawl stats > Open report and compare total requests, the By response breakdown and By file type with your log figures. Save the commands as a script and schedule a monthly review.
Platform-specific notes
WordPress (managed hosting)
Logs live with the host, not WordPress. Kinsta: MyKinsta > Sites > your site > Logs (access.log). WP Engine: User Portal > your environment > Logs. cPanel hosts: Metrics > Raw Access. Some hosts keep only a few days, so download regularly.
Shopify
Shopify does not give merchants raw server logs. Use Search Console > Settings > Crawl stats instead, and control crawl waste through robots.txt.liquid (Online Store > Themes > Edit code > Add a new template > robots). Shopify's default rules already block /search and many filter parameters.
Cloudflare
If Cloudflare proxies your site, the origin only sees cache misses. Raw request logs come from Logpush or Log Explorer, which require paid or Enterprise plans. On lower plans, use Security > Analytics or Security > Bots to see verified-bot traffic, and Radar's verified bot list to confirm Googlebot.
Vercel / Netlify
Built-in runtime logs are short-lived and focus on functions. For full request logs, configure Log Drains (Vercel: Team Settings > Log Drains; Netlify: Site configuration > Log drains, Enterprise) to a service such as Datadog or an S3 bucket, then run the same analysis on the exported data.
Wix / Squarespace
No server log access is available. Rely on Search Console Crawl stats and the URL Inspection tool, and keep your sitemap and internal navigation focused on priority pages.
Check that it worked
✅ How to verify
- Verified Googlebot hits to your priority sections make up a larger share than in the first review, and key money pages appear in the logs at least weekly.
- Requests to blocked parameter and search URLs drop sharply in the new log export.
- 3xx and 4xx responses are under roughly 5% of Googlebot requests, and 5xx is near zero.
- Search Console Crawl stats shows a matching trend in By response and a stable or rising total crawl requests figure.
⚠️ Common mistakes
- Trusting the user agent alone: scrapers commonly fake Googlebot, so unverified counts can be badly inflated.
- Analysing origin logs behind a CDN and concluding Google barely crawls the site, when most bot hits were served from cache and never reached the origin.
- Using robots.txt to block pages that carry a noindex tag, which stops Google from ever seeing the noindex and keeps them in the index.
- Reviewing only a day or two of data; crawl patterns vary widely day to day, so use at least 14 days.
Is this still an issue on your site? Run a free audit — ZORO checks this and 294 other SEO, AI-visibility, accessibility and security signals, with fixes for your platform.
Check my site freeFAQ
How do I know if Googlebot is crawling my website?
Search your access logs for the Googlebot user agent and verify the IPs with a reverse and forward DNS lookup ending in googlebot.com or google.com. Without log access, Search Console > Settings > Crawl stats shows total Google requests per day and example URLs.
How often should Googlebot crawl my pages?
There is no fixed rate. Homepages and frequently updated sections are often crawled daily; deep pages may be crawled every few weeks. Worry when important pages get no hits over 30 days or when most crawls go to low-value URLs.
Can I see AI crawlers like GPTBot in my server logs?
Yes. GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and others identify themselves in the user agent. Grep for their names to see what they fetch, and confirm against the IP ranges each company publishes before allowing or blocking them in robots.txt.
Do I need log file analysis for a small website?
For sites under a few thousand URLs, crawl budget is rarely a limit, and Search Console Crawl stats is usually enough. Log analysis pays off for large catalogues, sites with faceted navigation, frequent migrations, or unexplained indexing delays.
How to Fix Low Text Contrast and Unlabeled Form Fields (WCAG)
How to Fix Infinite Scroll Crawlability with Paginated URLs
How to Fix Excessive DOM Size: Trim Pages Under 1,500 Nodes
How to Fix Underscores in URLs: Switch to Hyphens Safely
How to Fix JavaScript Console Errors on Your Website
How to Fix Uppercase Characters in URLs (and Redirect Them)