How to Check Googlebot Activity in Your Server Logs

This check reports Not Applicable until someone reviews raw server access logs, because a page-level crawler cannot see which bots actually request your URLs. By the end of this guide you will have exported your access logs, separated genuine Googlebot traffic from fakes, measured which sections Google crawls and which it ignores, and fixed the biggest sources of wasted crawling.

By Updated ⏱ 2-3 hours for the first review, 30 minutes for repeat checks🎯 Advanced● Low impactTechnical
Illustration: Bot activity in server logs fix guide
TECHNICAL · FIX GUIDE Bot activity in server logs IMPACTEFFORT TIME2-3 hours for the first review, 30 minutes for repeat checks LEVELAdvanced

Why it matters

Server logs are the only first-hand record of what Googlebot actually fetches. If your product, service or article sections get few bot hits while filter URLs, old redirects or 404s get most of them, new and updated pages are discovered and refreshed slowly, which delays rankings. The same logs show whether AI crawlers such as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot can reach your content, which affects whether it appears in AI answers.

Logs also expose scrapers that spoof the Googlebot user agent to bypass rate limits. The impact is low for small sites but grows with site size.

✕BeforeNo one has looked at the logs: 62% ofGooglebot requests hit /search?q= and?sort= URLs, /products/ gets 4% of✓AfterVerified Googlebot hits are reviewedmonthly; /products/ and /blog/ receivemost crawls, parameter URLs are

What you need

Access to raw HTTP access logs (SSH, hosting panel, or CDN log export) covering at least 14-30 days
A terminal with grep, awk, sort and host (macOS, Linux, or WSL on Windows), or a log analyzer such as Screaming Frog Log File Analyser
A list of your priority sections, e.g. /products/, /services/, /blog/
Google Search Console access for cross-checking Crawl stats

Step-by-step fix

1

Export 14-30 days of raw access logs

Get the unfiltered access log, not an analytics report; JavaScript-based analytics never records bots. Typical locations:

  • Nginx: /var/log/nginx/access.log plus rotated access.log.*.gz
  • Apache: /var/log/apache2/access.log (Debian/Ubuntu) or /var/log/httpd/access_log (RHEL)
  • cPanel: Metrics > Raw Access, download the archive per domain

If a CDN sits in front of your server, the origin log may miss cached hits, so export from the CDN instead. Combine the files into one working file.

text
mkdir ~/logreview && cd ~/logreview
cp /var/log/nginx/access.log .
zcat /var/log/nginx/access.log.*.gz > older.log
cat older.log access.log > all.log
wc -l all.log
Pro tip: Retention is often only 7-14 days by default. Raise it (e.g. `rotate 30` in /etc/logrotate.d/nginx) before you need the data.
2

Filter requests by bot user agent

Pull every line whose user agent claims to be Googlebot into its own file, then do the same for the AI crawlers you care about. Googlebot Smartphone should dominate; Google indexes mobile-first, so a large desktop share is unusual. Count each bot to see the overall mix.

text
grep -i 'googlebot' all.log > googlebot.log
grep -ci 'googlebot' all.log
grep -i 'googlebot' all.log | grep -ci 'android'   # Googlebot Smartphone
grep -Eio 'GPTBot|OAI-SearchBot|ClaudeBot|PerplexityBot|bingbot|Applebot' all.log | sort | uniq -c | sort -rn
Pro tip: Also check for `Googlebot-Image` and `Storebot-Google` if you run an image-heavy site or an online store.
3

Verify the hits are real Googlebot

Anyone can fake the Googlebot user agent. com**, and a forward lookup of that host must return the same IP. json`.

log before analysing.

text
awk '{print $1}' googlebot.log | sort | uniq -c | sort -rn | head -20
host 66.249.66.1
# 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
host crawl-66-249-66-1.googlebot.com
# crawl-66-249-66-1.googlebot.com has address 66.249.66.1
grep -v '203.0.113.45' googlebot.log > googlebot-verified.log
Pro tip: Spoofed Googlebot traffic can be rate-limited or blocked at the firewall; blocking a verified Googlebot IP never should be.
4

Break crawls down by section and status code

In the default combined log format, field 7 is the requested path and field 9 is the HTTP status. Count hits per top-level directory, per status code, and for URLs with query strings. This shows where Google spends its crawling and how much of it lands on errors or redirects.

text
# Hits per top-level section
awk '{split($7,a,"/"); print "/"a[2]}' googlebot-verified.log | cut -d'?' -f1 | sort | uniq -c | sort -rn | head -20

# Status code mix
awk '{print $9}' googlebot-verified.log | sort | uniq -c | sort -rn

# Parameter URLs
awk '$7 ~ /\?/' googlebot-verified.log | wc -l
awk '$7 ~ /\?/ {split($7,a,"?"); print a[2]}' googlebot-verified.log | cut -d'=' -f1 | sort | uniq -c | sort -rn | head

# Most-crawled 404s
awk '$9==404 {print $7}' googlebot-verified.log | sort | uniq -c | sort -rn | head -20
Pro tip: If your log format is custom, check the `log_format` directive (Nginx) or `LogFormat` (Apache) and adjust the field numbers.
5

Compare crawl share against your priority sections

Put the section counts next to your list of important sections and look for these patterns:

  • A priority section such as /products/ gets a small share, or key URLs never appear at all
  • Parameter, internal-search or calendar URLs take a large share
  • More than 5-10% of hits return 301/302 or 404/410
  • 5xx responses or slow responses, which make Google crawl less

List the worst offenders with example URLs; these drive the fixes in the next step.

Pro tip: Check specific money pages too: `grep -c '/products/blue-widget' googlebot-verified.log`. Zero hits in 30 days means Google barely knows the page exists.
6

Cut crawl waste and strengthen priority paths

Fix what the logs revealed:

  • Block infinite URL spaces (internal search, sort and filter parameters) in robots.txt
  • Update internal links that point to redirected URLs so they link to the final URL directly
  • Return 410 or redirect old 404s that still get crawled, and remove them from sitemaps
  • Link priority sections from the main navigation and list them in your XML sitemap with accurate lastmod dates
  • Fix recurring 5xx errors and slow endpoints

Only block URLs you do not want indexed; robots.txt stops crawling, not indexing of already-linked URLs.

text
User-agent: *
Disallow: /search
Disallow: /*?sort=
Disallow: /*?filter=
Disallow: /*&sessionid=

Sitemap: https://www.example.com/sitemap.xml
Pro tip: Test new robots.txt rules on a few sample URLs before deploying; a stray `Disallow: /products` can hide an entire catalogue.
7

Re-run the analysis and cross-check Search Console

Wait 2-4 weeks, export fresh logs and repeat steps 2-5 with the same commands so the numbers are comparable. Then open Google Search Console > Settings > Crawl stats > Open report and compare total requests, the By response breakdown and By file type with your log figures. Save the commands as a script and schedule a monthly review.

Pro tip: Crawl stats only covers Google and samples example URLs, so treat it as a sanity check, not a replacement for logs.

Platform-specific notes

WordPress (managed hosting)

Logs live with the host, not WordPress. Kinsta: MyKinsta > Sites > your site > Logs (access.log). WP Engine: User Portal > your environment > Logs. cPanel hosts: Metrics > Raw Access. Some hosts keep only a few days, so download regularly.

Shopify

Shopify does not give merchants raw server logs. Use Search Console > Settings > Crawl stats instead, and control crawl waste through robots.txt.liquid (Online Store > Themes > Edit code > Add a new template > robots). Shopify's default rules already block /search and many filter parameters.

Cloudflare

If Cloudflare proxies your site, the origin only sees cache misses. Raw request logs come from Logpush or Log Explorer, which require paid or Enterprise plans. On lower plans, use Security > Analytics or Security > Bots to see verified-bot traffic, and Radar's verified bot list to confirm Googlebot.

Vercel / Netlify

Built-in runtime logs are short-lived and focus on functions. For full request logs, configure Log Drains (Vercel: Team Settings > Log Drains; Netlify: Site configuration > Log drains, Enterprise) to a service such as Datadog or an S3 bucket, then run the same analysis on the exported data.

Wix / Squarespace

No server log access is available. Rely on Search Console Crawl stats and the URL Inspection tool, and keep your sitemap and internal navigation focused on priority pages.

Check that it worked

✅ How to verify

  • Verified Googlebot hits to your priority sections make up a larger share than in the first review, and key money pages appear in the logs at least weekly.
  • Requests to blocked parameter and search URLs drop sharply in the new log export.
  • 3xx and 4xx responses are under roughly 5% of Googlebot requests, and 5xx is near zero.
  • Search Console Crawl stats shows a matching trend in By response and a stable or rising total crawl requests figure.

⚠️ Common mistakes

  • Trusting the user agent alone: scrapers commonly fake Googlebot, so unverified counts can be badly inflated.
  • Analysing origin logs behind a CDN and concluding Google barely crawls the site, when most bot hits were served from cache and never reached the origin.
  • Using robots.txt to block pages that carry a noindex tag, which stops Google from ever seeing the noindex and keeps them in the index.
  • Reviewing only a day or two of data; crawl patterns vary widely day to day, so use at least 14 days.

Is this still an issue on your site? Run a free audit — ZORO checks this and 294 other SEO, AI-visibility, accessibility and security signals, with fixes for your platform.

Check my site free

FAQ

How do I know if Googlebot is crawling my website?

Search your access logs for the Googlebot user agent and verify the IPs with a reverse and forward DNS lookup ending in googlebot.com or google.com. Without log access, Search Console > Settings > Crawl stats shows total Google requests per day and example URLs.

How often should Googlebot crawl my pages?

There is no fixed rate. Homepages and frequently updated sections are often crawled daily; deep pages may be crawled every few weeks. Worry when important pages get no hits over 30 days or when most crawls go to low-value URLs.

Can I see AI crawlers like GPTBot in my server logs?

Yes. GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and others identify themselves in the user agent. Grep for their names to see what they fetch, and confirm against the IP ranges each company publishes before allowing or blocking them in robots.txt.

Do I need log file analysis for a small website?

For sites under a few thousand URLs, crawl budget is rarely a limit, and Search Console Crawl stats is usually enough. Log analysis pays off for large catalogues, sites with faceted navigation, frequent migrations, or unexplained indexing delays.