How to Fix robots.txt Blocking CSS, JS and Key Pages

Your robots.txt contains Disallow rules that stop Googlebot from crawling your whole site or the CSS and JavaScript files your homepage loads. By the end of this guide you will have found the exact rules causing the block, rewritten robots.txt so Googlebot can reach your pages and assets, and confirmed the fix in Google Search Console.

By Updated ⏱ 15 minutes🎯 Beginner● High impactTechnical
Illustration: robots.txt blocks important paths fix guide
TECHNICAL · FIX GUIDE robots.txt blocks important paths IMPACTEFFORT TIME15 minutes LEVELBeginner

Why it matters

Google renders pages the way a browser does, so a blocked stylesheet or script can make your page look broken, empty or not mobile-friendly to Googlebot. That can hurt rankings and leave content loaded by JavaScript out of the index. A leftover Disallow: / is worse: Google stops crawling the site entirely.

Blocked URLs can still appear in results as bare links with no description. txt also lose access to the content they would quote.

✕Beforehttps://example.com/robots.txt has"User-agent: *" followed by "Disallow:/assets/", so Googlebot cannot fetch✓Afterrobots.txt only blocks private pathssuch as /cart/ and /admin/. Googlebotcan fetch the homepage and every CSS

What you need

Access to edit robots.txt (FTP/hosting file manager, CMS SEO settings, or your code repository)
A terminal with curl, or a browser
Google Search Console access for the site (Owner or Full user)

Step-by-step fix

1

Download and read your live robots.txt

Fetch the exact file that crawlers see. It can differ from the file in your CMS because of caching, CDN rules or a staging copy that went live. Look for three things:

  • Disallow: / under User-agent: * or User-agent: Googlebot
  • Disallow lines that match asset folders or file types, such as /assets/, /static/, /wp-includes/, /*.js$ or /*.css$
  • Any User-agent: Googlebot group at all
text
curl -s https://example.com/robots.txt

# Also check that it returns 200 and isn't redirected to a staging host
curl -sI https://example.com/robots.txt
Pro tip: If robots.txt returns 404, nothing is blocked. If it returns a 5xx error, Google may treat the whole site as blocked until the server error is fixed.
2

List the CSS and JS files your homepage loads

The audit checks every stylesheet (<link rel="stylesheet" href>) and script (<script src>) on your homepage that is served from your own domain. Files on other domains, such as a CDN or Google Tag Manager, are not affected by your robots.txt. Extract the paths so you can test each one against your rules.

text
curl -s https://example.com/ | grep -Eo '(src|href)="[^"]+\.(js|css)[^"]*"' | sort -u
Pro tip: Write down the folder each file sits in, such as /assets/, /_next/static/ or /wp-content/themes/. Those folders are what your Disallow rules must not cover.
3

Remove any site-wide Disallow: / for Googlebot

A Disallow: / usually comes from a staging site or from a "hide from search engines" setting that was left on at launch. Delete that line from the User-agent: * group, and from any User-agent: Googlebot group. An empty Disallow: line means "allow everything".

text
# Before (blocks the whole site)
User-agent: *
Disallow: /

# After
User-agent: *
Disallow:
Pro tip: Use HTTP authentication or a noindex header on staging hosts instead of robots.txt. Then the rule can't follow the code to production.
4

Unblock asset folders and file types

Delete the Disallow rules that match the paths you found in step 2. If you need to keep a broader folder blocked, add a more specific Allow rule. Google applies the rule with the longest matching path, and when an Allow and a Disallow match with equal length, Allow wins.

Wildcards are supported: * matches any characters and $ anchors the end of the URL.

text
User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /search

# Keep a folder blocked but let its assets through
Disallow: /includes/
Allow: /includes/*.css$
Allow: /includes/*.js$

Sitemap: https://example.com/sitemap.xml
Pro tip: `$` blocks a match when the file has a query string such as `app.js?v=3`. Use `Allow: /includes/*.js` without `$` if your assets carry version parameters.
5

Check for a separate Googlebot group

Googlebot obeys only the most specific group that names it. If a User-agent: Googlebot group exists, Googlebot ignores everything under User-agent: *. That means fixing the * group does nothing if the Googlebot group still blocks assets.

Either delete the Googlebot group or apply the same fixes to it.

text
# Googlebot reads ONLY this group, not the * group
User-agent: Googlebot
Disallow: /assets/   # remove this line

User-agent: *
Disallow: /admin/
Pro tip: Rules for other bots, such as GPTBot or CCBot, don't affect Googlebot. Keep or drop them based on your AI crawling policy.
6

Make sure revenue pages are crawlable too

While the file is open, confirm nothing blocks pages that earn money or leads, such as /products/, /collections/, /pricing, /services/ or /blog/. A broad rule like Disallow: /p also blocks /pricing and /products/ because matching is by prefix. Block only low-value URL patterns, such as internal search results, cart, account and faceted filter parameters.

text
# Too broad - also blocks /pricing and /products/
Disallow: /p

# Precise
Disallow: /p/preview/
Disallow: /*?sort=
Pro tip: Use robots.txt to manage crawling, not to remove pages from the index. To deindex a page, allow crawling and add `<meta name="robots" content="noindex">`.
7

Publish and verify the result

txt, and purge any CDN cache for that URL. txt > Open report. Confirm the new version was fetched; you can choose Request a recrawl** from the three-dot menu.

txt.

text
# Confirm the deployed version is live (bypass cache)
curl -s -H 'Cache-Control: no-cache' "https://example.com/robots.txt?nocache=$(date +%s)"
Pro tip: Google caches robots.txt for up to 24 hours. A recrawl request from the robots.txt report speeds this up.

Platform-specific notes

WordPress

WordPress serves a virtual robots.txt unless a physical file exists in the web root; a physical file always wins, so check over FTP first. Edit the virtual file in Yoast SEO > Tools > File editor, or in Rank Math > General Settings > Edit robots.txt. Remove old rules such as `Disallow: /wp-includes/` or `Disallow: /wp-content/plugins/`, because they block theme and plugin CSS and JS. Keep the default `Disallow: /wp-admin/` with `Allow: /wp-admin/admin-ajax.php`.

Shopify

Shopify's default robots.txt doesn't block theme assets. To change it, go to Online Store > Themes > ... > Edit code, then Add a new template > robots, which creates robots.txt.liquid. Edit the Liquid loop instead of hard-coding the file, so Shopify's default rules keep updating. Check that no custom `Disallow` was added for `/collections/` or `/products/`.

Wix

Go to Dashboard > Marketing & SEO > SEO > Robots.txt Editor (under Tools and settings) > View File. Delete custom Disallow lines you added, or click Reset to default. Wix loads core scripts from its own static domains, so blocks usually come only from custom rules.

Webflow

Go to Site settings > SEO > Indexing and edit the robots.txt field, then republish the site. The 'Disable Webflow subdomain indexing' toggle only affects the .webflow.io domain. If your custom domain shows `Disallow: /`, the rule was typed into the robots.txt field.

Next.js

With the App Router, robots.txt is generated from app/robots.ts or a static app/robots.txt, and a file in /public conflicts with it. Never disallow `/_next/`, because your JS and CSS chunks are served from `/_next/static/`. Check environment-based logic that returns `disallow: '/'` when a variable like VERCEL_ENV is missing in production.

Check that it worked

✅ How to verify

  • Search Console > Settings > robots.txt report shows the new file fetched with status 'Fetched' and no Disallow: / under Googlebot or *.
  • URL Inspection > Test live URL on the homepage lists no page resources as 'Blocked by robots.txt', and the rendered screenshot shows the styled page.
  • Every CSS/JS path from step 2 passes a check with a robots.txt parser, such as Google's open-source robotstxt library or Python's `urllib.robotparser`.
  • Re-run the audit: it should report that Googlebot can reach the homepage and its CSS/JS.

⚠️ Common mistakes

  • Fixing the `User-agent: *` group while a `User-agent: Googlebot` group still blocks assets. Googlebot reads only its own group.
  • Editing the CMS's robots.txt setting while a physical robots.txt file on the server or a CDN rule overrides it.
  • Using short prefixes like `Disallow: /p` or `Disallow: /s` that accidentally match /products/, /pricing, /shop or /static/.
  • Relying on robots.txt to hide pages from search. Blocked URLs can still be indexed from links; use noindex or authentication instead.

Is this still an issue on your site? Run a free audit — ZORO checks this and 294 other SEO, AI-visibility, accessibility and security signals, with fixes for your platform.

Check my site free

FAQ

Should robots.txt block CSS and JavaScript files?

No. Google needs your CSS and JS to render pages and judge layout, mobile usability and JavaScript-loaded content. Blocking them can lower rankings or leave content unindexed. Only block scripts that never affect what a page displays, and even then there is little to gain.

How long does it take Google to see robots.txt changes?

Google usually caches robots.txt for up to 24 hours. You can speed this up with Request a recrawl in the Search Console robots.txt report. Pages that were blocked are then recrawled over the following days or weeks, depending on crawl demand.

Where is the robots.txt Tester in Search Console?

Google retired the standalone robots.txt Tester in late 2023. Use Settings > Crawling > robots.txt to see fetched versions and errors, and use URL Inspection > Test live URL to check whether a specific page or resource is blocked.

Do Allow rules override Disallow in robots.txt?

For Google, the most specific rule wins, meaning the one with the longest matching path. If an Allow and a Disallow rule are equally specific, Allow wins. So `Allow: /assets/css/` overrides `Disallow: /assets/` for files in /assets/css/.