How to Fix robots.txt Blocking CSS, JS and Key Pages
Your robots.txt contains Disallow rules that stop Googlebot from crawling your whole site or the CSS and JavaScript files your homepage loads. By the end of this guide you will have found the exact rules causing the block, rewritten robots.txt so Googlebot can reach your pages and assets, and confirmed the fix in Google Search Console.

Why it matters
Google renders pages the way a browser does, so a blocked stylesheet or script can make your page look broken, empty or not mobile-friendly to Googlebot. That can hurt rankings and leave content loaded by JavaScript out of the index. A leftover Disallow: / is worse: Google stops crawling the site entirely.
Blocked URLs can still appear in results as bare links with no description. txt also lose access to the content they would quote.
What you need
Step-by-step fix
Download and read your live robots.txt
Fetch the exact file that crawlers see. It can differ from the file in your CMS because of caching, CDN rules or a staging copy that went live. Look for three things:
Disallow: /underUser-agent: *orUser-agent: Googlebot- Disallow lines that match asset folders or file types, such as
/assets/,/static/,/wp-includes/,/*.js$or/*.css$ - Any
User-agent: Googlebotgroup at all
curl -s https://example.com/robots.txt # Also check that it returns 200 and isn't redirected to a staging host curl -sI https://example.com/robots.txt
List the CSS and JS files your homepage loads
The audit checks every stylesheet (<link rel="stylesheet" href>) and script (<script src>) on your homepage that is served from your own domain. Files on other domains, such as a CDN or Google Tag Manager, are not affected by your robots.txt. Extract the paths so you can test each one against your rules.
curl -s https://example.com/ | grep -Eo '(src|href)="[^"]+\.(js|css)[^"]*"' | sort -u
Remove any site-wide Disallow: / for Googlebot
A Disallow: / usually comes from a staging site or from a "hide from search engines" setting that was left on at launch. Delete that line from the User-agent: * group, and from any User-agent: Googlebot group. An empty Disallow: line means "allow everything".
# Before (blocks the whole site) User-agent: * Disallow: / # After User-agent: * Disallow:
Unblock asset folders and file types
Delete the Disallow rules that match the paths you found in step 2. If you need to keep a broader folder blocked, add a more specific Allow rule. Google applies the rule with the longest matching path, and when an Allow and a Disallow match with equal length, Allow wins.
Wildcards are supported: * matches any characters and $ anchors the end of the URL.
User-agent: * Disallow: /admin/ Disallow: /cart/ Disallow: /checkout/ Disallow: /search # Keep a folder blocked but let its assets through Disallow: /includes/ Allow: /includes/*.css$ Allow: /includes/*.js$ Sitemap: https://example.com/sitemap.xml
Check for a separate Googlebot group
Googlebot obeys only the most specific group that names it. If a User-agent: Googlebot group exists, Googlebot ignores everything under User-agent: *. That means fixing the * group does nothing if the Googlebot group still blocks assets.
Either delete the Googlebot group or apply the same fixes to it.
# Googlebot reads ONLY this group, not the * group User-agent: Googlebot Disallow: /assets/ # remove this line User-agent: * Disallow: /admin/
Make sure revenue pages are crawlable too
While the file is open, confirm nothing blocks pages that earn money or leads, such as /products/, /collections/, /pricing, /services/ or /blog/. A broad rule like Disallow: /p also blocks /pricing and /products/ because matching is by prefix. Block only low-value URL patterns, such as internal search results, cart, account and faceted filter parameters.
# Too broad - also blocks /pricing and /products/ Disallow: /p # Precise Disallow: /p/preview/ Disallow: /*?sort=
Publish and verify the result
txt, and purge any CDN cache for that URL. txt > Open report. Confirm the new version was fetched; you can choose Request a recrawl** from the three-dot menu.
txt.
# Confirm the deployed version is live (bypass cache) curl -s -H 'Cache-Control: no-cache' "https://example.com/robots.txt?nocache=$(date +%s)"
Platform-specific notes
WordPress
WordPress serves a virtual robots.txt unless a physical file exists in the web root; a physical file always wins, so check over FTP first. Edit the virtual file in Yoast SEO > Tools > File editor, or in Rank Math > General Settings > Edit robots.txt. Remove old rules such as `Disallow: /wp-includes/` or `Disallow: /wp-content/plugins/`, because they block theme and plugin CSS and JS. Keep the default `Disallow: /wp-admin/` with `Allow: /wp-admin/admin-ajax.php`.
Shopify
Shopify's default robots.txt doesn't block theme assets. To change it, go to Online Store > Themes > ... > Edit code, then Add a new template > robots, which creates robots.txt.liquid. Edit the Liquid loop instead of hard-coding the file, so Shopify's default rules keep updating. Check that no custom `Disallow` was added for `/collections/` or `/products/`.
Wix
Go to Dashboard > Marketing & SEO > SEO > Robots.txt Editor (under Tools and settings) > View File. Delete custom Disallow lines you added, or click Reset to default. Wix loads core scripts from its own static domains, so blocks usually come only from custom rules.
Webflow
Go to Site settings > SEO > Indexing and edit the robots.txt field, then republish the site. The 'Disable Webflow subdomain indexing' toggle only affects the .webflow.io domain. If your custom domain shows `Disallow: /`, the rule was typed into the robots.txt field.
Next.js
With the App Router, robots.txt is generated from app/robots.ts or a static app/robots.txt, and a file in /public conflicts with it. Never disallow `/_next/`, because your JS and CSS chunks are served from `/_next/static/`. Check environment-based logic that returns `disallow: '/'` when a variable like VERCEL_ENV is missing in production.
Check that it worked
✅ How to verify
- Search Console > Settings > robots.txt report shows the new file fetched with status 'Fetched' and no Disallow: / under Googlebot or *.
- URL Inspection > Test live URL on the homepage lists no page resources as 'Blocked by robots.txt', and the rendered screenshot shows the styled page.
- Every CSS/JS path from step 2 passes a check with a robots.txt parser, such as Google's open-source robotstxt library or Python's `urllib.robotparser`.
- Re-run the audit: it should report that Googlebot can reach the homepage and its CSS/JS.
⚠️ Common mistakes
- Fixing the `User-agent: *` group while a `User-agent: Googlebot` group still blocks assets. Googlebot reads only its own group.
- Editing the CMS's robots.txt setting while a physical robots.txt file on the server or a CDN rule overrides it.
- Using short prefixes like `Disallow: /p` or `Disallow: /s` that accidentally match /products/, /pricing, /shop or /static/.
- Relying on robots.txt to hide pages from search. Blocked URLs can still be indexed from links; use noindex or authentication instead.
Is this still an issue on your site? Run a free audit — ZORO checks this and 294 other SEO, AI-visibility, accessibility and security signals, with fixes for your platform.
Check my site freeFAQ
Should robots.txt block CSS and JavaScript files?
No. Google needs your CSS and JS to render pages and judge layout, mobile usability and JavaScript-loaded content. Blocking them can lower rankings or leave content unindexed. Only block scripts that never affect what a page displays, and even then there is little to gain.
How long does it take Google to see robots.txt changes?
Google usually caches robots.txt for up to 24 hours. You can speed this up with Request a recrawl in the Search Console robots.txt report. Pages that were blocked are then recrawled over the following days or weeks, depending on crawl demand.
Where is the robots.txt Tester in Search Console?
Google retired the standalone robots.txt Tester in late 2023. Use Settings > Crawling > robots.txt to see fetched versions and errors, and use URL Inspection > Test live URL to check whether a specific page or resource is blocked.
Do Allow rules override Disallow in robots.txt?
For Google, the most specific rule wins, meaning the one with the longest matching path. If an Allow and a Disallow rule are equally specific, Allow wins. So `Allow: /assets/css/` overrides `Disallow: /assets/` for files in /assets/css/.
How to Fix a Missing XML Sitemap (and Get It Found)
How to Fix a Missing robots.txt File (Step-by-Step)
How to Fix Non-200 URLs in Your XML Sitemap
How to Fix www vs non-www Duplicate Hostnames with 301s
How to Fix Missing or Inaccurate Sitemap lastmod Dates
How to Fix a Missing HSTS Header (Strict-Transport-Security)