SEO for Developers · Business Tips

How to check robots.txt

A public text file at the root of your domain, read against three questions: is anything you want indexed disallowed, is anything Google needs to render the page blocked, is the sitemap listed. Then the one thing robots.txt can't do — keep a page out of Google.

Sandipan Das
Sandipan Das · full-stack developer, Kolkata, IndiaPublished · 6 min read

How to check robots.txt, in one paragraph

To check robots.txt, open yourdomain.com/robots.txt in a browser — it's a public text file — and read it against three questions: is anything you want indexed under a Disallow line, is anything Google needs to render the page (the theme's CSS and JavaScript) blocked, and is the sitemap listed. Then confirm how Google reads it in Search Console's robots.txt report.

Google's description of the file is the plainest one: it "tells search engine crawlers which URLs the crawler can access on your site", and it's used "mainly to avoid overloading your site with requests". It is a traffic sign for crawlers, not a lock on the pages — and that distinction is the whole of the last section.

I check it on every site before anything else, because a single wrong line here makes every other SEO effort pointless, and it's the fastest fix there is.

It's one file, read the same way on every site, and my team reads it first on every audit.

Open the file and read the blocks

yourdomain.com/robots.txt. If it isn't there, that's fine — no file means no restrictions. If it is, read it top to bottom; it's short.

The file is groups of rules. Each group starts with a User-agent line naming which crawler it applies to (an asterisk means all of them; Googlebot means Google), followed by Disallow lines for paths that crawler may not fetch and Allow lines for exceptions. Google reads four directives — User-agent, Disallow, Allow and Sitemap — and ignores the rest. A Disallow: / with nothing after the slash blocks the entire site; a Disallow: /cart/ blocks one folder.

Read each Disallow and ask what's under that path. /wp-admin/ is fine. /cart/ and /search/ are fine — those pages don't belong in the index. /wp-content/ is not fine, because that's where the theme's CSS and JavaScript live, and Google needs to fetch those to render the page properly. And / on its own is the one that gets sites de-listed after a launch.

The three questions

Every robots.txt check comes down to these. The free robots & sitemap checker asks them for you — it reads the file and the sitemap and flags a blocked site, blocked assets, and a missing or empty sitemap.

  • Is anything you want indexed disallowed? Compare the Disallow paths with the pages that earn — services, products, posts. If a Disallow covers them, remove it. Search Console's Page indexing report confirms the damage under the reason "Blocked by robots.txt".
  • Are CSS and JavaScript blocked? Google renders pages the way a browser does. If the stylesheet and scripts are under a blocked path, Google sees an unstyled page and may judge it broken. Allow the asset folders.
  • Is the sitemap listed? A Sitemap: line with the full URL tells every crawler where the sitemap is, no Search Console needed. If it's missing, add it; if it names a sitemap that no longer exists, fix it — How to create an XML sitemap.

Test it the way Google reads it

Search Console has a robots.txt report that shows the file Google last fetched, when, and whether it parsed. That's the version that counts — not the one on your server right now, if the two differ.

Two things to check there. First, that Google fetched the file at all: a robots.txt that returns an error can make Google hold off crawling entirely. Second, the copy Google holds against the copy you see — after an edit, Google picks up the new file on its next fetch, not instantly. For a single URL, URL Inspection says plainly whether crawling is allowed.

One host, one file. www.yourdomain.com and yourdomain.com are different hosts to a crawler, and so are http and https. The file that matters is the one on the host Google indexes — check the canonical host, then its robots.txt.

Testing robots.txt the way Google reads it: the Search Console robots.txt report, whether Google fetched the file, URL Inspection for one URL, and the canonical host
Test robots.txt the way Google reads it — the robots.txt report for the file, URL Inspection for one URL, on the canonical host

The WordPress check, and the staging leftover

On WordPress there's a setting — "Discourage search engines from indexing this site" — that adds a noindex to every page. Older WordPress versions also wrote a Disallow: / into the virtual robots.txt. Ticked on the staging copy, forgotten at launch, it's a common reason a new site never appears.

Settings → Reading → the checkbox. What I do on launch day:

  • Untick it on the live site, then view a page's source to see the noindex gone, and read yourdomain.com/robots.txt in a browser for any Disallow: / left behind.
  • Check the plugins that write their own — a security or SEO plugin may have its own setting.
  • Look for a physical file — a robots.txt in the site's root folder overrides the one WordPress generates, so there can be two truths; the one the browser shows is the one Google reads.

Blocked is not hidden

The point to remember: a Disallow in robots.txt does not keep a page out of Google. In Google's words, robots.txt is "not a mechanism for keeping a web page out of Google" — a blocked URL can still be indexed and shown, without a description, if something links to it.

To keep a page out of the index, use a noindex tag on the page, or put it behind a login. And the two don't combine: if you block a page in robots.txt and put a noindex on it, Google can't fetch the page to read the noindex, so the block wins and the URL may stay listed. Block what you don't want crawled — carts, searches, endless filters — and noindex what you don't want listed. Different tools, different jobs. How to fix indexing problems has the full map.

Blocked is not hidden: a Disallow in robots.txt stops the crawl but the URL can still be indexed; a noindex on the page keeps it out of the index
Blocked is not hidden — robots.txt stops the crawl, a noindex keeps the page out of the index, and a blocked page can't show Google its noindex

What's easy to get wrong

Four mistakes, in the order I find them.

  • Disallow: / left from staging. One character. Read the live file on launch day.
  • Blocking the asset folders. /wp-content/, /assets/, /static/ — Google needs the CSS and JS to render the page it's judging.
  • Using robots.txt to hide a page. It doesn't. Blocked pages still get indexed through links; noindex is the tool.
  • Checking the wrong host. The file on www and on non-www can differ; so can http and https. Check the canonical host's file.

robots.txt says where crawlers may go. It never says what Google may show. Block the crawl, noindex the page — and read the file on launch day.

The sitemap the file should point at, the indexing reasons, and the services.

If you'd rather have someone read it for you

Run the free robots & sitemap checker on your domain — if the site is blocked or the assets are, it says so. If the file is fine and the site still isn't showing, the problem is further in, and that's what a website audit finds.

Schedule a Meeting

Questions

Questions this post answers

Where should robots.txt be?

At the root of the host: yourdomain.com/robots.txt, one file per host and protocol. A file in a subfolder is ignored, and www.yourdomain.com and yourdomain.com each read their own — check the one on the host Google indexes as the canonical.

What happens if there is no robots.txt?

Nothing bad: no file means no restrictions, and crawlers fetch whatever they find. Most sites should still have one, if only for the Sitemap line and to keep crawlers out of carts, searches and admin paths — none of which affect what Google may show.

Does robots.txt block Google from indexing a page?

No. It stops Google from fetching the page, but a URL that's linked from elsewhere can still be indexed and shown without a description. To keep a page out of Google, use a noindex tag on the page (which Google has to be allowed to fetch to read) or put the page behind a login.

How do I test robots.txt for one URL?

Paste the URL into Search Console's URL Inspection: the crawl section says whether crawling is allowed and when Google last fetched it. For the file as a whole, Search Console's robots.txt report shows the copy Google holds and whether it parsed; the free robots & sitemap checker reads the live file from outside.

Share:X·LinkedIn·Email

Related

  • SEO for Developers

    How to fix indexing problems

    How to fix indexing problems, in one paragraph To fix indexing problems, open Search Console → Page indexing, read the reason row each page sits in, and match it to one of five kinds: Google can't reach the page, was…

    · 7 min read · by Sandipan Das
  • SEO for Developers

    Alternate page with proper canonical tag: when to fix it

    Search Console told me three of my pages were fine as "alternate" copies, yet all three were the exact pages my visitors land on. So "Alternate page with proper canonical tag" isn't always good news, and the two "Duplicate" rows…

    · 14 min read · by Sandipan Das
  • SEO for Developers

    How to check if a page is indexed by Google (3 ways)

    How do you check if a page is indexed by Google? Paste the page's full address into URL Inspection in Google Search Console. It will say "URL is on Google" or "URL is not on Google", with the reason. To…

    · 6 min read · by Sandipan Das

If you'd rather not do it yourself

Everything above works. You just need the time for it.

Follow the steps and fix it yourself. Or send the site to the team and get on with your day.

900+ projects completed since 2017, from Kolkata, India.