SEO for Developers

Excluded by noindex tag? Find where the noindex hides

The four places a noindex hides, and how to check them the way Google does

Sandipan Das
Sandipan Das · full-stack developer, Kolkata, IndiaPublished · 10 min read
Excluded by noindex tag? Find where the noindex hides — cover image

Your page's code says "index". Search Console, Google's free report on your site, says "Excluded by 'noindex' tag". Both can be true at once, because a noindex (a "don't list this page" rule) can sit where View Source never shows it.

Below are the four places it hides, how to check each one the way Google does, and the Next.js setup I now use so a rule meant for a test copy of a site never reaches the live one. I tested it all today, on my site and a test app.

Why does Search Console say "Excluded by noindex tag"?

Google fetched the page, found a noindex rule and left it out. The rule sits in a meta tag (a line in the page's code) or an X-Robots-Tag header (a note the server sends with the page). Google's live test (a fresh check in Search Console) shows which. Remove it only if you want that page in Google.

The meta tag lives inside the page's <head>, so you can see it in View Source. The header travels next to the page, so you can't. That difference is why a page can "say index" and still be a noindex page.

Google's guide to noindex says there are two ways to set it: <meta name="robots" content="noindex"> in the HTML, or X-Robots-Tag: noindex as a header. Google also says it has to crawl (fetch) the page to see either one.

First, check if you meant it

This row isn't always a problem. On my own site, all 19 URLs in it are pages I want out of Google.

I opened the Page indexing report for sandipandas.dev today (19 November 2026; Google's data was last updated 16 November). "Excluded by 'noindex' tag" had 19 pages. I fetched every one and read its robots tag:

Pages in the row

What I found

10 blog category pages (9 on www, 1 old copy without www)

noindex, follow meta tag. On purpose: they only list posts.

8 project filter pages, like /project?stack=nextjs

noindex, follow. On purpose: the same projects, filtered.

1 order status page

noindex, nofollow. On purpose: it's for customers, not search.

None of them sent an X-Robots-Tag header. My real posts sent index, follow. Then I ran Google's own check on one category page, and it agreed: "Indexing allowed? No: 'noindex' detected in 'robots' meta tag".

So sort your list into two piles first: pages you want in Google, and pages you don't (tag pages, filters, thank-you pages, logins, carts). The second pile is fine where it is. Only the first pile needs fixing. If you'd like someone to sort a long list like this for you, that's part of my website audit.

Chart of the 19 sandipandas.dev URLs excluded by noindex tag, split into blog category pages, project filter pages and one order status page, next to my Next.js test results
My own "Excluded by 'noindex' tag" row on 19 Nov 2026: every page in it is meant to be there.

The four places a noindex hides

When a page you want is in the row, the rule is in one of four places. Two of them don't show up in View Source.

  1. The meta tag in the HTML. Search the page source for name="robots" and name="googlebot". Google reads both. A theme, a page builder or an SEO plugin may add it.

  2. The X-Robots-Tag header. It travels with the page, not inside it, so View Source can't show it. It can come from server settings (for example an .htaccess file, a settings file on Apache servers), a CDN rule (a CDN is a network that serves your site from many places), your framework's config, or your host.

  3. A CMS setting. In WordPress that's Settings → Reading → "Discourage search engines from indexing this site", or a setting in your SEO plugin for one post or a whole post type (posts, pages, products and so on).

  4. A staging rule that reached the live site. A staging site is a private copy where you test changes. It should have noindex. The trouble starts when that rule is copied, built or promoted into the live site.

If two rules disagree, Google's robots meta tag docs say "the more restrictive rule applies". So a page with index in the HTML and noindex in the header is a noindex page. That's the classic "my page says index" case.

Diagram of the four places a noindex can hide: robots meta tag in the HTML, X-Robots-Tag response header, WordPress search engine visibility setting, and a staging rule left on the live site
Two of the four places can't be seen in View Source. Check headers too.

Check what Google sees, not what you see

Use Google's live test first. It fetches the page now and tells you where it found the noindex.

  1. In Search Console, paste the full URL into the bar at the top (that's the URL Inspection tool).

  2. Click Test live URL and wait a minute.

  3. Under Page availability, read Indexing allowed? On my category page it said "No: 'noindex' detected in 'robots' meta tag", so I knew to look at the HTML, not the server.

  4. Click View tested page to see the HTML and headers Google got.

The saved result (not the live test) shows what Google saw at its last crawl. If the live test says "Yes" but the saved one says "No", you've already fixed it and Google just hasn't been back.

For many pages at once, I use a small script. It asks for each page with a Googlebot-style user agent (the name a visitor sends to say what it is) and prints the status code (200 means OK, 3xx means a redirect), the X-Robots-Tag header and every robots meta tag. It is not Google: a site or firewall may treat real Googlebot differently, so confirm with the live test. It also reads only the HTML the server sends, not changes JavaScript makes later, and it looks for quoted name values only.

// scripts/check-noindex.mjs — shows the noindex rules for each URL.
// Usage: node scripts/check-noindex.mjs https://example.com/ https://example.com/about
// Needs Node 18+ (built-in fetch). Reads the first response only (no redirects followed).
const GOOGLEBOT_UA =
  'Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 ' +
  '(KHTML, like Gecko) Chrome/131.0.0.0 Mobile Safari/537.36 ' +
  '(compatible; Googlebot/2.1; +http://www.google.com/bot.html)'

for (const url of process.argv.slice(2)) {
  const res = await fetch(url, { headers: { 'user-agent': GOOGLEBOT_UA }, redirect: 'manual' })
  const html = await res.text()
  const metaTags = html.match(/<meta[^>]+name=["'](robots|googlebot)["'][^>]*>/gi) ?? []
  console.log(url)
  console.log('  status:       ', res.status, res.headers.get('location') ?? '')
  console.log('  X-Robots-Tag: ', res.headers.get('x-robots-tag') ?? '(none)')
  console.log('  meta tags:    ', metaTags.length ? metaTags.join(' ') : '(none)')
}

Checked with Node 22.23 on 19 Nov 2026. If the status is a 3xx, run it again on the address in "location".

Next.js: one switch, set per environment

Don't hardcode noindex in a layout. Read it from one environment variable, and make the build fail if that variable is missing.

In the Next.js App Router (the app/ folder setup), a layout wraps every page under it, so a robots rule in the root layout lands on the whole site. An environment variable is a setting you give the site per place it runs (live, staging, your laptop). This is the setup I tested:

// lib/site.ts — one place that decides if this build may be indexed.
// Set SITE_ENV=production only on the live site. Anything else = noindex.
const siteEnv = process.env.SITE_ENV

if (!siteEnv) {
  // Fail the build instead of guessing. A silent guess is how a live
  // site ends up with noindex (or a staging site ends up in Google).
  throw new Error('SITE_ENV is not set. Use "production" on the live site, anything else elsewhere.')
}

export const allowIndexing: boolean = siteEnv === 'production'
// app/layout.tsx — robots rule comes from lib/site.ts, never hardcoded
import type { Metadata } from 'next'
import type { ReactNode } from 'react'
import { allowIndexing } from '@/lib/site'

export const metadata: Metadata = {
  title: { default: 'Test site', template: '%s | Test site' },
  robots: allowIndexing
    ? { index: true, follow: true }
    : { index: false, follow: false },
}

export default function RootLayout({ children }: { children: ReactNode }) {
  return (
    <html lang="en">
      <body>{children}</body>
    </html>
  )
}

Checked with Next.js 16.4.6, React 19.3.1 and TypeScript 7.0.2: the build (next build) and the strict type check (tsc --noEmit) both pass. The robots field is the one in the Next.js metadata docs.

Here is what my test app sent, checked with the script above:

Setup

What every page sent

Built with SITE_ENV=production

index, follow

Built with SITE_ENV=staging

noindex, nofollow

SITE_ENV missing

Build stopped with my error. Nothing shipped.

Built as staging, then started with SITE_ENV=production (no new build)

Still noindex, nofollow

Meta tag index, plus an X-Robots-Tag rule in next.config.ts

Meta index, follow, header noindex

Two of those rows matter most. First, on pages Next.js builds ahead of time, the rule is fixed at build time. If you build once for staging and reuse that same build for the live site, the noindex comes with it. Build again with the live setting. Second, a header rule hides well. This is the line to search your code for:

// next.config.ts — the kind of rule that hides a noindex from View Source
headers: [{ key: 'X-Robots-Tag', value: 'noindex' }]

Vercel: previews get a header you didn't write

If you host on Vercel, some addresses get X-Robots-Tag: noindex from Vercel itself.

Vercel's guide on preview indexing says it adds that header to preview deployments (test copies made for each branch) and to production deployments opened through their own unique deployment address. It does not add it when a preview is served from your own domain that you've assigned to a Git branch (a separate line of work in your code), such as a staging subdomain. So:

  • If Google lists a preview or deployment address on vercel.app in this row, that's expected. Make sure your real domain is the one you link to and list in your sitemap.

  • A staging subdomain on its own domain is not covered. Protect it with a password, or add the header yourself for that environment.

WordPress: one checkbox, and your SEO plugin

Untick "Discourage search engines" on the live site, then check your SEO plugin's settings for the post type.

WordPress's own core notes say that since version 5.3, sites with that box ticked show <meta name='robots' content='noindex,nofollow' />, and the old Disallow: / line in robots.txt was removed for them. Since 5.7, the code behind it is a function called wp_robots_no_robots. Your SEO plugin can also set noindex per post, per category or for a whole post type, so check its settings for the type of page Google listed. I did not run a WordPress test today; this part comes from WordPress's own docs and code.

After the fix: tell Google

Make sure Google can fetch the page, run the live test again, then ask Google to check.

  1. Check robots.txt doesn't block the page. robots.txt is a file that tells crawlers what not to fetch. Google says that if the page is blocked there, "the crawler will never see the noindex rule", so it can't see you removed it either.

  2. Run the live test. "Indexing allowed?" should now say Yes.

  3. For one or two pages, click Request indexing. For many pages, open the row in the Page indexing report and click Validate fix. Google's Page indexing report help says "Validation typically takes up to about two weeks, but in some cases can take much longer".

Allowed to index isn't the same as indexed.

A noindex in the header beats an "index" in the HTML. Check what the server sends, not only what the page says.

What's easy to get wrong

  • Fixing pages you meant to hide. Tag, filter and thank-you pages belong in this row. Sort first.

  • Checking only View Source. Headers don't show there. Use the live test or the script.

  • Blocking in robots.txt and adding noindex. Google can't read the noindex on a page it isn't allowed to fetch.

  • Reusing a staging build for the live site. In my test the noindex stayed after I changed the setting without building again.

What I tested, and what I didn't

Tested on 19 Nov 2026: my site's Page indexing report and all 19 URLs in the row (browser fetch while logged out), one URL in Google's live test (it crawled as "Google Inspection Tool smartphone"), and a Next.js 16.4.6 test app on a cloud test server, checked with the script above using a Googlebot-style user agent. Not tested: real Googlebot on the test app, how Google reports a header-only noindex in Search Console, a live Vercel preview, WordPress itself, and other search engines.

If you'd rather have someone do this

I've been building and fixing websites since 2017, across 900+ projects, and Search Console clean-ups are a regular part of my technical SEO work. If your important pages are stuck in this row and you'd rather not dig through headers yourself, book a short call and we'll look at it together.

Questions

Questions this post answers

Is "Excluded by noindex tag" bad for SEO?

Not by itself. It only means Google found a noindex rule (an instruction not to list the page) and left those pages out. For tag pages, filters, logins or thank-you pages that's what you want. It's only a problem when a page you want people to find is in the list. Open the row, look at the example URLs, and fix only those.

Why does my page say index but Google says noindex?

The noindex may be somewhere you didn't look: an X-Robots-Tag header (a note the server sends with the page) from the server, host or framework config, or a second meta tag named "googlebot". Google follows the stricter rule when two disagree. Run the URL Inspection live test in Search Console; "Indexing allowed?" tells you whether it found the rule in a meta tag or a header.

How long does it take after I remove noindex?

Google has to fetch the page again first. For one page, Request indexing in URL Inspection (Search Console's page checker) asks it to. For many pages, Validate fix in the Page indexing report starts a check that Google says typically takes up to about two weeks, and sometimes much longer. Being allowed to index doesn't mean Google will index every page.

Should I use robots.txt or noindex to hide a page?

Use noindex to keep a page out of Google's results, and keep the page open to crawling (fetching) so Google can read the rule. robots.txt (the file that tells crawlers what not to fetch) stops crawling, not listing, and a blocked page can't show its noindex. For private pages, a login is safer than either.

How do I check for an X-Robots-Tag header?

Use the URL Inspection live test in Search Console and click View tested page to see the headers Google got. On your own computer, run the script above, or curl -I with the page address (curl is a command-line tool that fetches a page; -I shows only its headers). Look for a line starting with X-Robots-Tag.

Share:X·LinkedIn·Email

Related

  • SEO for Developers

    How to create an XML sitemap

    How to create an XML sitemap, in one paragraph To create an XML sitemap, let the platform generate a list of every URL you want Google to index — canonical addresses only, each returning 200, with an honest last-modified date…

    · 6 min read · by Sandipan Das
  • SEO for Developers

    How to check robots.txt

    How to check robots.txt, in one paragraph To check robots.txt, open yourdomain.com/robots.txt in a browser — it's a public text file — and read it against three questions: is anything you want indexed under a Disallow line, is anything Google…

    · 6 min read · by Sandipan Das
  • SEO for Developers

    How to fix indexing problems

    How to fix indexing problems, in one paragraph To fix indexing problems, open Search Console → Page indexing, read the reason row each page sits in, and match it to one of five kinds: Google can't reach the page, was…

    · 7 min read · by Sandipan Das

If you'd rather not do it yourself

Everything above works. You just need the time for it.

Follow the steps and fix it yourself. Or send the site to the team and get on with your day.

900+ projects completed since 2017, from Kolkata, India.