noindex and robots.txt: why your page isn't on Google

If a page isn't showing up in Google, the most common reason isn't bad SEO. It's a single line that tells search engines to stay away.

Two different switches

• robots.txt is a file at yoursite.com/robots.txt. It tells crawlers which paths not to crawl.
• noindex is a meta tag in the page (or an X-Robots-Tag header). It tells search engines not to list that page.

Where noindex comes from

• WordPress: Settings → Reading → Discourage search engines left on after launch.
• SEO plugins: a post type or category set to noindex.
• Staging copies pushed live with their settings.
• Server or CDN rules that add an X-Robots-Tag header.

A safe robots.txt

Keep it short, like the example above. Block admin areas and internal search results if you like, but never Disallow: / on a live site. Always include a Sitemap: line pointing to your XML sitemap.

How to fix

1. Run the website checker: the Indexing line shows whether the homepage has noindex.
2. Remove the noindex setting in your CMS or plugin, or the header rule on your server.
3. Fix robots.txt if it blocks pages you want found.
4. In Google Search Console, use URL Inspection → Request indexing for key pages.

When noindex is right

Thank-you pages, internal search results, login pages, cart and checkout, and thin tag archives are often better left out of search. The audit lists every noindex page so you can confirm each one is intentional.