In this guide
When a page is missing from Google, the cause is often a single line that tells search engines to stay away. Check these technical blockers first; they are quick to find and quick to fix.
Key takeaways
- Most missing pages are blocked by one line: a noindex tag or header, or a robots.txt rule.
- A canonical that points elsewhere tells Google to show another page instead.
- A page that redirects or returns an error cannot be indexed at its own address.
- New pages need links and a sitemap entry before Google finds them.
1. A noindex instruction
<meta name="robots" content="noindex"> in the page, or an X-Robots-Tag: noindex header from the server, asks search engines not to show the page. It is often left on by accident after a site launch, or set by an SEO plugin on a whole category of pages.
The meta tag checker reads both the tag and the header, so you see exactly where it comes from.
2. Blocked in robots.txt
A Disallow rule in robots.txt stops crawlers from reading the page. Google can still list a blocked address it finds elsewhere, but without a description, and it cannot see a noindex on a page it may not read.
Test the exact path with the robots.txt tester, for Googlebot and for other crawlers.
3. A canonical pointing elsewhere
<link rel="canonical"> tells search engines which address is the main one. If a page names another page as canonical, Google usually shows that other page instead. A common mistake is a theme that sets the home page as canonical on every page.
4. The page redirects or returns an error
If the address redirects, Google indexes the destination, not the address you checked. If it returns 404, 500 or times out, it drops out. The redirect checker shows the full path and the final status.
5. Not found yet
A new page with no links to it may simply not be discovered. Link to it from a page that is already indexed, add it to your XML sitemap, and request indexing in Google Search Console.
Check it in two minutes
- Run the meta tag checker on the page: look at Indexable, robots directives and canonical.
- Run the robots.txt tester with the page's path.
- Run the redirect checker to confirm the page answers 200 directly.
- If all three are clean, use URL Inspection in Google Search Console to see Google's own view.
Checklist
- Run the meta tag checker: Indexable must be yes, the canonical must point to the page itself.
- Test the path in the robots.txt tester for Googlebot.
- Confirm the page answers 200 directly in the redirect checker.
- List the page in your XML sitemap and link to it from an indexed page.
- Request indexing in Google Search Console's URL Inspection.
Questions
How long does Google take to index a new page?
From a few hours to a few weeks. Pages linked from well-visited pages and listed in the sitemap are usually found faster.
Should I block pages in robots.txt to remove them from Google?
No. To remove a page, let Google crawl it and see a noindex, or return 404/410. Blocking it in robots.txt hides the noindex from Google.