Google’s John Mueller answered a query on Reddit a few seemingly false ‘noindex detected in X-Robots-Tag HTTP header’ error reported in Google Search Console for pages that shouldn’t have that particular X-Robots-Tag or some other associated directive or block. Mueller instructed some potential causes, and a number of Redditors supplied cheap explanations and options.
Noindex Detected
The one who began the Reddit dialogue described a situation that could be acquainted to many. Google Search Console reviews that it couldn’t index a web page as a result of it was blocked not from indexing the web page (which is completely different from blocked from crawling). Checking the web page reveals no presence of a noindex meta aspect and there’s no robots.txt blocking the crawl.
Here’s what the described as their scenario:
- “GSC reveals “noindex detected in X-Robots-Tag http header” for a big a part of my URLs. Nonetheless:
- Can’t discover any noindex in HTML supply
- No noindex in robots.txt
- No noindex seen in response headers when testing
- Stay Check in GSC reveals web page as indexable
- Website is behind Cloudflare (We’ve got checked web page guidelines/WAF and many others)”
Additionally they reported that they tried spoofing Googlebot and examined varied IP addresses and request headers and nonetheless discovered no clue for the supply of the X-Robots-Tag
Cloudflare Suspected
One of many Redditors commented in that dialogue to recommend troubleshooting if the issue was originated from Cloudflare.
They supplied a complete step-by-step directions on how one can diagnose if Cloudflare or anything was stopping Google from indexing the web page:
“First, evaluate Stay Check vs. Crawled Web page in GSC to verify if Google is seeing an outdated response. Subsequent, examine Cloudflare’s Rework Guidelines, Response Headers, and Staff for modifications. Use curl with the Googlebot user-agent and cache bypass (Cache-Management: no-cache) to verify server responses. If utilizing WordPress, disable search engine marketing plugins to rule out dynamic headers. Additionally, log Googlebot requests on the server and verify if X-Robots-Tag seems. If all fails, bypass Cloudflare by pointing DNS on to your server and retest.”
The OP (orginal poster, the one who began the dialogue) responded that that they had examined all these options however have been unable to check a cache of the location through GSC, solely the dwell web site (from the precise server, not Cloudflare).
How To Check With An Precise Googlebot
Curiously, the OP said that they have been unable to check their web site utilizing Googlebot, however there may be really a means to try this.
Google’s Wealthy Outcomes Tester makes use of the Googlebot consumer agent, which additionally originates from a Google IP deal with. This instrument is helpful for verifying what Google sees. If an exploit is inflicting the location to show a cloaked web page, the Wealthy Outcomes Tester will reveal precisely what Google is indexing.
A Google’s wealthy outcomes help web page confirms:
“This instrument accesses the web page as Googlebot (that’s, not utilizing your credentials, however as Google).”
401 Error Response?
The next in all probability wasn’t the answer nevertheless it’s an fascinating little bit of technical search engine marketing information.
One other consumer shared the expertise of a server responding with a 401 error response. A 401 response means “unauthorized” and it occurs when a request for a useful resource is lacking authentication credentials or the supplied credentials are usually not the fitting ones. Their answer to make the indexing blocked messages in Google Search Console was so as to add a notation within the robots.txt to dam crawling of login web page URLs.
Google’s John Mueller On GSC Error
John Mueller dropped into the dialogue to supply his assist diagnosing the problem. He mentioned that he has seen this situation come up in relation to CDNs (Content material Supply Networks). An fascinating factor he mentioned was that he’s additionally seen this occur with very outdated URLs. He didn’t elaborate on that final one nevertheless it appears to suggest some form of indexing bug associated to outdated listed URLs.
Right here’s what he mentioned:
“Glad to have a look if you wish to ping me some samples. I’ve seen it with CDNs, I’ve seen it with really-old crawls (when the problem was there way back and a web site simply has plenty of historic URLs listed), perhaps there’s one thing new right here…”
Key Takeaways: Google Search Console Index Noindex Detected
- Google Search Console (GSC) could report “noindex detected in X-Robots-Tag http header” even when that header is just not current.
- CDNs, reminiscent of Cloudflare, could intrude with indexing. Steps have been shared to verify if Cloudflare’s Rework Guidelines, Response Headers, or cache are affecting how Googlebot sees the web page.
- Outdated indexing knowledge on Google’s aspect can also be an element.
- Google’s Wealthy Outcomes Tester can confirm what Googlebot sees as a result of it makes use of Googlebot’s consumer agent and IP, revealing discrepancies that may not be seen from spoofing a consumer agent.
- 401 Unauthorized responses can stop indexing. A consumer shared that their situation concerned login pages that wanted to be blocked through robots.txt.
- John Mueller instructed CDNs and traditionally crawled URLs as potential causes.