.Google's John Mueller addressed a question about why Google indexes pages that are actually prohibited coming from creeping by robots.txt and why the it is actually safe to neglect the relevant Search Console reports about those crawls.Robot Website Traffic To Concern Parameter URLs.The person inquiring the concern chronicled that bots were making links to non-existent query criterion Links (? q= xyz) to webpages along with noindex meta tags that are additionally blocked in robots.txt. What prompted the inquiry is actually that Google.com is actually creeping the hyperlinks to those webpages, receiving shut out through robots.txt (without noticing a noindex robotics meta tag) then getting shown up in Google.com Explore Console as "Indexed, though blocked out by robots.txt.".The person talked to the following question:." However listed here is actually the major question: why would Google index webpages when they can not also view the information? What is actually the conveniences during that?".Google.com's John Mueller verified that if they can not crawl the web page they can't view the noindex meta tag. He likewise makes an intriguing acknowledgment of the web site: search driver, encouraging to ignore the outcomes because the "average" consumers will not observe those results.He composed:." Yes, you are actually proper: if our experts can not crawl the web page, our company can not find the noindex. That mentioned, if our team can not creep the web pages, after that there is actually not a whole lot for us to mark. So while you might find some of those pages with a targeted internet site:- inquiry, the ordinary customer will not observe all of them, so I definitely would not fuss over it. Noindex is likewise great (without robots.txt disallow), it only implies the URLs are going to find yourself being crept (as well as find yourself in the Search Console record for crawled/not listed-- neither of these conditions result in issues to the rest of the web site). The important part is actually that you don't make all of them crawlable + indexable.".Takeaways:.1. Mueller's response verifies the limits in operation the Internet site: search accelerated hunt driver for analysis reasons. Among those main reasons is actually given that it's not connected to the regular search index, it's a different point completely.Google.com's John Mueller commented on the site hunt driver in 2021:." The short answer is that a web site: question is actually certainly not indicated to become complete, nor used for diagnostics purposes.A site question is actually a certain kind of search that restricts the outcomes to a certain internet site. It's essentially just the word website, a bowel, and after that the website's domain.This question restricts the outcomes to a details website. It's certainly not suggested to be a detailed compilation of all the web pages coming from that web site.".2. Noindex tag without making use of a robots.txt is great for these sort of conditions where a robot is actually linking to non-existent pages that are actually receiving found by Googlebot.3. URLs with the noindex tag will produce a "crawled/not listed" item in Explore Console and that those won't have an adverse effect on the remainder of the site.Go through the inquiry and also address on LinkedIn:.Why would Google index webpages when they can't even observe the material?Featured Picture by Shutterstock/Krakenimages. com.