.Google's John Mueller addressed an inquiry regarding why Google marks pages that are disallowed coming from creeping through robots.txt and why the it is actually safe to neglect the related Explore Console records regarding those crawls.Robot Website Traffic To Inquiry Specification URLs.The person talking to the concern documented that bots were making hyperlinks to non-existent inquiry criterion Links (? q= xyz) to pages with noindex meta tags that are also blocked in robots.txt. What prompted the inquiry is actually that Google.com is crawling the web links to those web pages, acquiring shut out by robots.txt (without noticing a noindex robots meta tag) then obtaining shown up in Google Explore Console as "Indexed, though blocked through robots.txt.".The person talked to the observing question:." However listed here is actually the big question: why would Google.com mark webpages when they can't also view the web content? What's the advantage because?".Google.com's John Mueller affirmed that if they can't crawl the web page they can't see the noindex meta tag. He additionally helps make an exciting acknowledgment of the web site: hunt operator, encouraging to ignore the outcomes since the "common" users won't view those results.He wrote:." Yes, you are actually right: if our company can not crawl the webpage, our company can't view the noindex. That said, if our experts can't crawl the web pages, after that there is actually not a whole lot for our company to mark. Thus while you may observe some of those web pages with a targeted internet site:- concern, the typical customer won't view them, so I definitely would not fuss over it. Noindex is actually also fine (without robots.txt disallow), it just suggests the Links will end up being crept (and also wind up in the Explore Console record for crawled/not catalogued-- neither of these conditions cause issues to the rest of the site). The important part is actually that you don't make all of them crawlable + indexable.".Takeaways:.1. Mueller's solution affirms the constraints being used the Internet site: hunt accelerated search operator for analysis factors. Some of those factors is actually given that it's certainly not connected to the routine search mark, it's a separate point completely.Google.com's John Mueller talked about the site search driver in 2021:." The brief response is actually that a website: concern is actually not implied to be total, nor made use of for diagnostics functions.A website query is a particular kind of hunt that restricts the end results to a certain internet site. It's generally just the word site, a digestive tract, and after that the web site's domain.This question limits the end results to a particular internet site. It is actually certainly not meant to become an extensive assortment of all the webpages from that web site.".2. Noindex tag without making use of a robots.txt is alright for these kinds of circumstances where a robot is actually connecting to non-existent webpages that are actually obtaining found out by Googlebot.3. URLs along with the noindex tag are going to produce a "crawled/not catalogued" entry in Browse Console and also those will not have an unfavorable effect on the rest of the site.Review the question and respond to on LinkedIn:.Why will Google mark pages when they can not also view the information?Included Image through Shutterstock/Krakenimages. com.