Did you know that Google Search checks about four billion host names each and every day for robots.txt purposes? Gary Illyes said in the December Search Off The Record podcast “we have about four billion host names that we check every single day for robots.txt.”
