Last April 2024, Gary Illyes from Google said he was on a mission to make web crawling more efficient, he wanted to “figure out how to crawl even less, and have fewer bytes on wire.” Gary updated us saying “there’s lots more in works” for this but implies some improvements have already been made.
