The past, present, and future of robots.txt

In this episode of Search Off the Record, Gary, Lizzi, and Martin from the Google Search team are joined by special guest David Price, a Product Counsel for Google.  They discuss human-initiated crawl requests vs. bot-initiated crawl requests, the robots.txt file, and more!

Links: