AI giants promise publishers new way to block some scraping
Full story
AI companies are promising a new method for publishers to block data scraping. Currently, publishers must use several different blocking commands for every bot that crawls the web. The proposed solution would replace this with a single, standardized instruction. This command would be recognized and honored by the bots.
Publishers currently have to implement multiple blocking commands for every bot that crawls the web to collect data, but the new proposed solution will allow them to enter a single, standardized instruction that's recognized and honored.