A publicly listed cloud service company, Cloudflare, has announced the availability of a new free tool that was designed to block bots from scraping websites hosted on its network for training artificial intelligence models.
In an accompanying blog post announcing this update, the Cloudflare team shared some data on how its clients are faring against the increased bot traffic using content scraping to train generative AI models.
The new tool is reportedly made available to all users of cloud service providers from the free plan onwards.
Developers can also block specific bots used to scrape data and train models by modifying their site’s robots.txt file. Developers can block particular bots that are used for scraping data and training models by changing their site’s Robots.txt file—a small file residing on your website that typically informs bots which pages it may or may not access—according to many AI providers, including Google, OpenAI, and Apple.
“Customers do not want AI bots to visit their websites, and especially not those that do so dishonestly,” the company explained in its official blog.