Cloudflare, a major internet infrastructure firm, has rolled out a new system that will allow millions of websites to block artificial intelligence (AI) bots from accessing their content without permission. This includes popular sites such as Sky News, The Associated Press, and Buzzfeed.
The system will eventually allow websites to charge AI firms for using their content without permission. This is a response to complaints from writers, artists, musicians, and actors who accuse AI firms of using their work without proper compensation.
Cloudflare’s technology targets AI bots, also known as crawlers, which are used by AI firms to collect data from the web. So far, the technology is active on a million websites and has been praised by publishers such as Roger Lynch, CEO of Condé Nast.
However, some experts argue that stronger legal protections are still needed. The system will initially apply to new Cloudflare users and those who participated in a previous effort to block crawlers. While publishers are generally okay with search engine crawlers, they have a problem with AI crawlers that collect content without directing visitors to the original source.
Cloudflare’s CEO, Matthew Prince, argues that this breaks the unwritten agreement between publishers and crawlers and deprives content creators of revenue. The company is now developing a “Pay Per Crawl” system, which will give content creators the option to charge AI firms for using their content.
According to Cloudflare, there has been a significant increase in AI bot activity, with over 50 billion requests to their network every day. This has led to concerns about AI crawlers disregarding existing protocols for excluding bots. In response, Cloudflare previously developed a system that would send the worst offenders to a “Labyrinth” of web pages filled with AI-generated junk.
While this new technology is a welcome development for content creators, some argue that it is only a temporary solution and that stronger legal protections are needed. Filmmaker Baroness Beeban Kidron, who is campaigning for more protection for the creative industries, praised Cloudflare for taking decisive action and called on AI companies to contribute to the communities in which they operate by paying their fair share of taxes and compensating creators for their work.