Cloudflare added a new feature in their dashboard; subscribers can now block AI bots.
According to Cloudflare, the motivation behind this is to allow sites to eventually charge a “toll” for certain bots, AI in particular, to crawl the data. The initial functionality is simply to block the AI bots.
Eventually, Cloudflare envisions an ecosystem in which AI requests access to a site, the gatekeeper processes the request, charges the AI source, and allows access. Cloudflare sees itself as the gatekeeper ( as the CDN for about 20% of websites), splitting the fee with the site owner.
Why this is a bad idea for most sites
There may be a few sites on the web with content of such value, and size of such scale, that this model may provide a notable revenue stream. But, overwhelmingly, sites would find it a detriment to implement.
If a person asks their AI assistant for a local restaurant, where to get a car repaired, or any transaction request, blocking AI effectively precludes a business from being considered part of the response.
Even for informational purposes, being part of the AI ecosystem allows businesses to enter the shopping funnel. While not all AI responses provide the source of the information immediately, the source is available upon request. Blocking the AI removes a business’s ability to inform and even influence a person during the purchase process.
Rather than block AI bots, focus resources on making your information valuable and digestible for AI so they can use it to inform responses to users’ requests. Be part of the conversation.