Blog | FELD M

Pay-per-Crawl & Co.: Companies vs AI Crawlers | FELD M

Written by Dr. Ramona Casasola-Greiner, Dr. Matthias Böck | Nov 11, 2026, 9:00:00 AM

AI traffic series - Part 4 of 5

In this fourth installment of our AI traffic blog series, we're looking at best practices companies are already using to respond to bot traffic, as well as current developments aimed at addressing "the bot problem" more broadly. 

Content providers and website operators are looking for strategic solutions to protect themselves against uncontrolled bot crawling, or at least to get paid for it. But first off, what is pay-per-crawl?

The series at a glance

Previous articles:

Next article (coming soon!)

  • Part 5: AI agents and web analytics: New rules for attribution

 

"Pay-per-crawl": Sending bots to the checkout

In early July 2025, Cloudflare, a major content delivery network (CDN) and security provider, made headlines with the introduction of a “pay-per-crawl” model.

The idea behind it? Website operators using Cloudflare – ca. 20% of all websites – can set a requirement that AI crawlers have to pay for access to their content.

Cloudflare blocks all AI bots by default on new sites. Website operators then have three options for each bot: Allow it to access the content for free, block it entirely, or charge it for access.

And how do they do that? The technical foundation for this is the long-dormant HTTP status code 402 Payment Required. When an AI bot requests a page, Cloudflare can check whether a “charge” rule has been configured for that bot on the domain. If so, the bot receives a 402 response instead of the content, along with a price. 

Only when the bot presents a payment token (with Cloudflare acting as the billing intermediary) is the request served with a 200 OK response and the content is loaded. In the current implementation, bots need to be registered in order to be identified and charged at the right level. Bots that don't have an established payment relationship effectively receive a 403 Forbidden response, along with an indication that they may be able to access the content in future if they pay.

For publishers, it's first time they've been able to monetize AI models' use of their content for training. According to Cloudflare, many major publishers and platforms have already positioned themselves in this direction, including Getty Images and News Corp. They want to be compensated when AI providers request access to their data.

Cloudflare pay-per-crawl creates a framework for this without requiring every publisher to negotiate individual contracts. Website operators set the price for requests at the domain level, while AI companies can decide whether they are willing to crawl at that rate.

In the short term, this model has no direct impact on your analytics data. A bot that is blocked or rejected never appears in your analytics in the first place, which is a good thing. In the long term, however, it could reduce overall crawling volume if payment barriers become more widespread.

From an analytics perspective, one particularly interesting possibility is this: If pay-per-crawl becomes the standard, we could eventually have economic metrics, such as “Bot X paid for 1,000 requests this month.” That would effectively provide evidence of the value of your content from the perspective of the AI industry.

But we are still a long way from that point.

For now, pay-per-crawl shows that the industry recognizes unlimited AI access as a problem and is looking for market-based solutions. Cloudflare itself puts it this way:

“If the Internet is going to survive the age of AI, we need to give publishers the control they deserve and build a new economic model that works for everyone”

–Matthew Prince, co-founder and CEO of Cloudflare. 

From an analytics perspective, this is a development worth watching. If you use Cloudflare, consider whether you want to participate in its AI crawler management, for example through Super Bot Fight Mode.

Cloudflare already offers the option to block AI bots by default until you explicitly allow them. It also offers a crawler verification program where bots have to indicate whether they're crawling for training or search purposes (those pesky two faces of AI bot traffic we've been talking about).

It also enables more granular blocking. You could allow a search bot while blocking a training bot, for example.

Together, these measures can significantly reduce the load generated by bot traffic, which in turn can reduce the burden on your analytics data.

Need support managing bot traffic on your site?

 

Bot deception and tar pits 

Beyond pay-per-crawl, the community is also developing other ideas to deal with bots –  effectively by tricking or redirecting them.

One particularly creative approach is a honeypot, also known as a bot labyrinth.

Cloudflare is experimenting with this as well through the Skywire project, also known as AI Labyrinth. Instead of just blocking bots, AI Labyrinth redirects them into an endless path of artificial pages. The bots don't realize they've been trapped, and are served page after page of automatically generated, worthless content.

The labyrinth pages are generated at the CDN edge, keeping this traffic away from your servers. Instead of draining your resources, the bot wastes its own crawling content with no real value. It keeps bots occupied and discourages them without affecting real user traffic.

From an analytics perspective, you should ideally also see these bot requests disappear from your actual traffic numbers because they never reach your origin server. But there's one important caveat: if a bot spends hours inside the labyrinth, it could generate thousands of hits that may still be counted somewhere.

The goal, though, is that these requests never make it into your main analytics property because Cloudflare routes them away.

Other administrators are using challenge systems. Developer Xe Iaso, for example, built a system called “Anubis” that presents every visitor with a computational challenge that takes around 30 seconds to complete before the page loads. 

You're probably thinking, "that sounds like a nightmare for human visitors," and you'd be right. That's why so far it's only been tested on a site with an extreme bot problem.

The result? Bots without specific adaptations give up because they cannot complete the challenge. Unfortunately, 90% of real users gave up too.

This tells us just how seriously the situation is being taken.

As a data analyst, the key is to work closely with your technical teams. If you try out mechanisms like these, be sure to decide in advance how you'll measure their impact. For example, you could track how many challenges were displayed and how many were completed, and compare it to traffic anomalies. Otherwise, you might end up wondering why you experienced a sudden traffic drop, when in reality, your users simply gave up.

 

 

Benchmarking and best practices

To finish our whistle-stop tour of this topic, let’s look at some concrete examples of how companies and projects are dealing with the AI bot problem.

 

Publishers and media companies

Many major publishers reacted quickly to bot traffic by using robots.txt to block specific bots from accessing their content.

For example, The New York Times, CNN, and others have blocked OpenAI’s GPTBot as well as other scrapers. Apple responded with Applebot-Extended, which allows websites to opt out of AI training without being removed from regular search.

More than half of news sites now use these opt-out options. Many of these publishers are also part of Cloudflare’s Crawler Governance Initiative: In the future, more than 20% of the web – based on Cloudflare’s share of the web – will be blocked from AI bots by default.

Major names such as BuzzFeed, TIME, The Atlantic, Quora, and Stack Overflow have joined Cloudflare to either implement pay-per-crawl or at least enforce stricter access controls.

From an analytics perspective, this means less bot traffic because it is blocked upstream, before it reaches the website, and a clearer overall picture, since most AI crawlers no longer receive the content from these sites. However, this is a deliberate strategic decision, which we'll explore in more detail in the next section.

 

Tech platforms and open source

Projects such as SourceHut, a developer-focused Git hosting platform, experienced dramatic performance issues caused by AI crawlers. At times, the volume of requests was so high that it resembled a DDoS attack (Distributed Denial-of-Service-Angriff.

Their solution: Network-level blocking.  For example, they blacklisted entire IP ranges belonging to Google Cloud and Microsoft Azure because most of the bot traffic originated from these networks.

Of course, this also blocks legitimate users who happen to access the site through these cloud networks. But for SourceHut, maintaining stability mattered more. Linux Weekly News experienced a similar “bot storm” and reported that AI bots were slowing down page loads for real readers, in some cases making the site effectively inaccessible.

Some projects have taken even more drastic measures. According to the ModX blog, some have even blocked entire countries because a disproportionate amount of scraping traffic was coming from certain regions and other measures had proven ineffective.

From an analytics perspective, this approach tackles the problem at its source: The traffic never reaches the website in the first place, so it doesn't need to be filtered later. However, such measures should really be considered a last resort.

 

E-commerce and smaller sites

Among mid-sized businesses and online retailers, increasing evidence suggests that conversion rate drops can correlate with bot waves.

One online retailer discovered that a “new referral” had suddenly appeared in Google Analytics and was generating a large amount of traffic. In reality, it was a bot leaving behind a UTM parameter that Google Analytics interpreted as a traffic source.

The solution was to add the relevant host to the referral exclusion list and add an additional GA4 filter targeting the “Campaign Source” using a regex for suspicious patterns.

Other online retailers report in forums that they have introduced CAPTCHAs on specific pages to prevent bots from continuing through the site, for example, before checkout.

The goal is to prevent bots from generating large numbers of test orders that are never completed but can still trigger events. From an analytics perspective, CAPTCHAs are a double-edged sword because they also create friction for real customers and can hurt conversion rates. However, in severe cases, they may be justified.

Overall, the pattern is clear: Most organizations use a combination of measures. They proactively block bots wherever possible through robots.txt, firewalls, or dedicated anti-bot services, while analytically filtering whatever traffic still gets through.

In other words, many organizations are taking a two-pronged approach: Protecting website performance while maintaining data quality.

If you want more information on how other companies are dealing with AI crawlers, and whether pay-per-crawl or your own bot management strategy could be relevant for you, get in touch and we'll support you.

 

Coming soon: 

  • Part 5: AI agents and web analytics: New rules for attribution