Part 3: Filtering AI bot traffic: How to clean up your data in GA4, Adobe & Piano

AI traffic series – Part 3 of 5 

There are multiple ways to stop bot traffic from affecting your data, or to at least erase its impact afterwards.

In the following post, we'll give you an overview of your options – from built-in features in analytics tools to new, innovative approaches – including their respective pros and cons. We'll also discuss whether it's easier to flip the approach on its head and isolate real users rather than desperately trying to keep every bot out of your data.

Series overview 

Previous articles:

• Decoding AI traffic: A series for analytics professionals

• Part 1: AI crawlers and AI SEO: The two faces of AI traffic

• Part 2: AI traffic and data quality: How AI bots distort your data

Coming soon:

• Part 4: Pay-per-crawl & co.: How companies are responding to AI crawlers

• Part 5: AI agents and web analytics: New rules for attribution

Automated bot filtering in web analytics tools: Google Analytics, Adobe Analytics and Piano 

 The easiest way to defend your data against bot traffic is to use the built-in bot filtering methods in your analytics tool. 

Google Analytics

Google Analytics has long offered an option to filter out known bots and spiders. In Google Analytics 4 (GA4), this selection is active by default.

GA4 automatically filters known bots and spiders out of its reports, based on an IAB list that is updated regularly. As a result, hits from bots that can be clearly identified through their user agent don't even show up in GA4 data.

In Universal Analytics (GA3), you had to select this option manually in the view settings; in GA4, it happens by default. However, GA4 can't catch every single bot – newer bots in particular, or those that aren't on the standard list, aren't as easy to filter out, so you’ll probably find yourself needing additional custom filtering (more on that below!).

Adobe Analytics

Adobe Analytics also offers bot filtering functions. Adobe likewise uses the IAB/ABC International Spiders and Bots List, which identifies known bots through their user agents.

The Adobe admin settings (report suite settings) include a section on bot rules. Here you can activate the standardized bot filtering, which automatically excludes bots from your reports.

Adobe also offers individual bot rules (more on this later), but even the basic function saves a lot of work.

Important to know: Hits that were identified and filtered as bots only show up in special bot reports, not in the normal traffic numbers. Adobe treats bot data separately, so it doesn't distort your data.

Piano Analytics

Piano Analytics (formerly AT Internet) also uses automated filtering. According to the Piano documentation, analytics data collection is scrubbed by default of all hits that appear on a robot list maintained by an organization – presumably the IAB list as well.

Piano filters bots with the help of IP addresses and user agent patterns, so they don't show up in your data. However, Piano also points out that newer bots may not be identified right away. Thus, an unusual surge in traffic that could be traced back to a bot might mean that the automatic filter isn't working yet. In this case, additional custom filtering is needed.

Automated filtering:

These out-of-the-box solutions should be your first line of defense. They save a lot of work – in GA4 the option is active by default, while in Adobe and Piano you should check whether the option is selected. And they exclude a whole bunch of known bots from your reports.

However, don't take this at face value. AI crawlers love to change their name and their disguise! Given the dynamic bot landscape, something can always slip through the cracks. That's why many organizations we work with also rely on additional custom filtering or segmentation on top of the basic filters.

Custom filtering and segmentation for AI bots

 For more granular control, you should apply custom filtering rules tailored to the AI bots that visit your website. The key element here is the user agent string: every time a bot sends a request to a server, it also transmits its user agent header, which usually contains the bot's name or purpose, e.g., GPTBot: 

You can use this ID to filter out or flag the corresponding bot. 

GA4

In Google Analytics 4, you can use RegEx filters.

One tip from experience: In GA4, use the 'User ID' or 'Browser' field in a comparison or exclusion filter. A suggested regex for uncovering suspicious bots looks like this:

This code would locate every user agent or traffic source name that includes words like "bot", "crawler", "AI", "OpenAI", "ChatGPT", or "Bard". With it, you can apply filtering in GA4 that excludes every hit with this kind of pattern before it can show up in your reports.

Alternatively, you could use this regex for exploration or segmentation, so that it filters out bot sessions and displays them separately. Bear in mind: a pattern that's too broad can also exclude legitimate traffic. For example, the string "…ai…" could also appear in the user agent of a human browser.

Test your filters thoroughly and on an ongoing basis, so that you exclude neither too few nor too many page views from your data. That said, filtering by user agent only works when bots are polite enough to announce themselves. Theoretically, nothing stops them from sending any string they choose with the request.

Adobe Analytics

 In Adobe Analytics, you can also add custom filters to your liking. For example, you could create a rule saying: 

Adobe allows combined conditions with an "or" link. You could even enter IP address ranges if you notice specific cloud data centers, like AWS or Azure, sending a large amount of bot traffic your way. However, user agent-based filtering is more effective because bots change their IPs frequently.

Important: Adobe also offers the option to upload a CSV list of bot rules – which is useful if you want to maintain a curated and frequently updated list of known AI crawler strings. Bot filtering is then applied consistently to all hits.

Piano Analytics

In Piano Analytics, you can configure custom traffic exclusions in the Data Management module.

According to the Piano documentation, you can set criteria such as city, organization (ISP), or even user-agent to exclude traffic. For example, if you notice that 90% of suspicious traffic comes from a specific ISP—such as “Google Cloud”—you could temporarily exclude that entire ISP.

But once again, a filter that's too broad will hit genuine users too, because the ISP also carries legitimate proxy connections. So it's better to keep your criteria narrow: for instance, user agent contains 'GPTBot' or 'python-requests', for those bots that don't even bother to pose as a browser.

If Data Management isn't enough, Piano also offers a support team you can contact to blacklist certain IP ranges in coordination with them. This manual step is a bit more cumbersome than in GA or Adobe, where you can handle it yourself directly in the UI.

Another best practice is to create segments for bot traffic rather than deleting everything outright. Why? Because then you can observe bot traffic trends on your site without interfering with the main metrics.

You could, for instance, define an 'AI Bots' segment using criteria such as the user agent matches mentioned above, or country = 'not set' where bots don't provide any geodata – and alongside it a 'Human Users' segment, for example: browser contains 'Chrome|Firefox|Safari|Edge'.

In your analyses, you can then either report on the human segment alone or show both separately to see whether bot activity is currently spiking. That transparency helps raise internal awareness ('30% of our traffic last week came from AI crawlers') and lets you check how effective your countermeasures are.

In summary:

With custom filtering and segmentation, you can identify and exclude the majority of AI traffic from your analytics data. It takes some initial work – collecting the relevant patterns, testing the regex, etc. – but it’s worth the time investment.

Don't hesitate to combine multiple approaches: technical characteristics (user agent, headless browser indications), behavioral characteristics (e.g., more than 50 pages per session in under a minute – something no human would ever do), and origin (cloud provider IP). The latter can be identified through log file analysis.

Be sure to regularly check your filters as well. The world of AI bots is constantly evolving; new bots are emerging, and old ones are changing their fingerprints.

One example: According to BuzzFeed, as of 2026, 79% of top news sites block AI training bots via robots.txt. But bots that find themselves blocked may well just start crawling under a more generic agent instead. It's a game of cat and mouse.

We can set up bot filtering for you:

Get in touch

Server-side analysis: Recognizing bot traffic in logs

Another way to manage AI traffic is to analyze server log files. Unlike client-side analytics, which rely on JavaScript and tracking pixels, the web server logs every request in minute detail, including those from bots. Looking through your access files can help you unmask suspicious activity, for example:

• User agent patterns: Do agents like “ChatGPT” or “python” appear there, or is there a conspicuously high number of user agents without the usual browser identifiers?
• Request frequency: If you see hundreds of requests from a specific IP within seconds, that’s typical bot behavior.
• Distribution across URLs: Bots tend to request less popular or deeply nested URLs. If /sitemap.xml or large numbers of paginated pages suddenly show up among the top 100 requested URLs in your log files, that's an indicator.
• Status codes: A suspiciously high number of 404 and 410 errors might indicate that crawlers are testing hallucinated paths or crawling through hidden parts of your site.

Log analysis is how the SourceHut team, for example, noticed that AI crawlers were pushing their service to its limits. It was visible in the IP ranges, many of them from Google and Azure, and in the sheer frequency of the requests. They drew the obvious conclusion and blocked entire cloud provider networks.

The open-source documentation platform Read the Docs likewise reported that blocking known AI crawlers reduced its traffic by 75%, from 800 GB to 200 GB per day, saving about $1,500 a month in bandwidth costs. These numbers show the massive impact server monitoring can have, and how valuable the insights from this approach can be.

As the person responsible for analytics, you can team up with your DevOps or hosting team on this. Set up log file dashboards (e.g., via Kibana or Splunk) that visualize suspicious traffic. Or use small scripts that regularly evaluate the top user agents and IPs. You can then feed that data back into the filters described above and update them in a targeted way.

Moreover, if your bot filters at the analytics level aren't enough, you can also consider server-side measures to rein in aggressive crawlers: robots.txt, rate limiting, or firewall rules.

How about turning the tables: Should you filter for human users instead of filtering to exclude bots?

Given the growing number and diversity of bots, you might ask: could it be more efficient to isolate the real users and focus entirely on them, instead of hastily chasing the new bots out of your data? In other words: couldn't we just treat every interaction as a bot by default and only mark those that fulfil specific criteria as real and human?

In fields like ad fraud detection or security, similar approaches already exist – under the heading of challenge response: users have to execute JavaScript or solve a CAPTCHA, for instance, to pass the test and continue.
Applying this to web analytics in the narrower sense can be tricky, but here are a few thoughts:

Browser abilities: A real user typically browses with a standard browser that accepts cookies, executes JavaScript, and renders CSS. With that in mind, you could mark all sessions that don't generate JS events as suspicious.

If you use custom events (scroll depth, button clicks, etc.), real users will almost always have entries, while AI crawlers won't.
You could also segment on 'session contains at least one interaction event' versus 'session contains none'; the former are highly likely to be human.

User journey: Humans usually show relatively "uneven" behavior. Their time on a page varies – seconds here, minutes there – and while they often work their way from landing pages into deeper levels, they seldom visit every page along the way.
Bots, on the other hand, either have extremely uniform patterns (a request every few seconds for hours on end) or perform superhumanly fast actions (100 pages in five seconds). With a little data science, patterns like these could be clustered automatically. In practice: you can define, say, that sessions with more than X page views per minute, with identical intervals between hits, or with unusually high crawl depth are more likely to be bots. Everything else gets counted as 'human-like'.
Sources and technique: The vast majority of real visitors arrive via familiar traffic sources like search engines, direct visits, or social media referrals.
Many bots, however, don't have a referrer or arrive from obscure sources. Real visitors also tend to arrive on mainstream devices and current browser versions. A sudden 5,000 sessions from something like 'Firefox 48 on Windows XP' – a combination that's years out of date – almost certainly isn't human. So you can be confident in filtering out combinations like these.

Verification through actions: You could also opt to link your key metrics to actions that bots don't carry out. For example, since bots don't usually fill out forms, you could calculate your conversion rate based on submitted forms rather than on total sessions. This would give you an indicator that's less susceptible to error.
But this whitelisting-style option comes with a catch: It fundamentally changes the view of your data, and poses the risk of cutting out real users by mistake – those who only stayed on a page for a few seconds, for example. Aside from that, bots use tricks to seem more human; for example, some use headless Chrome browsers to be able to execute cookies, JavaScript, etc. You can recognize them by the string "HeadlessChrome" in their user agent, but they would still pass your "humans-only filter".
Nonetheless, working with positive indicators can be a useful complementary approach. For instance, you could define a KPI in your reports along the lines of 'verified user visits', counting only those sessions that registered a JS-based heartbeat or an interaction. This would give you a metric guaranteed to be bot-free – one that shows how genuine user numbers are changing, even when raw traffic fluctuates.

Tips from our experts: Parallel tracking and bot sandbox

Some organizations take a more pragmatic path: they separate bot traffic from the start with the help of parallel tracking properties. For example, you route all hits carrying certain bot signatures into property B, and everything else into property A, your main analytics property. In GA4, this could be handled through server-side tagging or corresponding filter logic.

That gives you a kind of sandbox for observing bot activity. Perhaps you'll find certain crawling patterns interesting, while your main data stays clean.

If you take this approach, be sure to communicate internally about what you're filtering. It’ll help you avoid an awful moment where someone asks, "Why are our visitor numbers down?" in a management meeting and nobody remembers that filtered-out bot traffic is the culprit. Explain to stakeholders that gross visits and cleaned visits are two different things, and that a drop in the cleaned figures actually represents an improvement in data quality.

How to usefully segment your AI traffic

With these approaches in your toolbox, the next question is how should AI traffic be categorized once it's made it into your reports? Not every bot is the same – and segmentation can help you respond to problems in a more targeted way. Here are a few segments worth creating:

By bot purpose

You could separate two categories: training bots, which crawl everything and everyone for AI models, and search or assistant bots, which show up selectively to fetch an answer for a user.
Training bots like GPTBot, ClaudeBot, or Common Crawl typically hit at high frequency, crawl broadly, and come back repeatedly.
Assistant bots like ChatGPT-User, Bing's chat agent, or Bard show up sporadically and with a clear purpose, often carrying a referrer or a distinctive query parameter.

Segmenting these groups separately lets you see whether your content is frequently being pulled into ChatGPT queries – which arguably counts as a positive signal.

By bot "family" or origin

You could also create segments based on the big players: OpenAI bots, Anthropic bots, Google bots (incl. Googlebot & Google-Extended), and others like Meta or Perplexity. This gives you an internal report on who uses your content the most. Maybe you'll discover that OpenAI is responsible for 70% of the total bot hits – then you know who to talk to when a discussion about pay-per-crawl becomes relevant for you.

By impact on performance

Another option is to segment bot sessions by intensity: heavy hitters (bots with 1,000 requests per day) vs. light bots (with sporadic hits). The heavy hitters are the main culprits when it comes to data quality and server load.

By whether they respect your robots.txt

You could also track whether bots respect your robots.txt – for instance, by adding a /trap-page entry with a Disallow rule. If that page gets crawled anyway, the bot is ignoring your robots rules. Bots like these deserve special attention: block them.

Valuable vs. less valuable AI traffic

Some bots bring indirect traffic – e.g., Bing Chat, by mentioning your website in a generated answer that the user then clicks on. Others only take what they need, like GPTBot. A bot arriving with a bing.com referrer may mean that Bing briefly crawled your page in response to a user's query – potentially valuable, since there was a real person behind it.

The technical implementation of this segmentation again runs through filter or segment conditions:

• In GA4, you could pass a custom parameter bot_type along with the hit if you're able to detect it server-side, or classify it retrospectively using regex on the user agent.

• In Adobe, you might reserve an eVar and map the bot type into it via a processing rule.

• In Piano, a calculated field does the same job.

  The value of segments like these lies in fine-tuning both your defenses and your analysis. You might decide to block training bots outright while continuing to allow search bots, because you want the traffic that comes via Bing Chat, for instance. Your analytics can then show you whether that decision is having the intended effect.

   

Steps in Google Analytics, Adobe Analytics and Piano Analytics

Now let's look at exactly how you can exclude or segment bot traffic in the big three analytics tools.

Google Analytics 4

• Check the standardized bot filter:  In GA4, bot filtering is enabled by default. However, it doesn’t hurt to check the property settings to see if the “Exclude known bots” box is checked, if available. For GA4, this option no longer exists at the UI level, as it is enabled by default.

   

• Exclude internal and dev traffic: Bots aside, don't forget internal traffic. Visits from your own team and from development environments should be filtered out too, so they don't muddy the picture. GA4 offers traffic filters by IP or by parameter for exactly this purpose.

• Custom definitions: As things stand, GA4 unfortunately doesn't let you view the user agent directly as a dimension in reports. There is a workaround, though: use GTM or gtag to implement a custom field that reads the user agent from navigator.userAgent and sends it to GA4 as a custom dimension. Better still, send only a flag such as isBot = true/false via JavaScript, which keeps the data volume down.

   

• Regex data filters: In GA4, consider adding a further rule under Admin > Data Streams > Configure Tag Settings > Define Internal Traffic. Some teams tag bot traffic as 'internal traffic' by setting a traffic_type parameter when the user agent belongs to a bot – at which point GA4 filters it out at property level. Hacks of this kind take a bit of technical tinkering, but they do the job.

   

• Use explorations: In GA4's Explore section, you can apply ad-hoc filters – for example, Browser contains 'Bot' – to get a sense of how much would actually be filtered out.

Adobe Analytics

• Activate IAB bot lists: Make sure the "bot filtering with IAB lists" option is active in your report suites. You can find it under Report Suite Settings > Bots > Known Bots.

• Set up custom bot rules: Go to Admin > Report Suites > Your Suite > Edit Settings > General > Bot Rules and click "Add Rule". Typical entries are: user agent contains "GPTBot", or user agent contains "Claude", etc.

• Don't forget to click save: Adobe will then start applying the rules within roughly 30 minutes. Historical data will stay as is – bot rules don't work retroactively, only on newly incoming hits.

• Test reports: Adobe provides 'Bots' and 'Bot Pages' reports. It's worth looking at these to check whether your rules are actually taking effect.

Piano Analytics

• Data manager exclusions: Go to Data Management > Exclusions and create new exclusion rules – for example, 'user agent contains GPTBot; action: exclude'.

• Reaching out to support: Where a stubborn bot keeps slipping through, gather the evidence and bring it to Piano's support team.

• Segmentation: Create segments such as 'Exclude Bots' using the same criteria as above, so that you can switch between views in your standard reports with a simple dropdown.

Whichever tool you're using, it's advisable to try changes out in a test or staging environment first. Filters that discard traffic outright deserve particularly careful checking. There's nothing more frustrating than discovering you've accidentally filtered out real people.

Want to get AI bot traffic under control in GA4, Adobe, or Piano? We can support your team and work alongside them on everything from the technical setup to figuring out what your numbers are telling you afterwards. 

Contact us

The next article in this series is coming soon! It's already available on the German version of our blog:

Teil 4: Pay-per-Crawl & Co.: Wie Unternehmen KI-Crawlern begegnen