How do I stop AI from scraping my website?

Something fundamental has shifted on the web: the majority of traffic hitting your small business website is no longer human. Automated bots now outnumber people online for the first time in a decade, and most of that machine traffic isn't visiting to send you customers.

According to Imperva's 2025 Bad Bot Report, automated traffic now accounts for 51% of all web traffic, with bad bots alone representing 37% of everything moving across the internet. By June 2026, the gap had widened further — tracking data shows automated requests making up 57.5% of HTML web traffic versus 42.5% for humans.

AI crawler traffic specifically surged roughly 757% across 2024, and the pace hasn't slowed. Cloudflare's CEO Matthew Prince summed up the milestone bluntly: "Welp, that happened faster than I predicted."

Here's what makes this surge painful for small business owners: most of that crawling serves no purpose that benefits you. Cloudflare attributed 51.8% of AI crawler requests to training purposes in May 2026, versus only 9.3% for search. The other 90% is essentially extraction — your content harvested to improve someone else's model, with little or nothing sent back.

The referral imbalance is staggering when you compare bots side by side:

• Google's traditional search crawler: roughly 4.9 pages crawled per referral
• Perplexity: approximately 111 pages crawled per referral in Q1 2026
• ClaudeBot: 23,951 pages crawled per referral — and by mid-2025, 70,900 requests per visit

As Cloudflare put it, "AI was taking everything and sending back nothing." For a small business site built to attract local customers — the kind of site we build and manage at AI Business Sites — that's bandwidth, server resources, and content consumed with no return.

This isn't just an annoyance. Every training bot request consumes hosting capacity you pay for, and every scraped page is content you invested in that may surface in a competitor's AI answer without a link back to you. Over 2.5 million sites had fully disallowed AI training as of August 2025, and Cloudflare customers blocked 416 billion AI scraping requests in just five months.

Yet blocking everything isn't the answer either. As Cloudflare notes, "If you run a small site, the problem isn't just that someone could train models on your content — it's that nobody can find you in the first place." The challenge is protecting your content without vanishing from the search results and AI answers where customers actually look.

Blanket-blocking every bot feels safe, but it's a blunt instrument that can quietly strangle your search visibility. The smarter play is granular control: let the crawlers that send you customers through, and shut the door on the ones that just take.

The numbers explain why this matters. Training, not search, drives most AI crawling — Cloudflare attributed 51.8% of AI crawler requests to training purposes in May 2026, versus just 9.3% for search, according to aggregated crawler statistics. Meanwhile, the referral imbalance is stark: ClaudeBot crawled 23,951 pages per referral in Q1 2026, while Google's traditional search crawler ran a reasonable 4.9:1 ratio. That's the difference between a bot that harvests and a bot that funnels visitors to your door.

Cloudflare's Bot Management suite now reflects this reality. Its taxonomy separates AI traffic into Search, Agent, and Training categories, so you can block the harvesters without touching the discoverers. As Cloudflare's announcement explains, new domains will see Training and Agent bots blocked by default on ad-supported pages starting September 15, 2026, while Search bots stay allowed. Cloudflare is also testing a "content use" signal in robots.txt with three levels — immediate, reference, and full — giving you control not just over whether a bot visits, but what it does with what it finds.

One critical caveat: multi-purpose crawlers like Googlebot combine search and training functions. Cloudflare applies the most restrictive applicable rule, so blocking Training can block Googlebot entirely — even if Search is allowed. As Cloudflare CEO Matthew Prince put it, "You can't opt out of one without opting out of both, which is crazy."

Before you block anything, measure the value first, as search experts recommend. A practical audit looks like this:

• Analyze server logs to identify which bots are hitting your site and how hard.
• Compare crawl volume against actual referral traffic and conversions from each source.
• Check whether your brand appears in AI-generated answers — awareness value counts even when clicks are low.
• Use WAF-level blocking for non-compliant bots that ignore robots.txt directives entirely.

The goal is precision, not lockdown. A site that gets found on Google but stops feeding free training data to models that send nothing back is winning the exchange. That balance — visible to customers, invisible to freeloaders — is exactly what we build into every AI Business Sites launch, where search discoverability is the priority from day one. Getting found and staying protected aren't competing goals; they're the same strategy done carefully.

For most small business owners, the question isn't whether to block AI crawlers — it's how to do it without breaking the search visibility that brings in actual customers. The good news: you don't need a dev team to implement meaningful protection. Cloudflare's new defaults, rolling out September 15, 2026, will automatically block Training and Agent bots on ad-supported pages while keeping Search bots allowed for discoverability — but multi-purpose crawlers like Googlebot will be blocked if you restrict Training, since they combine both functions. This makes granular control essential.

Start with three practical steps that work on any platform:

• Enable Cloudflare's free tier and turn on "Block AI Scrapers and Crawlers" — it stops known training bots at the edge before they hit your server.
• Add an llms.txt file to your root directory — adoption reached 10.13% of nearly 300,000 domains surveyed in 2026, and it signals your preferences to compliant crawlers.
• Use WAF-level rules to block non-compliant bots like ChatGPT-User and Perplexity-User that ignore robots.txt entirely.

Then measure what matters. Cloudflare attributed 51.8% of AI crawler requests to training purposes in May 2026 versus only 9.3% to search, and Anthropic's Claude makes 70,900 page requests for every one website visit. Check your analytics for the "AI Assistant" channel — Google Analytics now tracks referrals from ChatGPT, Gemini, and Claude — and compare crawl volume against actual conversions. If AI-referred visitors convert 42% better and spend 48% more time on-site (as retail data showed in March 2026), that traffic may justify the crawl cost.

AI Business Sites handles this configuration during the 7-day build: we set up Cloudflare, configure the managed robots.txt with the new content-use signals (immediate, reference, full), deploy llms.txt, and implement WAF rules for non-compliant bots — so your site launches with protection baked in, not bolted on later. The monthly management includes monitoring crawl-to-referral ratios and adjusting rules as bot behavior evolves, because the landscape shifts faster than any static block list can keep up.