How Regex101 Keeps Bots Out of Its Ad Numbers

Regex101 gets crawled constantly. It serves a large developer audience, its pages are full of shareable permalinks, and that makes it a standing target for every scraper, scanner, and headless browser on the internet.

Firas Dib, who built and maintains it, now runs four layers between that automated traffic and his ad numbers: known-bad IP blocklists, CrowdSec behavioral rules, Cloudflare challenges for anything suspicious, and one change to when the ad code runs.

He built all of it after a conversation with us about what his impressions looked like. We brought the anomaly; he found what was causing it.

TL;DR: Automated traffic was inflating Regex101's ad impressions. Rather than each side guessing from its own half of the data, Firas and Carbon worked the root cause together. He shipped four mitigations. The last one, holding the ad request until a visitor has stuck around, transfers to almost any site.

---

Clean numbers are worth defending

Advertisers buy on CPM, or cost per thousand impressions, so every counted impression is money moving. When a bot renders a page and triggers the ad code, that money moves without a human ever seeing the ad. The advertiser paid for nothing, and it eventually shows up in what advertisers are willing to pay everyone.

Publishers sometimes hear "invalid traffic" as an accusation. It almost never is. Scrapers, headless-browser crawlers, and, increasingly, AI agents fetching pages on someone's behalf hit any popular developer reference around the clock, and nobody asked the site owner first. Some of it does come from inside the house: your own uptime monitor and security scanner are automated visitors too, and they are worth excluding from ad requests deliberately rather than leaving them to be filtered as though a stranger sent them.

Those impressions are also the denominator under every performance number a publisher looks at. We put real figures on what good looks like in what CTR actually looks like for developer ads; a benchmark is only worth as much as the impression count underneath it.

But the numbers still have to be defensible, for the publisher as much as for the network. Impression volume that moves in ways a site's actual audience doesn't gets looked at, and a publisher who can point at the cause, whether that is SEO compounding over months, a Google algorithm change, or a post landing on the Hacker News front page, is in a much better position than one who can't account for the change at all. Invalid traffic that stays unresolved can eventually put ad delivery on hold, which is what briefly happened here while we sorted it out. A publisher who filters hard is protecting their own CPM and their own reputation with buyers.

What Regex101 is

Regex101 is an IDE for regular expressions — building, testing, and debugging them across multiple flavors, with a pattern debugger, a code generator, and a community library of shared patterns. It was created, developed, and is maintained by Firas Dib, and in the repository wiki's words it started as "a simple hobby project which has now grown to become one of the largest regex testing services in the world."

Heavy traffic, a technical audience, and endless linkable URLs together make close to the worst case for automated traffic. A scraper that only fetched raw HTML was never much of a problem for ad measurement. A scraper that executes JavaScript looks, to an ad script, exactly like a reader.

The conversation came first

We could have handled this the way ad networks usually do: send a notice, describe the anomaly in the vaguest possible terms, and leave the publisher to guess. Firas would have been debugging blind against numbers he couldn't see.

So we asked for a call instead. We walked through what the traffic looked like arriving at our end; he went through what his logs showed on his. Neither view was enough on its own. A publisher can see request patterns, user agents, and behavior over time that a network never will. A network can see impression anomalies across hundreds of sites and knows what normal looks like. Invalid traffic sits exactly in the gap between those two views, which is why it's so rarely solved from one side.

Firas designed the fix. It's better than anything we'd have specified from the outside, because he knows his traffic and we don't. The call was worth as much as the fix. A publisher who has troubleshot with us once raises the next anomaly early, before it costs either side a pause.

What Firas built

His mitigations, in his own words:

Traffic filtering is a complex problem, and achieving the desired results required a multi-layered approach. The abusive traffic did not originate from a single source, ip range, or actor - but rather a distributed network composed of multiple actors. Some legitimate crawlers, some questionable automated traffic, and some clearly abusive.

1. I used abuseipdb.com to block IP addresses with established abuse histories.
2. I deployed crowdsec, an open-source security engine, with custom rules designed around the patterns I had discovered in the abusive traffic
3. Traffic that was suspicious, but did not warrant an immediate block, was asked to complete a Cloudflare challenge in order to proceed.
4. Perhaps most crucially, as a final layer of defense, I wait a brief period of time after loading the site before requesting the advertisement from Carbon. This prevents short-lived automated visits from generating ad requests while having little effect on genuine visitors who remain on the page.

Firas Dib, creator of Regex101

Each layer does a different job, because the traffic never came from one place. Firas found a distributed mix: some legitimate crawlers, some questionable automation, some clearly abusive. No single block rule covers that spread.

Known-bad IP blocklists are the cheap baseline. Dropping addresses with established abuse histories at the edge removes traffic that is already known-bad to somebody else. It costs almost nothing and clears out the least sophisticated share of the problem.

Behavioral detection is where the real work happens. CrowdSec reads request patterns rather than reputation, so it catches traffic that has never been reported anywhere, including the scraper running from a residential proxy pool behind a plausible user agent. Firas wrote custom rules around the patterns he found in his own abusive traffic. The traffic that hits your site is shaped by what your site does, and no off-the-shelf ruleset knows that.

A tiered response handles the two cases differently. Clearly abusive traffic gets blocked; traffic that looks suspicious without warranting a block gets a Cloudflare challenge instead. That distinction makes aggressive filtering survivable, because a false positive costs a real user one interstitial instead of a locked door. Sites whose only verb is "ban" end up tuning their rules too loose, because being wrong is too expensive.

Moving the ad request later

Regex101 now waits a short period after the page loads before it requests the ad.

The change does something the other three layers can't. Blocklists, behavioral rules, and challenges all work by identifying the bot. This one sidesteps identification. It moves the ad request to a moment most automated clients never reach.

A crawler that executes JavaScript will typically render enough of a page to extract what it came for and move on, because it's optimizing for throughput across millions of URLs and has no reason to linger. Someone reading a regex explanation is there for a while. A short delay between page load and ad request separates those two populations without having to tell them apart.

Be clear about what the delay costs, because it is the same mechanism that filters the bots: on a visit shorter than the delay, the ad request never fires at all. That is a lost impression, not a shortened one, and fast-bouncing humans are filtered exactly like crawlers. On a reference tool where people sit and work through a pattern, Firas reports it having little effect on genuine visitors. On a site where sessions last seconds, you will count meaningfully fewer impressions. Know which one you are before copying the approach.

If you run a high-traffic developer site

Work through it in roughly this order:

1. Check whether you have the problem. Don't reach for client-side analytics to do it. A crawler that executes JavaScript will often fire your analytics as well as your ad code, in which case page views and impressions inflate together and the ratio between them stays reassuringly flat. How closely they track depends on your setup, since analytics can be blocked, consent-gated, or fired at a different point in the page lifecycle than the ad request, which is exactly why the ratio is a weak thing to lean on. Your server logs are where the answer is, and a behavioral engine like CrowdSec is built to read them. Then ask your network what they're seeing, because their view of the anomaly is the half you don't have.
2. Drop known-bad IPs at the edge. Ingesting a public blocklist is a small job that raises the floor immediately.
3. Add behavioral detection, then write custom rules for it. Rules describing how your site gets abused are what actually work.
4. Give yourself a setting between "allow" and "ban." Challenge the suspicious. It's what lets you tune the rules tight enough to matter.
5. Look at when your ad code runs. If the ad request goes out the instant the page renders, every JavaScript-executing crawler you didn't catch is counting as a reader. Expect your impression count to drop after you add a delay. That fall is the fix working, not a new problem.
6. Talk to your network early. We can't see your logs; you can't see the anomaly across the network. The conversation is worth having before either side has a problem to solve.

That last one is the step publishers tend to hesitate over, expecting the conversation to be adversarial.

How we'd rather work

Carbon is a curated network, usually described by who we let in. The other half is harder to put on a homepage. When something goes sideways on a hand-picked site, the job is to solve it with the publisher rather than around them. A network of ~350 active sites can work that way. It's one of the real advantages of not being enormous.

Regex101 came out of this with cleaner numbers, a hardened site, and mitigations that will keep paying off long after the traffic that prompted them has moved on. If your impressions look strange, or you want a second pair of eyes on them, ask us.