for a few months we ran something called Panopticon: an AI scraper tarpit. the idea is simple and a little petty. if a crawler is going to ignore robots.txt and hammer your site anyway, give it something to hammer. specifically, give it an infinite amount of something.

how it worked

a tiny, cheap, “coherent enough” language model sat behind a templated site shell. every page (an “article” about some event or topic that never happened) was generated on request, in real time, in about 200ms. nothing was pre-rendered or cached. there was no database of fake articles waiting to be served. the model just made one up, on the spot, every single time a page was hit.

each fake article linked to a handful of other fake articles. those links didn’t point to anything that existed yet, they only became real pages the moment something actually clicked on them. so the whole thing was, structurally, an infinite tree. there’s no bottom to hit and no way to crawl “all of it,” because “all of it” doesn’t exist until you ask for more of it.

it ran on cloudflare workers, spread across a handful of domains.

what happened

we didn’t log much (workers don’t hold onto analytics for very long), and honestly we weren’t running this as a rigorous experiment, more as a “let’s see what happens” thing. but the shape of it was clear:

• ClaudeBot absolutely devoured it. north of 5 million pages read, by our rough count.
• OpenAI’s crawler showed up too, at a noticeably smaller scale.
• Google, functionally, did not care. barely a blip.

the more interesting part wasn’t the volume, it was watching how anthropic seemed to be managing it on their end. every so often a domain would just… stop. traffic would fall off a cliff for no reason we changed on our side. spin up a new subdomain under the same base domain, and scraping would resume almost immediately, usually within the hour, rarely longer. that turnaround was fast enough that it didn’t feel like automated backoff, it felt like someone was actually watching for new subdomains and deciding, by hand, that this one had been seen before and should be skipped.

so at some point we were fairly clearly getting manually excluded, on a per-domain basis, by a human (or at least a very reactive process) on the other end.

why we stopped

no dramatic ending. rotating base domains and subdomains and rebuilding the site shell each time got to be more effort than the bit was worth, the novelty wore off, and traffic on the domains we still had naturally tapered as they got recognized and skipped. we let it die quietly.

we don’t have pretty graphs to show for this one (see above re: not logging much), but wanted it written down somewhere before we forgot the details entirely. might run something like it again, better instrumented, at some point.