Home | Blog | Clownflare

Cloudflare and User Agent Sniffing

It looks like Clownflare has tweaked their User Agent sniffing routines.

Previously, they only blocked requests from the exact string used by actual Big Goog', but now any UA string containing the (case-insensitive) googlebot will be blocked. That's a bummer, as I used this technique to get around paywalls in the past and cookiewalls nowadays.

Yet, triggering a Sorry, you've been blocked message still doesn't blacklist your IP or anything. Re-requesting with another spoofed User Agent (like ia_archiver) works every time. The rest of their "Web Application Firewall" looks to be similarly (non-)effective.

How this is anything but the crudest of snake oils, I don't get.

More on this topic from others: