The state of bot detection: what the research says

Automated traffic has outgrown the defenses most sites still rely on. Over the past two years a clear picture has formed across academic security venues about what actually separates a human visitor from an automated one — and about which of yesterday's defenses no longer hold. This note summarizes that public research and what it means for anyone defending a web property. It cites the primary sources so readers can check them; where a result comes from a preprint that has not yet been peer reviewed, we say so.

The TLS handshake is a weak verdict on its own

TLS fingerprinting (JA3/JA4) reads the shape of a client's handshake to guess what software made the request. In a clean lab it looks excellent: When Handshakes Tell the Truth (Jarad and Bicakci, arXiv:2602.09606, 2026, preprint) reports a gradient-boosted classifier reaching near-perfect separation on a curated dataset. The same authors are candid about the catch — they did not test evasion. Tools such as curl-impersonate and uTLS copy a real browser's handshake byte for byte, and once the handshake is borrowed the signal collapses. The practical reading: a TLS fingerprint is a useful reputation input, never a standalone verdict.

Detection now lives in the browser environment

The most consistent finding across recent work is that the decisive signals come from the authenticity of the browser environment, collected in the page, rather than from the network layer. On the Internet, Nobody Knows You're an LLM Bot (Fayolle et al., arXiv:2606.30119, 2026, preprint) measured real visits from a dozen automation tools against honeypot sites and compared detection accuracy layer by layer: the IP layer and the TLS layer each carried only part of the signal, while environment checks in the browser — automation flags, implausibly fixed screen dimensions, permission and hardware inconsistencies — carried the bulk of it. A striking secondary result was that "stealth" patches often made automated clients more detectable, not less, by introducing contradictions a genuine browser never shows.

Interactive challenges are cheap to defeat

If your defense is a CAPTCHA, the economics now favor the attacker. Broken Gates: Re-evaluating Web Bot Defenses in the Age of LLM Agents (Ousat et al., Florida International University, arXiv:2607.18659, 2026, preprint) reports that commercial solving services clear the common interactive challenges at a cost of cents per thousand. The lesson is not that challenges are worthless, but that a boolean "pass the puzzle" gate is no longer a barrier by itself. The direction the literature points to is adaptive difficulty — scaling the cost of a challenge to the measured risk of the request — rather than a fixed test shown to everyone (see, e.g., Adaptive CAPTCHA, MDPI Electronics 15(11):2363, 2026, peer reviewed).

Proof-of-work sits in a similar place. There is no peer-reviewed measurement of modern proof-of-work against headless browsers; vendor write-ups show attackers extracting the puzzle and solving it off the page on a GPU. Proof-of-work earns its place through cost asymmetry at volume, not as an on/off barrier.

Residential proxies blunt IP reputation

Attackers increasingly arrive through residential proxy networks, which launder automated traffic through millions of real consumer IP addresses (Resident Evil, Mi et al., IEEE S&P 2019, peer reviewed). Latency-based detection — for example comparing application round-trip time against network round-trip time (CalcuLatency, Ramesh et al., USENIX Security 2024, peer reviewed) — can flag a proxied connection, but newer work shows the signal degrades against a capable adversary (Beyond RTT, NDSS 2026, accepted). Treat IP-based latency as a probability, never a proof.

AI crawlers, robots.txt, and signed agents

Automated AI traffic is now a first-class category. Measurements show that well-behaved crawlers honor robots.txt while abusive ones ignore it entirely (Somesite I Used To Crawl, IMC 2025, peer reviewed; and arXiv:2505.21733, 2025, preprint). The takeaway is simple: robots.txt expresses intent but has no teeth, so enforcement has to happen at the edge, not in a text file. On the other side, an emerging standard — Web Bot Auth, built on HTTP Message Signatures (RFC 9421, IETF Proposed Standard) — lets legitimate agents cryptographically prove who they are. The bot-specific layer is still a draft, but the direction is promising: a future where good automation identifies itself and everything else is treated as unverified.

What this means for defenders

The research converges on a handful of principles:

1. Classify passively and cheaply first; most traffic can be judged before any challenge.
2. Reserve interactive challenges for genuinely ambiguous traffic, and scale their difficulty to risk.
3. Never hard-block on a single signal — every individual signal, from TLS to IP latency, is forgeable or noisy.
4. Weigh many weak signals together rather than trusting any one of them.
5. Enforce at the edge. Intent files and client-reported claims are starting points, not decisions.

No single check is decisive anymore. Robust defense comes from combining independent, hard-to-forge signals and from raising the attacker's cost rather than hunting for one perfect test. That is the direction the public research points to, and it is the direction we build toward at Qlyne.

Sources

• Jarad, Bicakci. When Handshakes Tell the Truth: Detecting Web Bad Bots via TLS Fingerprints. arXiv:2602.09606, 2026 (preprint).
• Fayolle et al. On the Internet, Nobody Knows You're an LLM Bot. arXiv:2606.30119, 2026 (preprint).
• Ousat et al. Broken Gates: Re-evaluating Web Bot Defenses in the Age of LLM Agents. arXiv:2607.18659, 2026 (preprint).
• Adaptive CAPTCHA. MDPI Electronics 15(11):2363, 2026 (peer reviewed).
• Mi et al. Resident Evil: Residential IP Proxy as a Dark Service. IEEE S&P 2019 (peer reviewed).
• Ramesh et al. CalcuLatency. USENIX Security 2024 (peer reviewed).
• Beyond RTT: Adversarially Robust Two-Tiered Residential Proxy Detection. NDSS 2026 (accepted).
• Somesite I Used To Crawl. IMC 2025 (peer reviewed).
• Scrapers selectively respect robots.txt. arXiv:2505.21733, 2025 (preprint).
• HTTP Message Signatures. RFC 9421, IETF (Proposed Standard).