New York's crawler bill is an identity mandate for machine traffic

EFF and 17 other organizations have asked Governor Hochul to veto S9934A, New York’s Stealth Crawler Prohibition Act. The bill would require every web crawler to disclose its identity and explicit purpose, and would let media outlets obtain judicial subpoenas to unmask unidentified automated traffic “without any showing of misconduct or actual injury.” The coalition’s framing: legislation sold as protection for local journalism that instead works by “deanonymizing and criminalizing automated access to the open web.”

Strip the AI-politics packaging and this is a mandatory identity-attestation regime for non-human traffic, with a private unmasking mechanism attached — which is why it belongs on this beat even though nobody involved is talking about telecom. It’s the same question the agent-identity work is circling from the other direction: when a request arrives from software rather than a person, what must it assert about itself, who gets to compel that assertion, and what happens to the requests that decline. Most of the identity world has been assuming the answer trends toward more attribution for automated actors. Here’s a bill that legislates it, and the civil-liberties objection lands squarely on the attribution itself rather than on any downstream misuse.

The examples in the letter are the useful part. The Markup’s crawlers presented as ordinary Firefox browsers specifically so Amazon’s search results would look the way they look to a real shopper; ProPublica simulated a customer to catch pricing steering; EFF’s own Privacy Badger crawls anonymously to find trackers without exposing users. In each case the anonymity isn’t incidental to the method — it is the method, because a crawler that announces itself gets served a different page. Any attestation scheme for automated traffic has to reckon with that, and “just identify yourself” gets a lot less clean once identifying yourself changes the measurement.