207
Google’s AI Is Destroying Search, the Internet, and Your Brain
(www.404media.co)
This is a most excellent place for technology news and articles.
I run a few personal sites with some being available to the wider internet, and I have found that they generally do respect it. All the bots which advertise themselves as OpenAI, Anthropic, or Google do obey. However, I do occasionally see a stealth bot appear which uses a normal browser's user agent and those just do whatever they want.