And I’m disgusted at colleagues just copy pasting our IP into random cloud-based LLMs, makes me want to bang my head on the wall.

If you're talking about "intellectual property" and not an IP address, then that sounds like a security violation. At the very least, you need some vendor that can sign an NDA as one layer of protection, and a policy that dictates that nobody shares passwords, tokens, PII through these agents. Then firewalls, strict guidelines on where to use these things, make production access forbidden.

The problem is rushing into anything is a bad idea. Slow down, figure out how this shit works, implement policies, use what works, throw away what doesn't, don't radically change everything because of new shiny bullshit.

The people who jumped in headfirst at the first sign of LLMs and fired half of their employees? They all crashed hard and fucked themselves. That's the lesson in what not to do.