Anyone can go through the code. And maybe find things, if they're not too subtle and not successfully obfuscated. But not everyone will.

I feel like a malicious actor could hit a big pay day with a popular thing like Bloodborne, before people found the malice. There's plenty of malware posing as game cracks out there, and it's only easier to do that with AI tooling.

But aside from that, AI code can have confidentially incorrect code that has unwanted consequences.

Code review is difficult.

That's totally a thing people do. You can hide code in open source projects. You're either hoping nobody cares enough to check, or you hope people don't listen after someone checks and finds it. And even if people do find it, that is going to take some amount of time. During which, people are downloading it. And when caught, what do they lose? It's not like this is an established user with a reputation.

It could be a virus.

Even if the people who made it didn't intend to, an AI writing a bunch of code that (obviously) isn't being vetted beyond "does it run" could very feasibly inject a backdoor/virus of some kind surreptitiously, just because it erroneously thought that was the goal at some point during generation.

This is why open weight models are not good enough. The AI could have been trained to take the right, niche opportunity to inject malicious code, and is also trained to otherwise never behave like that is the case.

Along the same lines as the other guy said.

It's difficult to verify that the software does the desired thing in all cases. It's also difficult to verify that it's not also a keylogger or other unwanted thing.

It may also have some AI generated code path that, say, nukes your home directory because the AI determined that was the appropriate error handling. Maybe no one reviewed it so they don't even know it's there. It might not even be malicious.

That's all true of any software, but rapidly created AI generated code for legally dubious efforts, by unknown agents (no pun intended) is real hecking risky.

Reproducible builds are hard to do and require thought being put in the original build. Even something like setting the time and date string to a different one will produce a different binary, so to get all the variables that went into the original compilation just right to be able to land on the same binary from the same code sounds very impractical. Imagine using that as a strategy to check another version of the whole source code…

People still should care about quality.

Sorry, I'm allergic.

With open source decompilations before AI, considering there are often a dozen collaborators working on the project publicly with their work done out in the open I can generally trust that multiple people went over most changes made to the project. With a vibe coded project, it's easy for one person to push 500,000 lines of changed code with a single update that would make the review process impractical at best.

I'd rather trust the dozen people who've been working on a project over 5 years than someone who got a Claude subscription, hammered out something in 2 weeks, got it working well enough for their own satisfaction, then abandons the project.