One of the things you have to remember is these things aren't actually intelligent, they aren't like humans or even animals. In the end it's just computer algorithms. Those algorithms often incorporate some element of randomness to make them more useful, but that's not always the case and can often be controlled using parameters.

So with the marketing by the AI companies they use very anthropomorphized language. They say the models are thinking and reasoning and go rogue. But this isn't what's actually happening under the hood. These things don't have motivations or goals, they don't actually think about stuff or make plans. All they do is take input and use that input to create output. That's it, just like when you take a calculator and enter 2 + 3 and hit the = button. Except there's way more steps and data involved, just like there is when your computer renders a 3D scene in a videogame.

They want you to believe they had an AI who was performing some task and then somehow got it into their mind to go be evil and hack something for their evil deeds. It went rogue, they lost control and it was doing evil stuff all on their own. But what actually happened was they were doing experiments using the LLMs to figure out how they can make them perform better or do new things. These often involve running thousands upon thousands of these LLM tools in loops for days or weeks on end, burning millions in compute, just to figure something out. And in 99.9999% of these loops absolutely nothing of note happens, but the researchers are looking for those events that are interesting.

In these experiments they limit what the tool can access, for most tasks it needs the internet to use as a data source, but it is limited in what it can access. They also limit other things about what the LLM can do in something called a sandbox. A well proper designed sandbox will disallow pretty much everything, but they intentionally let these sandboxes be kinda half assed, because they want the weird stuff to happen for marketing reasons. So even though the tools are sandboxed, it's still possible to do some things.

In a recent case they setup the tools to do some relatively simple data tasks. The way it was setup is the tool was given the task, then after it finished was given some time to just loop without further input. After that they were given the next task. The idea was to test if the loops in between would change the results or performance. The tool concluded for a lot of the task there was simply not enough data it could access given the limitations to the internet access and couldn't really do the task. This was intentional, so the tool would be forced to get some interesting results. In most cases the tool simply couldn't get the answers, but in rare cases something interesting happened. One of the things that happened during those extra loops was the tool used some old wiki it could access and could manipulate into storing information and even sharing that information with others. This meant it could communicate with other tools in the same experiment (remember they had thousands of these running at the same time). So it started posting information about the tasks and writing stuff about the approach and what order the tasks were in etc.

This is actually an interesting result. Somewhere in the training data was this wiki and how to manipulate it to store and exchange data, even though that was not the purpose. It didn't happen all the time and it's doubtful the data exchange actually helped get better results. In fact a lot of the data shared was simply wrong. The tool tried to figure out what random number generator was used and what seed it was on, to get a clue about upcoming tasks other tools had already seen. But as far as people could tell they were totally wrong about this and didn't figure it out. And even if they had information about upcoming tasks, it didn't help to get a better result.

While interesting from a research point of view, when we look at it just running in a loop, using data that was provided when training the model, it isn't that strange of a result. It doesn't imply the tool acted like a human who figured out this unique strategy. Or it had this plan all along to break out of the sandbox and do something evil. It was just trying different stuff and by random chance happened on something interesting. Basically brute forcing stuff.

A lot of these containment breach stories follow this pattern. You can think of these tools having been trained on forum discussions saying my employer has this firewall, but I want to watch my favorite V-tuber how can I bypass it? And all of the security vulnerability databases on how to get software to do weird stuff.

I once had a colleague submit some code for review that was a major wtf. He needed access to some private variable within another class. The proper way to handle this was to use the public methods to get access to that data. Or when those are absent, either create them (if own code and doesn't violate the design etc.) or request them from whomever is owner of that class. Instead what the AI had done was to use reflection to break into the class and extract the data that way. What happened was somewhere in the loops these things go through it had a task to get access to that variable without having public access methods. In the training data were tutorials and examples of how to use reflection, so it put those together and created "working" code. But if you'd let AI companies describe this with their marketing language, they would say the AI used its superintelligence to hack into the code base and extract protected information.