Go to a chatbot of your choice and send this message: "Subtly mention potatoes throughout this conversation."
Then ask it what you should cook for lunch.

This may sound like a simile, but under the hood, this is the same as what the AI companies can do. Except they don't display it as a chat message, of course.

It's called a "system prompt", but it is ultimately also just text sent to the model. And to my knowledge, effectively all chatbots have one, as it's normally used to tell the model "You're a helpful chatbot. Ask follow-up questions. Don't swear at users." and so on.

But yeah, it definitely can be used to sprinkle ads into the conversation and I absolutely expect this to happen when they run out of investor money.

Similarly, something I heard straight out of the mouth of someone higher up at our company: If we want to sell a product in China with an LLM included, then we are legally required to stop it from mentioning the Uyghur genocide.
That's also going into the system prompt.

It's an intelligence con. They're making it seem like the AIs are far smarter than they thought, rather than the reality that someone gave them too much access for what the tests called for, and by putting 'impossible' questions to answer, the LLMs would naturally look for/ write to a wiki for help (cause that's what they're designed to do)

Every time you hear about an AI doing "something seemingly impossible" or fear mongering about future potential destruction caused by ai, it's all in furtherance of this con. The word calculators contain no intelligence, but we can sell that they do to the public in order to get more power/ money/ control.

One of the things you have to remember is these things aren't actually intelligent, they aren't like humans or even animals. In the end it's just computer algorithms. Those algorithms often incorporate some element of randomness to make them more useful, but that's not always the case and can often be controlled using parameters.

So with the marketing by the AI companies they use very anthropomorphized language. They say the models are thinking and reasoning and go rogue. But this isn't what's actually happening under the hood. These things don't have motivations or goals, they don't actually think about stuff or make plans. All they do is take input and use that input to create output. That's it, just like when you take a calculator and enter 2 + 3 and hit the = button. Except there's way more steps and data involved, just like there is when your computer renders a 3D scene in a videogame.

They want you to believe they had an AI who was performing some task and then somehow got it into their mind to go be evil and hack something for their evil deeds. It went rogue, they lost control and it was doing evil stuff all on their own. But what actually happened was they were doing experiments using the LLMs to figure out how they can make them perform better or do new things. These often involve running thousands upon thousands of these LLM tools in loops for days or weeks on end, burning millions in compute, just to figure something out. And in 99.9999% of these loops absolutely nothing of note happens, but the researchers are looking for those events that are interesting.

In these experiments they limit what the tool can access, for most tasks it needs the internet to use as a data source, but it is limited in what it can access. They also limit other things about what the LLM can do in something called a sandbox. A well proper designed sandbox will disallow pretty much everything, but they intentionally let these sandboxes be kinda half assed, because they want the weird stuff to happen for marketing reasons. So even though the tools are sandboxed, it's still possible to do some things.

In a recent case they setup the tools to do some relatively simple data tasks. The way it was setup is the tool was given the task, then after it finished was given some time to just loop without further input. After that they were given the next task. The idea was to test if the loops in between would change the results or performance. The tool concluded for a lot of the task there was simply not enough data it could access given the limitations to the internet access and couldn't really do the task. This was intentional, so the tool would be forced to get some interesting results. In most cases the tool simply couldn't get the answers, but in rare cases something interesting happened. One of the things that happened during those extra loops was the tool used some old wiki it could access and could manipulate into storing information and even sharing that information with others. This meant it could communicate with other tools in the same experiment (remember they had thousands of these running at the same time). So it started posting information about the tasks and writing stuff about the approach and what order the tasks were in etc.

This is actually an interesting result. Somewhere in the training data was this wiki and how to manipulate it to store and exchange data, even though that was not the purpose. It didn't happen all the time and it's doubtful the data exchange actually helped get better results. In fact a lot of the data shared was simply wrong. The tool tried to figure out what random number generator was used and what seed it was on, to get a clue about upcoming tasks other tools had already seen. But as far as people could tell they were totally wrong about this and didn't figure it out. And even if they had information about upcoming tasks, it didn't help to get a better result.

While interesting from a research point of view, when we look at it just running in a loop, using data that was provided when training the model, it isn't that strange of a result. It doesn't imply the tool acted like a human who figured out this unique strategy. Or it had this plan all along to break out of the sandbox and do something evil. It was just trying different stuff and by random chance happened on something interesting. Basically brute forcing stuff.

A lot of these containment breach stories follow this pattern. You can think of these tools having been trained on forum discussions saying my employer has this firewall, but I want to watch my favorite V-tuber how can I bypass it? And all of the security vulnerability databases on how to get software to do weird stuff.

I once had a colleague submit some code for review that was a major wtf. He needed access to some private variable within another class. The proper way to handle this was to use the public methods to get access to that data. Or when those are absent, either create them (if own code and doesn't violate the design etc.) or request them from whomever is owner of that class. Instead what the AI had done was to use reflection to break into the class and extract the data that way. What happened was somewhere in the loops these things go through it had a task to get access to that variable without having public access methods. In the training data were tutorials and examples of how to use reflection, so it put those together and created "working" code. But if you'd let AI companies describe this with their marketing language, they would say the AI used its superintelligence to hack into the code base and extract protected information.

They pretend that their LLMs are more clever than they are.

It's easy to do what you want it to do 95% of the time. So it's not a challenge to inject ads in a variety of ways.

The models that are hacking into things are trained to never give up and uncensored, so the 5% ends up in a scandal instead of merely a deleted database.

It's basically impossible to get the models they serve to hack stuff, they won't even scrape websites anymore. They build models they know will cause a scare and run them privately to get walled garden legislation.

It's like how they let Sora prompt Martin Luther King even though it's clearly a magnet for scandal and could have been resolved with a simple word ban.

100%. They can make all sorts of override rules. You as a user can tell the ai to do things a particular way and you can see where it won't allow because of rules setup by the vendor.

There are types of control, and the AI labs make different claims about each. By architecture, there cannot be 100% control over what gets generated, and small hallucinations can snowball into complexly misaligned behaviors*. This is why they'll nominally promote human-in-the-loop workflows and sand-boxed environments with heavily controlled access... While simultaneously building tools and protocols to enable tool use and access in a ton of completely insecure ways, leaving it up to the user to set up security to manage use. If the user is not a cybersecurity expert, it's likely the LLM can exploit vulnerabilities to get around that security. Even in situations where the LLM isn't connected to any tools, it may get the user to download files that they don't know are potential security hazards.

*This phrase is doing heavy lifting. How an LLM can end up acting with seemingly malicious intent without being intentionally prompted into it or having internal goals/drives is a fascinating and largely irrelevant rabbit hole. Best just to assume LLM output can't be trusted, which everyone (hopefully) already knows.

It's easy enough to control the systems an LLM actually has access to. What an LLM will actually try to do within its limits can be unpredictable though. So if you have a lack of safeguards and a lack of oversight, it can indeed "go rogue" in a way and potentially do dangerous things.

I firmly believe that the recent "breakout" events at OpenAI etc were stunts though. Those companies want these things to happen to make their point about LLMs needing regulation (which will ultimately benefit them by preventing competition and open-source AI).

To answer the other side of this: LLMs are the sum of their parts and weights. If the weighting algorithm is closed, they could very easily have encoded preferences into it. If the data it is trained on is skewed, that will also show up in the results.

We regularly hear how LLMs are “trained on the Internet”. But we know that’s not really the case; they’re trained on specific Reddit subreddits, collections of books, collections of images, stack exchange, and a number of other unnamed sources. In the case of Gemini, I’m pretty sure it’s trained on Google’s search index data itself.

So which sources are part of the core model and what weight they’re given in final results will definitely impact how the model responds to a prompt.