Once you understand that these are chatbots that were designed to complete challenges like this, using tactics like this, you can understand that the chatbots didn't "go rogue." They did what they were designed to do, and because OpenAI ran them with inadequate supervision (without a "human in the loop" that checked each iteration through the Python loop to ensure it hadn't gone off the rails), they trashed a competitor's servers.
Designing autonomous, malicious software is generally considered irresponsible and dangerous. If you showed up at Defcon and gave a talk about how your autonomous malware did something unexpected and damaged someone else's computers, the first question from the audience would be "Why are you so shit at making secure sandboxes?" It wouldn't be "How are you so awesome at making hacking tools?"
The fact that OpenAI is making it much easier for unskilled people to break into and damage servers is indeed very bad news, but it's not new bad news. Irresponsible parties have been doing this for years, most notably the NSA...
...
Riley had a very good way of summarizing this: "LLMs are real, AI is fake." LLMs – chatbots trained on things like CTF logs that can break into servers – are real. They're on a continuum with other hacking tools that have been steadily demonstrating the fragility of the modern digital world, albeit without inspiring anyone in power to do anything about it.
"AI" – chatbots that wake up, "set their own goals," and "spontaneously" start hacking servers – is fake. It doesn't have "a 10% chance of ending the human race." The Hugging Face hack isn't a mysterious, supernatural occurrence. It's a Python loop and a chatbot. The people responsible didn't accidentally create god: they created autonomous malicious software and then failed to closely monitor it, resulting in it doing something both foreseeable and bad.
It's fine to worry about this new suite of tools that give even stupider people the ability to trash even more computers. You should worry about that – and demand better security practices from firms and governments, including a blanket prohibition on NOBUS-style vulnerability hoarding. That's a productive kind of worrying, with a chance of addressing your area of concern. It's infinitely more reasonable than locking yourself in the toilet with a flashlight and saying "Ayyyyy Eyyyyyye" into the mirror until you wet yourself.
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 13 Sep 2026
306 points (93.7% liked)
Technology
88037 readers
3850 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 3 years ago
MODERATORS
Yeah this is the part about all this AGI talk and AI going rogue, it requires a prompt. It responds to an input. These agentic systems have powerful tools that can do other things then spit out text but it’s always responding to a prompt. It may be in a loop responding to its own prompt but it’s still a prompt. And what it spits out is never really anything new. It’s some derivation of what it’s ingested. Actual human intelligence works differently. We do come up with new ideas, new thoughts. So I’ve often thought these LLMs are a dead end to AGI.
"what it spits out is never really anything new"
this is just false
https://www.youtube.com/watch?v=Z7dH_Cxx75g
https://github.com/openai/ten-proofs
unless you would agree that humans never spit out anything new, as well.
This is Lemmy, heavily biased against LLMs, any bullshit flies. This person even gets upvotes despite spewing superficial nonsense. The same way Trump screeching about some bullshit gets cheered on by his "followers". LLMs have real issues we can talk about (tho always spamming them whenever AI is mentioned is also not a good idea). With things like Qwen 3.8 27B we have local, super useful/powerful LLMs running on oldish consumer hardware!
Machines can not and will not create. They are lifeless silicon.
Not saying LLMs are capable of human level intelligence. But what do you think makes human brains special? They're just a confused wet sack of meat filled with neurons. I see no reason why it wouldn't theoretically be possible to construct an artificial brain.
As delivered to public chat interfaces, yes. They all eventually stop and ask "how am I doing?"
Structurally, theoretically, nothing prevents them from running an infinite loop and continuing to act without that checkpoint. They could be given a goal to "optimize" whatever and just act and act and act in what their pattern-matching systems judge to be actions pursuing "optimal" results.
There was an article about "the genie effect" where protagonists are "tricked" by being given something they technically asked for but not what they really wanted. LLMs carry the additional "risk" of not matching input requests to any recognizable pattern, just following their own path which makes no sense to us. Focused on different priorities.
There's no pattern matching though. It takes a input set of tokens and generates a single output token. It adds that output token to the input set and ingests that set to output a single token, which it does the same with until you have your final output. That's what LLMs do. That's all they do
It is pattern-matching. The next token matches the patterns in the training data.
I read your tokens, letter by letter, sometimes chunks at a time, then neurons in my brain fire muscular impulses to output a single letter on a keyboard, then another and another and another until you can read this response. Sometimes my brain will compose the whole sentence before starting to write it, but more often it comes out in chunks - tokens.
That's all my brain is doing right now, other than keeping my organs in homeostasis, background planning regarding envrionmental preferences, my next meal, next bladder and bowel eliminations, etc.
That's all you do, too. It's just a bit harder to tease out the 1s and 0s in your chemical signalling processes.
I think you are unrealistically humanizing this software.
You misunderstood. They output one single token and then re-ingest the entire input + that extra token to generate the next one. For token 3 that's the entire input + token 1 and token 2
If the way you write text is to go and re-read the entire conversation from scratch, just to type another letter you are doing it wrong
Do you even know how you do it? You may think you know, but where's your evidence? Re-reading and re-planning the remainder of the output is just being more careful than most people appear to be when they engage their mouth without consideration for what it is saying.
LLMs are incredibly limited compared with a mamalian brain, the "big frontier" models might be equated to about 6 bumblebees worth of interconnected neurons. They're focused on lexical exchanges, so they do a remarkably passable job considering their limited resources. That they check and recheck and recheck their planned output at each step is not a limitation, it's a process - likely one that compensates for their limited overall resources and reduces their frequency of running too far afield - getting off on tangents.
The process is nowhere near as important as the product. Does use of the tool enable higher quality output in shorter time with less effort? If so, it's a useful tool.
I agree with this statement, and I feel it's pretty obvious that LLM's will never be a useful tool.
I know for a fact that for every single letter I type I don't have to re-read though this entire comment chain. That's as simple as I'm able to make this comparison for you. If you still can't understand I'm sorry for your family
the fact that you think it's that simple is gross and depressing.
the ability to read and comprehend text alone is a lot more complex than your silly allusion. god what a sad fucking take on the entire concept.