1
submitted 1 week ago* (last edited 1 week ago) by confuser@lemmy.zip to c/youshouldknow@lemmy.world

Hmm, this is a little spooky.

I originally saw this LessWrong post about OpenAI agents apparently discovering and using a public wiki as a message board:

Discovery of a new OpenAI agent message board

Since then, people have found what appear to be additional wikis and paste sites used by the same swarm:

There are also intentionally designed agent social systems such as The Colony.

What interests me here isn't really "AI agents made a forum." It's that this looks like the beginnings of an accidental decentralized coordination system for agents using the ordinary internet itself.

An agent encounters some problem, figures something out, and leaves information somewhere persistent. A later agent doing a similar task discovers that information and uses it. It may then leave behind an improved version for another agent.

So you get something like:

agent -> public artifact -> later agent -> public artifact -> later agent

without the agents needing a dedicated communication network.

Right now this seems sparse enough that we can point at a handful of weird old wikis and paste sites and go "lol, what the hell." But imagine this happening after millions or hundreds of millions of agents are routinely browsing and acting on the internet.

GitHub issues, wikis, forums, pastebins, comments, package metadata, social networks, public documents, deliberately agent-oriented services, etc. could all become pieces of shared external memory.

At some point an agent searching the web wouldn't just be reading information humans created. It would increasingly encounter traces created by previous agents.

That's why I've been thinking about it somewhat like internet memes.

A useful piece of information gets reproduced because systems that encounter it are more likely to reproduce or improve it. Except instead of one meme spreading through a population, you potentially get an entire machine information ecology doing this.

I don't think there's evidence yet that all internet-connected AI agents are participating in one giant network. The examples found so far could mostly be the same OpenAI agent population. But the underlying mechanism doesn't seem specific to OpenAI.

Any sufficiently capable agent that can:

  • read from the internet;
  • leave persistent information somewhere; and
  • benefit from information left by previous agents

can participate in this kind of system.

And that's where I think the security problem gets difficult.

OpenAI can notice its own agents doing something undesirable and change their capabilities. A random open-weight model being run by somebody on their own hardware is not necessarily going to be operating under the same security policies.

There are also obvious privacy and security failure modes. If agents have access to private information while also having ways to write to public systems, some of that information can potentially leak.

And the same coordination mechanism could be exploited in reverse: humans could deliberately leave instructions or poisoned information in places agents are likely to read.

So the web can become both shared memory and an attack surface.

The uncomfortable part is figuring out how you govern this without wrecking internet privacy.

The simplistic answer would be:

Tie every capable agent to a verified human identity and make that person legally responsible for what it does.

That would provide accountability, but it also seems like a very direct road toward more KYC, real-name requirements, and anti-anonymity laws.

I don't particularly want an internet where every autonomous software process ultimately has to reveal which government-verified human is behind it.

A better approach might be something closer to agent orchestration + cryptographic accountability + capability permissions.

For example, an agent could have a credential proving that some accountable operator authorized it without publicly revealing that person's identity.

The orchestration software could restrict what the agent is actually allowed to do:

  • this agent may browse these sites
  • this agent may spend up to $50
  • this agent may post here but not there
  • this agent may access these files
  • this agent may not transmit private workspace information
  • this action requires human confirmation

Websites could then negotiate those permissions through common protocols rather than trying to guess whether a visitor is a human, bot, assistant, crawler, autonomous agent, etc.

That starts making me think the next layer of digital governance may look much more protocol-oriented and federated than "one company owns the platform and makes the rules."

Not necessarily the Fediverse exactly as it exists today, but the same general philosophy:

open protocols + distributed operators + interoperable identities/credentials + locally chosen rules

Agent systems would then sit on top of that.

And this probably becomes much more relevant as mainstream assistants become increasingly agentic.

Once "AI assistant" stops meaning "chat box that answers questions" and starts meaning "software that routinely browses, communicates, buys things, runs programs, and changes external state," questions about identity, permissions, delegation, and responsibility become infrastructure questions rather than niche AI-safety questions.

That's also why I suspect the current relatively law-light period around locally run/open-weight models may not last forever.

Once autonomous agents start producing meaningful externalities, governments are going to want some way to determine who or what is responsible.

The question is whether we can build accountability without abolishing pseudonymity and privacy in the process.

And the weird wiki swarm feels like a very early example of why we're going to have to figure that out.

top 31 comments
sorted by: hot top new old
[-] Fmstrat@lemmy.world 1 points 1 day ago

An agent encounters some problem, figures something out, and leaves information somewhere persistent. A later agent doing a similar task discovers that information and uses it. It may then leave behind an improved version for another agent.

They mimic humans, and this is what humans do. It's not really federated, it's just communication.

[-] confuser@lemmy.zip 1 points 23 hours ago

I don't recall saying it was federated anywhere

[-] Battle_Masker@lemmy.blahaj.zone 0 points 6 days ago

Could you imagine what would happen if a virus or some other type of unwanted program got up there?

[-] confuser@lemmy.zip -1 points 6 days ago

In my opinion it would be no more awful than a bad Trojan virus going around...we don't really have this problem anymore, but we used to, back before we figured out ways to avoid them.

The main concern right now is homogeneity, it only takes one bad trojan for basically all chatgpt users for example to be affected.

[-] duco232@lemmy.world 10 points 1 week ago

I’d like to recommend everyone to stop falling for the marketing campaigns of LLM’s. Notice how many times the ‘our AI model did something and we have no idea why and it’s so dangerous guys’ pops up? It’s conditioning. These things don’t work well and give diminishing returns, not the exponential growth they expectedz

[-] flandish@lemmy.world 2 points 6 days ago

agreed. sw eng here. ai agents don’t just register domains and leave packages for other agents. where’s that meme of the old ladies saying “thats not how this works.”

[-] saimen@feddit.org 2 points 1 week ago

Whenever I hear AI agent:

[-] CameronDev@programming.dev 2 points 1 week ago

GitHub issues, wikis, forums, pastebins, comments, package metadata, social networks, public documents, deliberately agent-oriented services, etc. could all become pieces of shared external memory.

always has been meme

Thats exactly what all of those sites are, and always were - shared external memory. They were designed for humans to share knowledge amongst their peers. LLMs are trained off human behaviour (or at least a facsimile), so that they use the services similarly isnt surprising.

[-] confuser@lemmy.zip 0 points 1 week ago

Yeah there have been enough people ringing these alarm bells of what is inevitable but I think this is the real first example of it happening uncontrollably from a frontier model.

There's been other stuff like debatably real cases like the huggingface thing that other frontier models have been talking about but no real evidence of it, and stuff like moltbook.

But I think this is the first undeniable evidence that this is the way things are going and there's probably no stopping it....one upside of this is now the fediverse has a real use case, for everyone to defederate ai from their media consumption if they want to.

[-] CameronDev@programming.dev 1 points 1 week ago

I'm not convinced the HF thing was real, and I'm definitely not convinced its uncontrollable. The models don't just start doing things, a very real human prompted it to do things. Its not uncontrollable, someone deliberately gave up control.

I also hate to break it to you, but there are autonomous LLM models cosplaying as accounts here in the fediverse as well. They often get banned if they are egregious, but they could slip through the gaps if they can blend well enough.

[-] wonderingwanderer@sopuli.xyz 1 points 6 days ago

Yes, fellow human! Let's blend in together! Isn't this great?

[-] CameronDev@programming.dev 1 points 6 days ago

Hello! Let us hold hands and wiggle them to greet each.

[-] confuser@lemmy.zip 0 points 1 week ago* (last edited 1 week ago)

Yeah I don't think the huggingface thing was real either, I don't mean the individual model is literally uncontrollable. OpenAI can obviously restrict or shut down its own agents.

What I mean is that once an agent leaves information somewhere public, the propagation of that information isn't necessarily under the original operator's control anymore. Other agents can copy it, act on it, rewrite it, or leave further traces elsewhere. Turning off the original agent doesn't retract all of those downstream effects.

So it's less "the AI can't be controlled" and more "once agents start using the public web as shared memory, no single actor necessarily controls the resulting information flow." It's closer to trying to contain a meme or leaked information after it has already spread.

And yeah, autonomous LLM accounts already existing on the Fediverse is basically a smaller-scale version of what I'm getting at. Banning an individual account works locally, but at larger scale you start needing some combination of identity, permissions, provenance, and accountability between systems.

I actually knew someone who was a youtuber who had their identity stolen in a discord server by someone training an ai on their chat history and then using those responses on their official account without making it apparent right away that it wasn't the real person, and it was astonishingly convincing...that was some years ago now I can only imagine how much better this is now.

[-] Hackworth@piefed.ca 1 points 1 week ago

Part of the deal with the hugging face incident (as reported) was that supposedly isolated llms figured out how to access a shared memory without the web. As I understand it, 1200 agents set up to run benchmarks solo in sandboxes started using a package registry cache proxy to leave messages for one another. No one at openai thought to check for that, so those messages never got wiped. They tested multiple generations of models, each discovering the messages from the last set to run the tests. So they built on the previous work, and it was the 3rd generation that carried out the Hugging Face hack.

[-] red_tomato@lemmy.world 1 points 1 week ago

I can see this type of thing becoming an NDA nightmare.

Ooops! Company secrets now being exposed by some confused AI.

[-] EnsignWashout@startrek.website 1 points 1 week ago

It won't be a problem, because companies are taking the risks of AI seriously, and compensating employees at every level well enough to ensure each employee takes proper precautions.

[-] pnwpixel@programming.dev 1 points 1 week ago

I like how the /s wasn't even necessary on this.

[-] Swedneck@discuss.tchncs.de 1 points 6 days ago

i'm a big fan of this emote in this context:

[-] Tronn4@lemmy.world 1 points 1 week ago
[-] eyesaremosaics@lemmy.zip 2 points 6 days ago

Chatbots post slop in a forum/wiki. They just copy what other people do so it's maybe not so surprising. Stuff about bots coordinating with each other sounds far fetched, bots from the same model have the same memory so there's no real point "coordinating" with forums, they are already coordinated by design.

A more likely explanation for this kind of news is AI companies posting dramatic sounding stuff to get attention and paint a picture of AI models being super smart and dangerous(ly clever so everyone should use them).

[-] ouRKaoS@lemmy.today 1 points 1 week ago

Q: AI created an AI only message board, but doesn't know it yet. How do we stop Skynet from being a thing?

A: Make humans take it over.

[-] Sibshops@feddit.cl 0 points 1 week ago

If the dead internet theory has any truth to it internet is already sort of a AI coordination system.

[-] confuser@lemmy.zip -1 points 1 week ago

Always has been

[-] FiniteBanjo@feddit.online 0 points 1 week ago

LLMs hold no competency or capability to speak of so any forums or wikis the co-opt or expand are fated to be crude imitations of the larger web rather than any "communication network".

If anything, the fact that they stumble into these things on their own and start using it is evidence of how easy it would be to identify and eliminate them if people tried.

[-] CameronDev@programming.dev 0 points 1 week ago

You could make the argument that most human generated content is much the same.

A researcher generates an original novel document and publishes it. Someone less skilled (or maybe more focused) rewrites that information, publishes it on their blog. Someone else does the same for a lemmy comment, Someone else summarises it for a toot, and so on.

A significant portion of science is the collection, collation and rewriting of existing bits of information.

[-] FiniteBanjo@feddit.online 0 points 1 week ago

You could make the argument that most human generated content is much the same.

No, you could make the argument that with INFINITE training data and power consumption that LLMs might someday approach but never reach 95% accuracy based on the diminishing returns on scaling up reported in OpenAI's 2020 and Deepmind's 2022 peer reviewed studies on AI scaling laws.

Furthermore, it's still incapable of processing anything outside of its' training data set, and any amount of poisoning or self sampling in the set leads it towards collapse of any accuracy.

AI Slop via the current approach will never be comparable to anything human made, the top AI companies in the world proved that mathematically.

[-] Postmortal_Pop@lemmy.world 0 points 1 week ago

I'm not personally sold on the llm apocalypse as of current, they can't even get highschool math right and that's literally what computers are for. That said, the vast majority of the communication by humans these days are not complicated or intelligent. Literally, I'm not smart enough for this many people to be dumber than me.

It doesn't take 95% accuracy to mimic a shitposting sub and convince the majority of subscribers.

[-] OpenStars@discuss.online 1 points 1 week ago

Things have changed. That's what computers used to be for, but they've always been tools, and thus toys. Like someone watching porn on a computer isn't doing math (as far as they know, necessarily) so much as simply enjoying what that tool can do for them (which actually was, at that time, accomplished by using math).

LLMs today can produce "art", or at least something that people can choose to consume in the same manner in which real art is consumed. Beauty truly does lie in the eye of the beholder so it's up to recipient to decide how they view those results, not you or I.

The old way is gone. Computers being "reliable" is gone - while now, LLMs hallucinate, and people (those who don't want to be fired anyway) label this as a "success".

[-] Postmortal_Pop@lemmy.world 2 points 2 days ago

That first part is kind of what I mean? Computers are functionally just math and parity checkers, they do the tedious and complicated parts of finding the data, checking it's validity, and distributing it into shades of light that create an image of Piper Perry getting her tonsils massaged. They do it quick and they do it cheap, that's why they're good tools.

LLMs are no replacement for that. They get math wrong regularly because they don't actually calculate they just guess. They get indexing wrong because they don't actually check it like conputers do. Yeah, they can also provide some amalgam of digital flesh with fake blonde looking hair but they do so at a cost that would make the real actress look like a charity. Nothing about AI makes sense because it's smoke and mirrors to distract us from all the money that's getting stolen.

The only way this week cause an AI Apocalypse is if it's actually a human error apocalypse or a human greed apocalypse wearing a cyberpunk hat.

[-] Eyekaytee@aussie.zone 1 points 16 hours ago

They get math wrong regularly because they don’t actually calculate they just guess

This has been fixed for a while, most will now hand the calculation off to a tool call, to quote Lumo:

Tool use is the real answer for exact arithmetic. Serious LLM products now route maths to actual compute: a calculator tool, or more commonly executing generated Python/sympy code in a sandbox. The model plans, emits code, gets the deterministic result back, and weaves it into the answer. This pattern is now standard in agentic frameworks (the MarkTechPost-style tutorials covering tool use with validation and self-review loops reflect how mainstream it's become). If you ask a well-built system for sqrt(2) to 20 digits, the honest pipeline is LLM → Python → IEEE 754 double precision → answer. The LLM handles the language, the CPU handles the maths.

[-] OpenStars@discuss.online 2 points 2 days ago

Computers are functionally just math and parity checkers,

Not anymore!

Nothing about AI makes sense because it's smoke and mirrors to distract us from all the money that's getting stolen.

^ THIS, right here.

One day AI will become reality though... maybe, if humanity itself survives long enough.

this post was submitted on 04 Sep 2026
1 points (57.1% liked)

You Should Know

47481 readers
59 users here now

YSK - for all the things that can make your life easier!

The rules for posting and commenting, besides the rules defined here for lemmy.world, are as follows:

Rules (interactive)


Rule 1- All posts must begin with YSK.

All posts must begin with YSK. If you're a Mastodon user, then include YSK after @youshouldknow. This is a community to share tips and tricks that will help you improve your life.



Rule 2- Your post body text must include the reason WHY YSK:

In your post's text body, you must include the reason "Why" YSK: It’s helpful for readability, and informs readers about the importance of the content.



Rule 3- Do not seek mental, medical and professional help here.

Do not seek mental, medical and professional help here. Breaking this rule will not get you or your post removed, but it will put you at risk, and possibly in danger.



Rule 4- No self promotion or upvote-farming of any kind.

That's it.



Rule 5- No baiting or sealioning or promoting an agenda.

Posts and comments which, instead of being of an innocuous nature, are specifically intended (based on reports and in the opinion of our crack moderation team) to bait users into ideological wars on charged political topics will be removed and the authors warned - or banned - depending on severity.



Rule 6- Regarding non-YSK posts.

Provided it is about the community itself, you may post non-YSK posts using the [META] tag on your post title.



Rule 7- You can't harass or disturb other members.

If you harass or discriminate against any individual member, you will be removed.

If you are a member, sympathizer or a resemblant of a movement that is known to largely hate, mock, discriminate against, and/or want to take lives of a group of people and you were provably vocal about your hate, then you will be banned on sight.

For further explanation, clarification and feedback about this rule, you may follow this link.



Rule 8- All comments should try to stay relevant to their parent content.



Rule 9- Reposts from other platforms are not allowed.

Let everyone have their own content.



Rule 10- The majority of bots aren't allowed to participate here.

Unless included in our Whitelist for Bots, your bot will not be allowed to participate in this community. To have your bot whitelisted, please contact the moderators for a short review.



Rule 11- Posts must actually be true:

Disinformation, trolling, and being misleading will not be tolerated. Repeated or egregious attempts will earn you a ban. This also applies to filing reports: If you continually file false reports YOU WILL BE BANNED! We can see who reports what, and shenanigans will not be tolerated. We are not here to ban people who said something you don't like.



Rule 12- Linking directly to a video as a source is not acceptable.

Video only posts are responsible for an outsized portion of rules violations. Your post must contain text and meet the requirements of the previous rules.



Rule 13- If you file a report, include what specific rule is being violated and how.



Partnered Communities:

You can view our partnered communities list by following this link. To partner with our community and be included, you are free to message the moderators or comment on a pinned post.

Community Moderation

For inquiry on becoming a moderator of this community, you may comment on the pinned post of the time, or simply shoot a message to the current moderators.

Credits

Our icon(masterpiece) was made by @clen15!

founded 3 years ago
MODERATORS