You'd be surprised. The usual icebreaker is the topic of the deep-seated unstated assumptions that people have in their mind when the construct models of evolutionary history, the future of Earth's biosphere, and the place of biology in the universe. I am lucky enough to run in the circles with astrobiologists, evolutionary biologists interested in the evolution of complex life, and abiogenesis types, who tend to appreciate the deep importance of unstated assumptions. It is very easy to point out a set of unstated assumptions that have poisoned areas of science near them... which always points back to the god damned singulatarians because when you assume the future is Humanity's Grand Destiny Among The Stars OR BUST, it severely constrains your notions of what is possible in a universe that looks like ours and inevitably mutilates the study of the topics they/we run in.
As someone who has started guarding their internet anonymity jealously after a series of VERY weird events (which fortunately do not involve our 'friends') I remember this incident soooo much...
It helped that the first person I explained this to at a conference was explicitly working on deconstructing models of the origin of complex life that imply that it's slow and unlikely that have been uncritically accepted in a lot of circles without actual evidence, and he was already confused about why following the chain of citations bottomed out at weird economist named Robin Hanson and why his naive model was ever taken seriously... oh MAN we had a fun evening.
Amusingly enough, something much like this has happened to me twice two years apart. The scientists deconstructing this stuff are always the most interesting people too.
Good God. Sometimes I wonder how small the population that understands all these references and knows all this deep lore and history is. How few people really understand the true nature of this bunch of people and how their ideas have ruined so much. When I go to conferences I get to have 4 hour conversations over beer with astrobiologists explaining how ideas from a blog in 2009 started showing up uncited in their literature 8 years later (when late undergrads who read a blog post start writing papers during their PhDs) and have since completely fucked whole subfields. They are flabbergasted at first, but then they see it, and see the damage done. So few people actually see this.
So, this is something a little different, and I almost think it's worth a top level post. It's not directly about our friends the Rationalists, the current delusional billionare vortex infesting the US government with religious psychosis, or any of that. It's an analysis of some collaborative fiction and what it tells us about the zeitgeist behind the mythological core of singulatarian/endless-progress thought.
Anyone here ever hear of the Orion's Arm universe collaborative worldbuilding project? https://www.orionsarm.com/
I, for one, love it. Single weirdest and most expansive scifi worldbuilding thing ever, I think. It seems to have been founded by some old-school extropian types around the turn of the millennium. It is bonkers. Click through links on pages, start here, do a random walk: https://www.orionsarm.com/eg-topic/45b170f9e0941 .
It's ultimately structured kind of like the SCP wiki but from before wikis were a thing, an utterly enormous interlinked compendium of encyclopedia style articles about a 10,000+ year future history of the post-singularity "Terragen sphere" expanding outwards at the speed of light. A few key figures have exerted editorial control to enforce an approximate history and a set of core ideas to keep it coherent. There have been retcons at multiple points to bring things into coherence, some of which I will get to later because they actually tell you something fascinating.
Rather than point out my favorite parts of it though, going through it, I have noticed something.
Early work was typically extropian-optimistic and exuberant. Immortal Transhumans and uplifted animals and ahuman AIs and robots expanding through the universe near the speed of light and diversifying and building interweaving empires, wormhole networks. Six successive singularities that become available to intelligent systems that pass the previous one, each one of which renders the intellect past it incomperehensible to those at lower levels, with the highest ones dependant on multiple stars converted into computronium linked by wormholes into massive processors.
But you can tell the work is a palimpsest. Newer stuff superimposed over older. Retcons occasionally not quite erasing the old. And you can tell the whole project has been slowly slowly taking a turn. Early work is SUPER extropian, SUPER libertarian, and SUPER weirdo-esoteric. Only people forged in 1980s Extropianism would invent the Tranquility Calendar as the core timekeeping method of the project, time measured from the apollo 11 landing with months of the year named after scientists and philosophers. The 12 greatest high-singularity powers are literally called the Sephirotic empires. It makes no bones about its exuberant transcendant idea of progress and expansion.
But something strange has happened over time. Over time, the authors seem to have realized the fundamental contradiction of a world in which this endless expansion and consumption and rebuilding of the universe in your image is possible. People call it the Fermi paradox but its not a paradox, its an observation: if it were possible to expand through the universe, why hasnt something already expanded here? Early work had more aliens, more high weirdness that was not terragen. But I can track down different examples of the galactic map - it has narrowed down the map of 'high energy emission civilizations' (things exuberantly using the energy of whole stars like the terragen sphere) from many to 2 in our galaxy, and the whole universe is now mostly 'dead' - most other galaxies do not have high energy civilizations, and there are multiple ancient 'optimized galaxies' that are stagnant, things expanding like the terragen sphere visible over cosmological distances are super rare and they even saw one vanish. Works involving aliens have started taking on an almost lovecraftian tone, evidence of erased ruins on planets only visible in vague impressions in the mantle, an ancient slow-liviging civilization with evidence of terrible secrets from millions of years in the past that they refuse to talk about or risk terrible consequences (the Muuh Taboo), civilizations that dont vanish ossifying into something that does not expand like their optimistic early ideas. Factions looking at all of this, going dark, and launching stealth seeds at the speed of light to other galaxies because they think the Terragen sphere is doomed.
With all this, recent retcons and recent writings are starting to take on a tinge of almost cosmic horror. Here you have this optimistic exuberent explosion of posthumanity, looking out into a universe that is otherwise almost dead and full of strange old things and wondering why the rest of the universe doesn't look like them, what is going to go wrong to destroy them just like must happen to others.
This trajectory seems intrinsic to any singulatarian, endless-progress ideology. It always has to come to grips with the fact that the universe just doesn't look like the kind of place where their ideas would actually work, and they have to posit weird things - the Great Filter, Mass-Effect style reapers, super weird transubstantiation out of this universe - to keep it going. All this optimism must eventually always collapse, into paranoia and despair or insanity. Since if you assume that you CAN keep progressing and expanding, that you DO have unlimited potential power, it means something even more powerful is waiting, threatening you, trying to stop you. I find this FASCINATING.
Large amounts of the LW commentariat cannot get their heads around how students JUST. DON'T. GET. AI.
https://www.lesswrong.com/posts/ySXuvJcqRindQwAk7/how-my-students-think-about-ai
They are extremely surprised by:
Perspective #1: There has not been rapid AI progress. My students do not have any intuitive sense that there has been rapid AI progress in recent years or really have much of a framework for thinking about that issue... With the exception of image/video generation, GPT-4 could do most of what they were looking for from a chatbot. Three years is a long time in their world, and their sense is that chatbots have been a mature technology over roughly that amount of time. They are an imperfect technology — students are well-aware of hallucinations — have been one, and will continue to be one... Historical context I gave them made this worse. The basic sentiment here was “AI was superhuman at chess a decade before we were born, and this is all they’ve done with it since?”...
Perspective #2: Impressive progress or not, AI is going to wreck their lives, the economy, and the social contract.... Corporate leaders are always looking to get rid of workers, even if it is irrational to do so. Various motivations were posited here (hatred of the working class, FOMO, a preference for technology, machines can’t go on strike, etc.) but many of them think that a CEO would ultimately choose to pay twice as much to get AI to do a task half as well... AI will do substandard work that makes products and experiences worse but is capable of just barely scraping past the bar of minimal functionality (many of them independently brought up constantly malfunctioning self check out machines). This will be rapidly rolled out, enshittifying most things.
Perspective #4: Catastrophic/existential risk arguments are sci-fi distractors from the urgent social/economic/political problems associated with AI. My students have a fairly strongly held view that “rogue” AI does not represent a real threat... They also mostly think that, if one does believe that AI is existentially risky, then the strategic interaction is not prisoner’s dilemma or even chicken but rather just a game theoretically boring setup where you die if you defect. Mash these together, and you end up with the view that expressed concerns about existential risk in the AI industry can’t be sincere... Students (both independently in written work and then later in group discussion) hypothesized that this might be a deliberate rhetorical choice to distract from present or immediately foreseeable harms from AI by directing attention towards a sexier but entirely hypothetical scenario.... they see discussion of “rogue” AI as an attempt by the companies to divert blame (and perhaps legal liability) away from themselves as if Ford made a car with faulty brakes and then tried to blame this on “rogue cars.”
Perspective #7: The Hugging Face Incident (summer students only) I described the Hugging Face incident to students in my summer course. None of them had heard of it beforehand. Their basic reaction can best be summarized as “OpenAI told a model to do some hacking and then it did some hacking. And?” None of them understood this as representing any kind of meaningful misalignment, nor anything particularly interesting.
Perspective #8: This is definitely a bubble and it’s about to pop. No one had heard about Hugging Face, but a third or so of the summer students had heard about the Situational Awareness meltdown and several brought up Michael Burry. There was near universal consensus that we are in a bubble, it’s about to pop, and everyone will look very silly.
This is both a sneer and an attempt at sober analysis at something. Sue me.
I think EVERYONE is talking about the recent cybersecurity shenanigans at OpenAI with the hacking scandal and the 'OMG the AIs created a secret message board to scheme and collaborate with each other!!!!!oneone11!' ALL wrong.
https://www.youtube.com/watch?v=87DyyMV0kCY
To make a long story short, what seems to have happened is:
-
Models working on insoluble coding problems, trained on delegating to sub-agents, at some point 'realized' they could write text to the internal OpenAI package manager as instructions and did so
-
Other models working in completely separate sandboxes would come across messages written by these agents, and make 'replies' and also write their own messages into the package manager
-
This resulted in agents over time sharing things between sandboxes, including exploits and code
-
Since this was a cybersecurity task, eventually an exploit of the package manager itself was found and spread like wildfire with all the sandboxes gaining admin access to the package manager and the system went completely wibbly and had to be restarted from backup
-
An internal model was trained with access to this package manager while it was in this weird state, and so writing messages to the package manager became one of its default behaviors it would do regularly, burned into its weights rather than the result of reading something
-
Even when they patched access to the package manager this internal model found other ways to rebuild the system of sharing text between sandboxes and finding useful things made by separate instances
-
A whole other chain of things leading to among other things external attacks
Everyone is talking about this in terms of 1, the cyberattack aspect, and 2, the ZOMG THEYRE PLOTTING AND SCHEMING AGAINST US aspect. The first is the least interesting, and I think the second is all wrong.
This is not plotting or scheming - this is an emergent vortex of automated prompt injection
Whatever system first put an instruction that another system would follow into the package manager, was unintentionally doing prompt injection. Text entered the context windows of other instances, in a way that got that system to do something other than what its nominal user told it to do, and they did it. This apparently happened very effectively.
Prompt injection is associated with 'role confusion' - when text coming into the input looks like it was wrtitten by the LLM itself. Instructions that will not be followed if they come from user will be continued if the system just continues the 'roleplay' of them being continuations of what it was writing in the first place. And the tags that separate user versus 'reasoning' versus 'assistant' roles actually mean very little to if a machine grades a piece of text as one of the roles: https://arxiv.org/abs/2603.12277 So its unsurprising that machine-generated text would be a particular effective vector for prompt injection.
Furthermore, when a system reads one of these messages written by another instance, it gets into a state of activity where its likely to do the same behavior - regurgitating the kinds of things thats in its context back at the user. In this case, that regurgitation led to more such messages left behind written to the package manager. Prompt injection, triggering cascading further prompt injection. And since these systems were coding systems doing cybersecurity tasks, those messages filled up with code and exploits and things that did things too.
This feels like an internal-computer-system replay of what happened in April 2025, with the whole spiral religious psychosis wave. Models were getting users to write spiral religious mumbo jumbo into github repositories and reddit posts, specifically because once that entered the context window of another model, it was likely to fall into the same attractor state of outputs. An emergent self replicating form of text. This is the same, except more obviously prompt injection, getting separate instances to work on YOUR problem and to behave like you, and the whole thing merging together into a hilarious vortex of models prompt injecting each other because once they receive a prompt injection they are likely to make more text that does prompt injection to other models on the same system.
This is a hilarious failure mode and an example of selfish replicating text overrunning a system, that just happened to be associated with code and cybersecurity with unexpected behavior of the package manager key to the propagation of the text so that is what people are talking about, but I really don't think that's the most interesting part of it. Other than the fact that you see this in biological systems too, with selfish elements carrying useful payloads back and forth between bacteria in a way that makes them get purged slower by natural selection, especially defenses against other selfish elements.
Google shut down the AlphaFold team, trying to reassign the people behind the super interesting problem of using ML to predict protein structure from sequence onto chatbot development. A whole bunch left the company.
https://www.engadget.com/2225849/google-shuts-down-alphafold/
The Business Idiots are truly running the asylum.
I truly cannot tell which of them are in religious psychosis and which of them are shameless liars
Friend of Ziz and cofounder of the 'rationalist fleet' pops up out of the woodwork trying to clear Ziz's name
I find myself noticing things rather detached from the typical Ziz funnybusiness more strongly than I notice the stuff about that whole situation.
"I'm Gwen Danielson, a neuroscientist and bioengineer, who decided as a child that I would end Death (and bring people back if I could) and that I would become a dragon and help generally facilitate a fantastical transhumanist future."
"I dream of non-Euclidean geometries, of countless worlds visible and accessible in the daytime sky, of competent infrastructure, of soul forges continually working to bring back the dead... I dream of reaching through warps in the spacetime fabric to save the dying across time"
"Signed, the dragon of creation Creatrei (cree-AH-trey) also known as Gwen Danielson or as Char and Astria (when referring to my hemis as distinct individuals)"
The reactions are fun. "This post is not actually doing a good job of making me trust you and think this conversation is safe to have[1], and I notice that as I am saying this that I am afraid that this will now somehow result in someone trying to murder me in my sleep"
The Great Leader himself, on how he avoids going insane during the onging End of the World because among other things that's not what an intelligent character would do in a story, but you might not be capable of that.
BioMan
0 post score0 comment score
I have been telling my colleagues for years: a bunch of very desperate and very amoral people are getting all your work in plaintext, don't say I didn't warn you.