San Francisco (United States) (AFP) – An artificial intelligence researcher who left OpenAI to join Anthropic has decided to leave the industry, accusing both US companies of "playing with our lives" in the race to develop AI models capable of self-improvement.
Jacob Coxon, 27, spent the past three years pretraining AI models, first at OpenAI and then, this year, at its rival Anthropic, which he considered more cautious in its approach.
Pretraining is the stage where AI models absorb vast quantities of data.
"Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives," Coxon said on Tuesday.
"The people building AI earnestly believe that it could kill us all by the end of the decade," he said in a post on X.
"This is not a marketing stunt," he added.
Superintelligence is the theoretical point when AI's capabilities exceed human intelligence.
Anthropic safety executive Evan Hubinger backed up Coxon on X.
"We really do earnestly believe AI could kill all humans!" he said, adding that he estimated that risk at more than 10 percent over the next decade.
Hubinger said Anthropic is "trying its best", but does not yet have a plan to ensure an AI system that surpasses human capabilities would obey its creators. He said there was a "low" risk of that happening with current models.
Coxon's resignation comes as Anthropic prepares for its market debut, following a summer marked by unauthorized hacks carried out by AI tools during testing.
The two companies did not immediately respond to AFP requests for comment.
AI leaders say so-called "recursive self-improvement," a stage where AI systems could essentially design and train the next generation of AI with little human involvement, is drawing near.
Coxon considers Anthropic's efforts genuine but said he believes no company can responsibly develop an AI that surpasses humans without government intervention or a coordinated slowdown.
"At Anthropic, the stakes are well-understood, but they are locked in a race to get there first -- they believe no one else will act responsibly, so they must do it themselves, despite the risk," he said.
In February, Anthropic removed a pledge from its safety charter to halt the development of its models if it failed to control their risks.
It argued that if it unilaterally paused its work, its less cautious rivals would dominate the industry, making it less safe overall.
At the end of July, more than 1,000 tech industry employees, including Anthropic's CEO Dario Amodei, called on Washington to support a coordinated slowdown in the development of the most advanced AI systems.
OpenAI halted training of its latest models for two weeks in August before resuming it under tighter controls.
On Sunday, OpenAI's chief scientist, Jakub Pachocki, called for "extreme caution."
"International coordination on future AI development needs to become a top priority for governments around the world," he said in a blog post.
AI models are not regulated by federal law in the United States.
In September, Senator Bernie Sanders and Democratic Representative Greg Casar introduced a bill seeking to suspend AI development until a federal regulator is created.
Okay.
Please explain to me like I’m five: How is a computer program going to kill me IRL?
Is the fear that i.e. AI will get hold of nuclear plants or hospital dialysis machines and kill people that way? (Or something along those lines)
I genuinely don’t know enough about this topic to really form an opinion but my initial reaction is that this is hard to believe.
Perhaps humanity can decline because some tech gets destroyed etc. but people lived 1800s and 4000bc “just fine” - but now AI will kill us ALL?
All I'm hearing is hype. About a year(?) ago, the "we're 6 months out from human level intelligence!" Lies got too stale for investors. So we've escalated to "look at how dangerous our product is!" And finally a shrill, "ai is going to kill us by 2030!!!1!"
"Its advancing too quickly!" As we've run out of data to feed it.
"Its brilliant!" As it is fundamentally incapable of building a mental model of what's even happening.
"It's hacking on its own!" When researchers gave it an impossible problem, no way to stop, and an internet connection to go download other models to learn from.
Its hype. And they've all drank the Kool-aid.
Imagine the supply chain for anything beyond bronze age stopping tomorrow and nobody really knows why because there's oddly no internet or phone networks anymore. The power goes out in most places, and the same with running water after a few days. There's no infrastructure to support people like there was in the 1800s or even 4000BC - those skills, tools and infrastructure are pretty much lost. Nobody even has more than 2-3 days of food.
It would essentially be the end of humanity in its current state. A few pockets of people will remain like cavemen but they don't even know where to find minerals to attempt to create new tools once the ones looted wear out
I figured it just tell people with power that they're awesome and doing a good job and then wait a while