Excuse me? What happened to the mythical cybersecurity abilities of Claude Mythos? Physician, cure thyself!

Anthropic has not informed customers, and the malicious version was silently deleted the backdoored version without telling anybody.

Now we see Anthropic's attitude towards security when the incident does not make for good marketing. Also, considering Binance was just hacked for a massive $3B in stablecoins, and this backdoor allowed remote access to developer laptops, I have a feeling these things are connected. If that is true, then this is by far the worst cybersecurity incident involving AI, worse than even the much ballyhooed PHP2Shell.

Edit: From Molly White's article

Binance later wrote that their "initial investigation suggests that the security incident may involve a compromised DevOps engineer’s laptop", which allowed an attacker to “submit unauthorized withdrawal transactions.”

Okay, now I am really suspicious that these things are connected.

The boosters hope that the AI will replace mathematicians, but they are running into an awfully inconvenient fact: theorems and proofs are useless without people who care enough to understand them and make meaning from them.

Another nitpick: they say no mortal can resist the One Ring for very long, but even someone who just watches the movies knows that immortals can also be corrupted by it (which is why Gandalf and Galadriel were very careful to not accept it). I know there are so many examples of tech bros not understanding media, but Lord of the Rings is by far the worst offender.

No mortal can resist the Ring of Power for long. Anthropic’s co-founders carried it for longer than you could have.

We all know how LotR goes. Boromir got the One Ring, carried it back to Gondor, used its sheer might to defeat his enemies in Mordor, and saved the world. Whatever problems you may have with Boromir, we all know that it would be a lot worse if Sauron got the One Ring. What do you mean, destroy the Ring? Are you a Luddite? This technology is here to stay. It cannot be destroyed by any craft we here possess, unless you're crazy enough to try throwing it into a volcano or something.

If Davidad and Janus are right, and are not merely suffering from a sophisticated form of LLM psychosis

That's all I need to know. The statement is correct since it is vacuously true.

What terrifies me is how we will react. In the aftermath, like after 9/11, there will be an overwhelming pressure to DO SOMETHING NOW. Everyone in Washington and in San Francisco will be panicking. There will be no time to think, no time to talk, only time for DECISIVE ACTION. All our worst instincts will take over. And then…

When this bubble started, there have been so many people urging caution. Many people have urged for time to think and talk, to make sure that we could properly address the many negative aspects of AI. Nope, we went full speed ahead with this nonsense. DECISIVE ACTION. And here we are.

Really feels like this saying is quite applicable these days. "A lie can travel halfway around the world while the truth is still putting on its shoes."

Man, every time I hear about the cost of an actual AI application, it is always something obscene. At least this isn't as bad as Steve "Gas Town" Yegge spending $30k/year for a dozen subscriptions (that cost Anthropic ~$1M/year in token costs) to develop his game with 6 active players.

At this point I have zero patience for my colleagues who think that the AI companies could make the economics all work out. Doesn't help that mathematicians are the least cost-conscious people on the planet, and the AI companies are massively subsidizing their subscriptions in particular.

You know how AI proofs are supposed to be credible because of formalization with Lean? The AI generates a natural language proof and then supposedly generates a Lean program corresponding to the proof to verify that it's true. There is now some troubling news about that: there are frequent examples where the statement given in natural language is different from the meaning of the Lean formalization.

The paper even points out several obvious mismatches in OpenAI's famous Navier-Stokes solution between the English proof and the Lean formalization. There is even a spot where a 4 in the English proof turns into a 5 in the Lean proof (which causes further problems down the line). This is mostly a cosmetic error, and it doesn't mean that the Navier-Stokes solution is wrong, but there could very well be deeper problems, especially with the English proof. At minimum, this solution absolutely requires careful review by experts. OpenAI's behavior towards said experts has been appalling.

My viewpoint has been that the only successes that AI has had are in domains where failure is not costly (at least for someone with deep pockets), and especially when there is a formal system such as Lean that can reliably catch errors. So AI can be endlessly trained to write compiling Lean programs corresponding to formal proofs, but as soon as it runs into tasks that are not 100% bolted down, like writing the proof in English, the hallucination problem rears its ugly head. This reinforces my belief that AI is a chess engine but for Lean.

If I was treating this as a technical problem, I would say, why not just have an AI that generates the Lean proof, and at most have an optional assisting tool that translates it into English without perfect reliability? But this is not a technical problem. By outputting English proofs in the format of a research paper, OpenAI can more easily market that their AI is intended to fully replace mathematicians, and that this is only the herald of their general superintelligence (and not that math is the only thing working out for them).

Terry Tao talks about how he used to try to cooperate with the AI industry to achieve a positive outcome, but now he finally sees their true colors. Link

During this event, OpenAI requested an interview concerning my vision of the future of AI and mathematics. I accepted, and spoke with them for perhaps an hour. I had done similar interviews in various venues, and I assumed that, as with these other cases, they would eventually post the entire interview online, which talked about both the possibilities and risks of AI much as I have done in these other interviews. As it turned out, they only used a few snippets of that interview for that infamous advertisement instead. In retrospect, I should have pushed back harder on their decision; but I decided at the time that even a selective release of my commentary would help raise awareness of the potential for AI, and in particular on the possibility of the “best of both worlds”.

Since then, the situation has deterioriated markedly. Many of the people in the industry that shared my views have left or become sidelined, with most major tech companies now increasingly focused on the race to develop extremely powerful, autonomous AI technologies regardless of their actual value to society. The current drama surrounding the Navier-Stokes global regularity problem is the most dramatic and visible instance of this, but there have been multiple other such examples, and much of my commentary in the last few months has been aimed that the increasingly severe divergence between the current objectives of the AI industry, and of mathematics in general.

Much respect to artists for seeing all this coming from the very beginning, and holding the line.

I decided to take a look at the bitcoin white paper.

Usually, the introduction of a technical paper is fluff and people quickly move on to the technical parts. However, the casual claims made in the first paragraph of this paper have aged extremely poorly, to say the least. In a better world, Bitcoin would have remained as an obscure academic toy, and this introduction would have remained fluff.

While the system works well enough for most transactions, it still suffers from the inherent weaknesses of the trust based model.

What weaknesses are there in the trust based model? Let's find out!

Completely non-reversible transactions are not really possible, since financial institutions cannot avoid mediating disputes. The cost of mediation increases transaction costs, limiting the minimum practical transaction size and cutting off the possibility for small casual transactions, and there is a broader cost in the loss of ability to make non-reversible payments for non-reversible services. With the possibility of reversal, the need for trust spreads. Merchants must be wary of their customers, hassling them for more information than they would otherwise need.

It seems like this guy really loves non-reversible transactions! But as we've seen with the history of crypto, non-reversible transactions sound really good until you fall victim to a crypto scam and there is no way to appeal to the bank to reverse the charges. Reversibility actually increases trust because you no longer need to be absolutely certain that you're dealing with an honest person.

A certain percentage of fraud is accepted as unavoidable.

Almost like that is a problem of human nature. And it's not like cryptocurrency has a spotless record when dealing with fraud! The problem with fraud is not the third party (the bank), but with the second party (the merchant or customer you're dealing with).

The introduction is not long, and most of the paper concerns the technical details of the construction of Bitcoin. By itself, there really is no way to complain about a pile of definitions. But there are still dumb comments that have aged poorly in retrospect.

A block header with no transactions would be about 80 bytes. If we suppose blocks are generated every 10 minutes, 80 bytes * 6 * 24 * 365 = 4.2MB per year. With computer systems typically selling with 2GB of RAM as of 2008, and Moore's Law predicting current growth of 1.2GB per year, storage should not be a problem even if the block headers must be kept in memory.

But why would you want a block header with no transactions? If you wanted to, I don't know, replace the world's financial system, you would need to handle millions of transactions every 10 minutes. How big would the blocks be then? And remember that many copies of the same blockchain would need to be stored (certainly, every miner would need to store a copy). How many thousands or millions of times would that multiply things?

Businesses that receive frequent payments will probably still want to run their own nodes for more independent security and quicker verification.

Turns out it was a bold assumption to think that businesses would just run their own bitcoin miners.

The proof of security (Section 11) is extremely sketchy by modern standards. (They're assuming that all attackers would follow a certain format to attack and not try something different. I get it, proper proofs of security in cryptography are very subtle and difficult.) There is also a page of fluff making random calculations with the Poisson distribution. In any case, the security of Bitcoin requires that the collective computational power of the defenders exceeds the power of any attacker (so the defenders can make new blocks faster).

Bitcoin is very strange as a cryptographic system in that the defender must have more resources than any possible attacker. In most cryptographic systems, the system should be secure even if the attacker has vastly more resources than the defender. Your phone's cryptography should be secure even if some government agency dedicated their supercomputers to try and break it. This means that Bitcoin must waste tons of energy, since that is required to maintain security. Any more energy dumped into it will only increase security and not make the actual transactions faster, which makes Bitcoin horrendously inefficient.

As a purely academic idea in cryptography, it is an interesting curiosity, but the arguments for why it's useful are sketchy. There are other such curiosities that are much more interesting, like homomorphic encryption or secure multiparty computation. It would be a nice line on a CV, but not "incredible".

The true significance of Bitcoin was the terrible libertarian economic argument for it, and the chain of events that would transform it into nothing more than a speculative fashion trend. It has nothing to do with the technical details of Bitcoin. The technical and economic arguments for Bitcoin turned out to be so weak that nowadays, the only real support for Bitcoin is that maybe you can sell it for a higher price to a greater fool.

“California is, I believe, the only state to give health insurance to people who come into the country illegally,” Kauffman said nervously. “I think we probably should not be providing that.”

“So you’d rather everyone just be sick, and get everyone else sick?” another reporter asked.

“That’s not what I’m saying,” said Kauffman.

“Isn’t that effectively what happens?” the reporter countered. “They don’t have access to health care and they just have to get sick, right?”

Kauffman contemplated that one for a moment. “Then they have to just get sick,” he said. “I mean, it’s unfortunate, but I think that it’s sort of impossible to have both liberal immigration laws and generous government benefits.”

Do I need to comment on this one?

It is how professors talk to each other in ... debate halls? What the fuck? Yud really doesn't have any clue how universities work.

I am a PhD student right now so I have a far better idea of how professors talk to each other. The way most professors (in math/CS at least) communicate in a spoken setting is through giving talks at conferences. The cool professors use chalkboards, but most people these days use slides. As it turns out, debates are really fucking stupid for scientific research for so many reasons.

1. Science assumes good faith out of everyone, and debates are needlessly adversarial. This is why everyone just presents and listens to talks.
2. Debates are actually really bad for the kind of deep analysis and thought needed to understand new research. If you want to seriously consider novel ideas, it's not so easy when you're expected to come up with a response in the next few minutes.
3. Debates generally favor people who use good rhetoric and can package their ideas more neatly, not the people who really have more interesting ideas.
4. If you want to justify a scientific claim, you do it with experiments and evidence (or a mathematical proof when applicable). What purpose does a debate serve?

I think Yud's fixation on debates and "winning" reflects what he thinks of intellectualism. For him, it is merely a means to an end. The real goal is to be superior and beat up other people.

Just had a conversation about AI where I sent a link to Eddy Burback's ChatGPT Made Me Delusional video. They clarified that no, it's only smart people who are more productive with AI since they can filter out all the bad outputs, and only dumb people would suffer all the negative effects. I don't know what to fucking say.