A lot of people want local inference, and somehow they assume that the problem is solved now because we have open-source models, we’re going to be able to run them on our computers, and so we’re not going to need all the stuff in the data center.

We don't need them even without local inference. There was already more than enough total global compute for the most valuable use cases of LLMs (research, development, admin). The datacenter build-up is almost entirely fueled by utterly unrealistic expectations of future capabilities. Before and now the case for local inference is a matter of access, not capacity.

But from a global allocation perspective, actually it’s quite terrible to have everybody, like, have a bunch of the world’s limited supply of DRAM sitting in their own computer and being used five percent of the time.

Utilitarianism like this is a disease. By the same logic, it's inefficient that we allow other species of animals to exist and consume usable energy. Hell, why stop at other species? It's inefficient to allow humans to exist - let's kill all humans!

Now, I know that's not what he said, the point was to show that when someone uses utilitarianism as a serious argument, they are no longer worth listening to.

Like you want maximum utilisation in the cloud if you’re trying to deal with this very supply-constrained environment,

The supply is only constrained because you keep hoarding more than you need. Maybe it's time for some communization? So communities where data centers are built get most if not all of the profits from it?