tbh i have a theory that the ai bubble itself is what's holding us up. the llm angle has been explored deeply and optimised to ridiculous levels. what we got out of it is an incredibly powerful tool, but it's not agi yet. it does have the ability to absorb a lot of skills though, both with deliberate training and with incidental exposure during training, so teaching it new tricks is probably going to be a thing all the way until we reach agi, but imo we're not getting to agi just by building larger llms.

but the problem is, that's literally the only thing that's happening today at any reasonable scale. llms have become the same kind of safe investment slop that reboots and sequels are in media, sure it's guaranteed to be mediocre (in this case, by being expensive to run and having relatively little advantage over existing stuff) but at least there's very little chance they will fail outright. real r&d, on the other hand, is inherently risky, but it's how real progress gets made.

after the crash, hardware will be (relatively) cheap and plentiful, both to own and to rent online, and we'll see investment sour on llms. that will finally open the door for moving past them and finding the next thing. and if agi will ever be made, that's how it will be made, not by training ever larger llms just to distil them back to the same scale as before with minor tweaks to how attention works. which is pretty much what has been happening in the past 2-3 years.