Technology

There are a couple of reasons that might not work:

• Maybe we'll asymptotically approach a point that is lower than human-level cognitive capabilities
• Gradual improvements are susceptible to getting stuck in a local maxima. This is a problem in evolution as well. A lot of animals could in theory evolve, say, human level intelligence in principle, but to reach that point they'd have to go through a bunch of intermediate steps that lead to worse fitness. Gradual scientific improvements are a bit like evolution in this way.
• We also lose knowledge over time. Something as dramatic as a nuclear war would significantly set back the progress in developing AGI, but something less dramatic might also lead to us forgetting things that we've already learned.

To be clear, most of the arguments I'm making aren't really about AGI specifically but about humanities capability to develop arbitrary in principle feasible technologies in general.

I can't think of a scenario where we've improved something so much that there's just absolutely nothing we could improve on further.

Progress itself isn't inevitable. Just because it's possible doesn't mean that we'll get there, because the history of human development shows that societies can and do stall, reverse, etc.

And even if all human societies tends towards progress, it could still hit dead ends and stop there. Conceptually, it's like climbing a mountain through the algorithm of "if there is a higher elevation near you, go towards that, and avoid stepping downward in elevation." Eventually that algorithm brings you to a local peak. But the local peak might not be the highest point on the mountain, and while it is theoretically possible to have gotten to the other true peak from the beginning, the person who is insistent on never stepping downward is now stuck. Or, it's possible to get to the true peak but it requires climbing downward for a time and climbing up past elevations we've already been to, on paths we hadn't been on. One can imagine a society that refuses to step downward, breaking the inevitability of progress.

This paper identifies a specific dead end and advocates against hoping for general AI through computational training. It is, in effect, arguing that even though we can still see plenty of places that are higher elevation than where we are standing, we're headed towards a dead end, and should climb back down. I suspect that not a lot of the actual climbers will heed that advice.