What I also don't get is where they will get ever larger data sets that will make a significantly better model. Even if the benefits were linear+, you're running out of new fresh, quality resources to tap.

You add in another 50,000,000 corporate emails you harvested, and stastically, they're a whole lot like the 100,000,000,000 you already saw. Conversely, you destroyed the last copy of some 1929 treatise to scan it in, just to dilute it in a hundred billion other pages.

Moreover, the further into the "long-tail" of data sets you go, the less value they provide for mainstream audiences. If you tap an archive and Claude Code gets excellent at Sinclair ZX80 BASIC programming, that's not really what a bunch of React jockeys need.