Problem is passkey generation, expiration logic, etc., is often not well protected, so if a hacker is able to get it, they may not be able to target a specific account, but they can target random accounts with semi-bruteforce since there's no username to limit tries against or lock the account if there's too many tries. Just use a botnet and start throwing probable keys at it and you're likely to get a few.

Also, with a password repository, I protect that as exceptionally as I can, but I can always change my passwords and would do so of that was compromised. But I'm not talking about the passkeys of an individual being stolen, I'm talking about a compromised site having very little way of determining of a hacker has gotten enough information to narrow brute forcing down to a relatively manageable level with a botnet and using that to access random accounts for personal information scraping. And the user may have no idea it's happened to them since the bad tries never were associated with any login.