I blame the skip duration. Yt default is 5s, when playing at 2.5x speed (set via an extension) that becomes effectively 2s. It's perfect. If I need more I just press it again.
I have jellyfin set up and (at least on the webbrowser) the default is 30s. Seriously? Why would anyone need 30s skipping? I had the same issue as you before - but only on jellyfin, with YouTube I almost always hit the exact moment I need.