Pretty sure this used to be the fix for me:

https://forums.linuxmint.com/viewtopic.php?t=314918

( scroll down to comment about default.pa and tsched=0 )

But I just checked my default.pa and it is stock values, so I am not sure anymore

When experimenting, setting cpu governor to performance also helped.

You will not believe this. Solution was adding a line in some config file.