I have a personal server I connect to through Tailscale whenever I'm not home, however I've found that whenever I'm connecting remotely connection speed drops drastically from 100MB/s to <3MB/s.

I expect there to be some speed loss when connecting over the internet compared to locally, but 3MB/s doesn't make any sense especially considering that according to a python script I found that uses speedtest.net to test internet speed through a terminal, it reported 109Mbit/s download and and 76Mbit/s upload (~13MB/s; 9MB/s), which aren't amazing but leagues beyond 2MB/s. Moreover I also did a quick test with a friend of mine briefly using port-forwarding and they reported the same speeds, which tells me it isn't Tailscale slowing me down.

Is this just what happens when you connect over the internet? What trickery is afoot to allow me to download things from the interwebz using that sweet full 109Mbit/s bandwidth?

EDIT: tailscale status says the connection is direct