I did some testing, and it turns out I did not have Direct ~~Stream~~ Play forced. I think I enabled it at one point, but I didn't see much difference on the "worst case scenario" video I was testing with, so I turned it back off. Looking at the Dashboard page, I can see that, even with the setting forced, that particular video is still transcoding video from MPEG2VIDEO to H264, and audio from AC3 5.1 to AAC. I may have to look into re-encoding that particular series, though the files for that show should have followed the same process as most of my other files, so I'm not sure why they're different.
Some other videos that I tested did show improvements when forcing Direct Play, but the Dashboard still shows transcoding (hw) video from H.264 to H264 (note the missing period) and transcoding audio from DTS-HD MA 5.1 to AC3.
As far as my TV setup, I don't have external speakers. I'm essentially feral, and I don't feel like they're worth it in most situations. The built-in TV speakers get loud enough, and sound fine to me. I'd rather not have to defend that setup, so hopefully no one comes at me for that unless it's actually responsible for the audio delay.
The transition from analog to digital really hurt my desire to watch OTA TV (you caught me! I'm not under 25).
With analog broadcast, any weak signal or interference produced a little bit of static, but you could still see and hear what was being said. With digital, any weak signal means dropped frames and silence or weird glitches. You completely lose what's happening. Even with a powered antenna, I have frequent issues with weak signal. I could probably try to get a rooftop antenna installed, but there's no guarantee it would be any better. It's just easier to find other entertainment at this point.