But what would prevent somebody plug in a USB with and just wipe my drive?
If they have enough access to plug in a USB key, they have enough to smash the drive (or the entire machine) with a sledgehammer. Or move it to another machine that they control and wipe and reimage it there.
(Shades of the xkcd with the crypto-nerd and the pipewrench: there's always a non-technical, or less-technical, solution when it comes to security.)