Trans people who don't pass will also feel awkward walking in there, and many of them don't. Often times, you'll primarily just see people who pass going into the restroom matching their gender. Just the fact they identify as the gender doesn't often mean even they are comfortable going in there, so it's not like anyone presenting as masculine going into the women's restroom is even a common occurrence in places where trans people are accepted
Anyone who would be willing to visually change themselves to pass as the opposite gender doesn't need "being trans" as a concept to do so, and could be just as willing to engage in that regardless.
Gay people exist and they don't feel awkward walking in to those spaces, yet could just as well engage in sexual assault of people matching their gender without anybody questioning their right to be there.