There is a substantial difference between public image and actual personality. Society at large was unaware of his proclivities because he had an ace PR team stymie his worst impulses. It's much harder to stay mask off when you're actually interacting and living with someone, and Musk isn't very good at subtley.

Plus like the other user said, they started dating the year he did that weird pedo accusation at that Thai rescue squad. That was definitely the beginning of a LOT of people realizing he was off.