We defeated the Nazis, but not their ideas. Fascism is a collection of ideas, so it's an ideology and a political one at that. People had to invent these ideas. They were not an inherent part of human psychology. Fascism is a collective puzzle that we all have to solve together.
Violence in self-defense is necessary to stall for time. However, no matter how many fascists die, if fascist ideas are not defeated then there will always be more fascists. There is no benefit in breaking the social contract of tolerance first. We are in an information race, so the spreading of true information is always more useful than violence.
People should defended themselves regardless of the political ideology of their attackers. Once that's done for the day though it's back to spreading socialism. Fascism is growing because neo-liberalism denies people the ability to solve their economic problems. Which in our case are caused by late-stage capitalism. edit: typo
I know the right-wing infosphere has brainwashed multiple members of my family. I don't have a way to check the percentage of people I've known in my life that are now brainwashed. I know that my life would have been lesser had I not met every single one of them. I don't see the people in my life as a purity test, they're still the same people. What happened to them is a reminder that we must first and foremost defeat fascism, the political ideology.