I tend to agree, I also don't think its a useful term when thrown against many people because they might hold this kind of specific view but be progressive in many other ways, it means they can dismiss the term because they understand themselves to not be it, like when the right wing try to label leftists and progressives as "fascist", its just not accurate.

So what I prefer is to point out something that is harder to deny, I like to point out how happy they are to hold a view or position that is held by or promoted by fascists/nazis etc. Saying to somebody that they "side/agree with the fascists" on a view or policy seems to elicit a much more emotional reaction as its not something they can easily deny and they have to confront.