Brexit did not just pop-up into existence fully formed, done in a jiffy, and immediately afterwards British society returning to its otherwise eternal blissful state of everybody being perfectly just and fair.

From my own experience living in Britain as an EU immigrant before Brexit, it is my impression that said kind of people you mention aren't at all a small minority, possibly aren't even a minority.

Further, even most of the "thinking" types have to a lesser or greater extent fallen prey to the soft Jingoism relentlessly pushed by pretty much all of the Press in Britain, were the "Great" in Great Britain isn't part of the name of one of the British Isles but rather it describes the quality of the British Society, British People, British Politics and pretty much everything else British.

I mean, all it takes is to compare the portrayal of international gatherings in, say the BBC, with the portrayal of the same in the news of other countries, and invariably the thing is passed by the BBC as British politicians being given lots of attention and treated as important by the rest, which does not at all match what's reported in the press outside Britain.

And don't get me started on how even supposedly left-of-center newspapers in Britain publish opinion articles about the "scandal" of people sexually behaving in ways that are not the norm, which are incredibly similar to the anti-homosexuality tropes of a few decades ago (only it's not fashionable to be a sexual moralist against homosexuals anymore, so now different victims are chosen).

From my experience, prejudice in British society is totally normalized, at a level well beyond what I've seen in other countries in Europe I lived in - which is why IMHO it's so easy for anti-immigration scare stories to flare up and for things like transphobia to rise to the surface and grow.