I don't know about the UK, but I was taught that US slave owners would do collective punishments if any slave kids were caught swimming.

Swimming is a super power if you're running from dogs, because dogs lose he scent when you cross a river.

Also, a lot of US public swimming pools were destroyed by the local governments when racial segregation was banned.

A 'fun' anecdote from Wikipedia about British attitudes towards non-white people swimming:

In 1844 a swimming competition was held in London with the participation of two Native Americans. The British competitor used the traditional breaststroke, while the Native Americans swam a variant of the front crawl, which had been used by people in the Americas for generations, but was not known to the British. The winning medal went to 'Flying Gull' who swam the 130-foot length in 30 seconds – the Native American swimming method proved to be a much faster style than the British breaststroke. The Times of London reported disapprovingly that the Native American stroke was an unrefined motion with the arms "like a windmill" and the chaotic and unregulated kicking of the legs. The considerable splashing that the stroke caused was deemed to be barbaric and "un-European" to the British gentlemen, who preferred to keep their heads over the water. Subsequently, the British continued to swim only breaststroke until 1873.