I don't think that's exclusive to white men at all. We have seen a number of concerning anti-Semitic statements from Black NBA players and one particular Arab podcaster. The general rule seems to be something like "Rich / famous people are allowed to only mildly reject -isms that are common in the community in which they grew up."