In the United States, "yellow man" is considered pejorative, "white man" is not. In isolation this seems illogical as they are structurally analogous, but words do not have meaning in isolation. The meaning of a word entails its historical use by a society.
I dunno. I almost exclusively hear "white man" used as an insult. Not long-term historically certainly, but certainly now - it's almost always accompanied by the implication that white men are racist (and/or sexist).
