If you pool the various genetic causes of intersex conditions (neither unambiguously male nor unambiguously female) it's about the same percent of the population as natural red hair. Should we ignore people with red hair?
I'm not saying we should ignore them. Just that for most people the two concepts "sex" and "gender" aren't separate, and trans people are simply regarded as quirky. For example there are people born with 6 functional fingers per hand but we still say that humans have 5 fingers per hand and don't say that humans have anywhere between 0 and 8 fingers per hand.
Treating sex and gender separately makes it seem like they just accidentally coincide in case of 99.x% of the people.
