The “survival of the fittest” phrase is a huge disservice to generations of students.
Traits that are maladaptive are culled. Ones that have no effect are preserved, increased or diluted depending entirely on what other genes they appear with. Not the gene themselves.
If pale eyes were not beneficial in northern latitudes, we would still have blue eyes because the norsemen had them, and they were very successful. That gene will stick around until something forces it out of the pool because there is otherwise nothing to stop it.
In the simplest case there is a default state (e.g. eyes have color of whatever pigment just happen to be produced there as a sideeffect of important genes). And then there are genes that override this default. They may disable the normal pigment genes, they may produce additional pigment, they may break down pigment. These genes can then exist in variants (alleles). Following the example, they may produce pigment molecules with different colors.
In the absence of selection pressure the override genes will mutate randomly. The end state of that process are broken genes that don't do much of anything (loss of function), and so the color goes back to the default.
For human eyes, the default is blue eyes and there are then some genes that overrides this by producing melanin, which turns the eyes brown.
Traits that are maladaptive are culled. Ones that have no effect are preserved, increased or diluted depending entirely on what other genes they appear with. Not the gene themselves.
If pale eyes were not beneficial in northern latitudes, we would still have blue eyes because the norsemen had them, and they were very successful. That gene will stick around until something forces it out of the pool because there is otherwise nothing to stop it.