You don't seem to be asking that question at all. You seem to be including disparate outcome in the set of "unfair" things, by fiat, and from there looking for the logical closure of that concept under machine learning.
That closure is "Harrison Bergeron"... but one man's modus ponens is another's modus tollens.
I think I'm quite clear: we don't have an accepted and useful mathematical definition of what fairness means. This should, at least on some level, line up with our society's legal standards (as the corresponding mathematical definitions of privacy, security, etc. do) while allowing us to prove theorems or at least make well-defined conjectures about the properties of algorithms.
The question is what counts as fair. I'm not saying disparate outcomes is necessary or sufficient, though it receives a lot of attention. Even if we could quantify which disparate outcomes are adverse, it's not obvious how to design algorithms that meet those criteria.
That closure is "Harrison Bergeron"... but one man's modus ponens is another's modus tollens.