I am not ignoring anything, I'm looking at the broader picture, which includes non-biological evolution. Simple rebuttal to your specific point: The population of self-improving AIs will also go from 0 to many more.

In a broader sense evolution moved from very static simple domains to dynamic malleable complex domains. Biological evolution speed is glacial compared to cultural evolution speed. Even then, cultural evolution is fairly slow compared to technological evolution.