This is it. Evolutionarily, 30-35 is extremely late for procreation in humans. That doesn't give you enough time to raise them before you die at around 45 (around the life expectancy of wild humans).
That's true of most of the civilized period when humans were relatively safe from predators. Most of human history was not like that. As far as I know, hunter-gatherer humans without this form of safety generally didn't live past 40 except in exceptional cases, even if they survived as a child.
So you have very few fertile years left into which to fit multiple healthy children. 2.1 is needed to merely sustain a population (with all the medicine of industrialized society to keep them alive). For "wild" humans, that number would be considerably higher.
