A family having two kids would typically have one girl and one boy, which means the name ("the family") would carry on only half of the time, which means there would be no growth.

You'd need a higher rate to grow just based on that, plus there are untimely deaths and non-reproducing offspring...