The poors had refrigeration, in the form of ice boxes. Not refridgerators, just basically eskies that the ice man came and shoved a big block of ice into once a week. So basically you could only make ice cream (etc) on the day the ice man came, if you were poor.

...so people just made their ice cream on that day. It required a little planning, is all.