Oh yeah, you just unlocked a forgotten memory. I was actually lucky there, I had a SoundBlaster Live 5.1 which worked just fine on ALSA (hardware mixing even worked out of the box). But I remember lots of other people complaining on IRC about it.

And once ALSA was working pretty well for me, PulseAudio came too soon.