I ran my own mail server for 5 years, and it wasn't that hard - but then I didn't do much with it. A system with 3 users and not corresponding with anyone outside a few systems it easy.

Running fastmail's servers is some orders of magnitude harder.