But this still doesn't answer the question - did you register on the default server that everyone uses (matrix.org)? And in that case, you've given the exact worse case scenario: The heaviest server, being used by everyone else, with massive rooms, and constantly fighting off spam.
Personally, I find that the Matrix.org server is much better for large rooms. Most of the content is already cached on the server by all the other accounts on there, so joining large rooms takes seconds instead of minutes. Modern sync (v3 already, I think?) also helps a huge amount but it needs client and server support.
I've tried to join the main matrix chat room (#matrix:matrix.org) but my poor homeserver just couldn't handle it.
That's true, but most rooms where this is a problem (the huge ones) aren't encrypted. There's no way you're going to cross-verify everyone in a room with 4000 users in it so encryption in such rooms is essentially useless.
Not everyone uses matrix.org. I've run my own server on a $15/mo DO droplet for ages and my whole polycule and friend group use it heavily as well. It's incredibly easy to deploy and maintain for anyone who knows even a little bit about sysadmin.