They're pushing the redesign because it works. I'd guess a good portion of their userbase is actually relatively new or young users, who prefer to scroll through endless wall of pictures/videos like they do on instagram. I'd never use Reddit like that, but seeing the discussions over the past few years it's obvious most (vocal) users are on mobile, and they use the instagram look.
The redesign has one major advantage from their point of view (which I suspect was the reason behind it in the first place) - the individual posts on main screen now take up much more screen space. This allows to seamlessly insert fairly large ads masquaraded as posts, which wasn't possible with the old design.
> the individual posts on main screen now take up much more screen space
It's insulting how deliberate this is. I just tried using the redesign for ~5 minutes to see if it was as bad as I remembered, and the amount of dead space on my screen is striking. I have a normal 16:9 monitor and roughly 2/3rds of the front page is just totally blank, meanwhile you have post titles taking up 3-4 lines of the tiny column they're forced into.
If it wasn't for the Apollo reddit client I would have stopped using reddit a long time ago. I used to use it mostly in the browser on my laptop/desktop, but it has become unusable even when using old.reddit.com (and plugins to force old.reddit.com).
I now only use it as an app on my phone which has ruined a lot of the conversation because it's far more cumbersome to type on mobile, as well as stuff like switching between browser/reddit to copy a link or quote something on the web is a pain on mobile.
I'm surprised there isn't a growing competitor to reddit at this point (that I know of)
I find the Chat feature to be an awful addition as well. Sometimes when I'm trying to sell things, people will send a chat instead of an Inbox message and I end up not seeing it for days because there's not much of a notification and Apollo for iOS doesn't support chat (maybe no unofficial Reddit client does?)