If you buy devices from trustworthy brands and replace them when they stop getting security updates, it should be fine, right? After all, aren't 99% of home networks 'unsafe' according to your definition?
It doesn't follow. There are a lot of homes, so even if 1% of all home networks had "rogue" devices in them they'd dominate DDoS attacks. Besides, it's not HomePods or Withings smart scales or Hue bridges doing that as far as I'm aware, it's mostly cheap, unsupported, noname crap, so you can reduce your risks substantially by not buying questionable products.
How many of those get exploited on firewalled networks before they're remotely patched though?
My whole point above that it does actively hurt, with devices randomly misbehaving at exactly wrong times. It's not enough to set up everything once because devices get updated and change ports, domains, and protocols. It also makes everything more brittle, requiring multiple inter-VLAN proxies to be running at all times for seemingly unrelated devices to work. That SD card in your raspi died? You decided to update Docker on it and run into problems? No Sonos for anyone in the house until it's fixed.
There's a real cost to that paranoia, it's just another case of security/convenience tradeoff.