Well it isn't. First of all it's the obvious software deficiencies - it'll still boot a heavy OS (boot times can exceed a few minutes from col start), even operations like switching HDMI inputs take significantly longer.
Also several manufacturers (I know of some Samsung versions and Phillips) will position web services in first-level UX navigation making those TVs annoying to use as dumb TVs. There are even some models that will constantly nag you to connect them to the internet!
We managed to get zap time < 300ms on our IPTV platform, but that required A LOT of additional work configuring encoders, networking equipment and messing around with STB firmwares.
There's bunch of places between the video source and you that add their own buffers (not to mention the actual I-frame issues). Usually the most critical are encoders themselves (need to generate closed GOPs and a lot of commercial expensive hardware stuff doesn't give you enough control) and the player software on STBs (those usually have large buffers pre-set).
In the end it's rather large amount of work most IPTV (and digital cable) providers don't care about because... hey... where are you gonna go if you're unhappy with their service? :)
It's probably because of the span between consecutive I-frames (or intra-block sweeps if not using I-frames) in the video stream, but audio should switch "instantaneously" (within 1 frame).
Hmm, if that were the case, wouldn't the time spent waiting be variable? That would mean that you'd sometimes get instant switches, when the I-frame was the next frame you received after the switch.
I have an LG OLED TV. Never connected it to the Internet. Instant-on with the Apple TV, haven't interacted with the Smart TV OS at all and doesn't seem to slow it down.
Also several manufacturers (I know of some Samsung versions and Phillips) will position web services in first-level UX navigation making those TVs annoying to use as dumb TVs. There are even some models that will constantly nag you to connect them to the internet!