Without E2EE, the server can directly access and manage how the video and audio streams are sent to each client. It can downgrade video for participants who are struggling, prioritize throughput of audio data for certain clients, prioritize higher res video only for the user who's currently speaking, drop individual frames for clients that are getting behind, etc etc.
With E2EE, by definition, the server becomes a dumb relay, and all control moves into the clients. Depending on how E2EE is implemented, you then lose the ability to do some or all of these optimizations.
There are potential workarounds, but each comes with further tradeoffs in turn, in performance, privacy & complexity. That's not to say that it's completely impossible to optimize given E2EE, as you can move many optimizations to each client, or try to have clients expose just enough data for servers to optimize traffic without being able to read the contents (e.g. by splitting the audio & video streams so the server can manage them independently, or having each client upload separate high & low res video of their own video stream). It definitely makes many optimizations dramatically harder though, creates some serious engineering problems, and in practice rules out some optimizations completely.
With E2EE, by definition, the server becomes a dumb relay, and all control moves into the clients. Depending on how E2EE is implemented, you then lose the ability to do some or all of these optimizations.
There are potential workarounds, but each comes with further tradeoffs in turn, in performance, privacy & complexity. That's not to say that it's completely impossible to optimize given E2EE, as you can move many optimizations to each client, or try to have clients expose just enough data for servers to optimize traffic without being able to read the contents (e.g. by splitting the audio & video streams so the server can manage them independently, or having each client upload separate high & low res video of their own video stream). It definitely makes many optimizations dramatically harder though, creates some serious engineering problems, and in practice rules out some optimizations completely.