i did a test with just spamming date in a terminal and having a high fps video captured from my phone, it was usually under a frame (granted 60 fps so 1/60 sec)
Ah, no, that's not what I mean. It's the input devices. Mainly the mouse pointer.
I now remember there was a way to go around it (a bit cumbersome and ugly) which was to render the mouse pointer only locally. That means no mouse cursor changes for tooltips/resizing/different pointers in games, etc. But at least it gets rid of the lag.
But I also think that for people that didn't try a "snappier" alternative, it was possible not to realize it's there.
Try and make a comparison with Parsec of even the Steam's own streaming. You will notice a big difference if the issue still exists.