Apps don't normally have to care that deeply about UI latency, high cpu usage, various quirks of performance hits across platforms, VRAM, controller variations, achieving a solid 30-60 FPS minimum, battery life impact, etc. Unity, for example, is great at giving you the tools to do exactly what you suggest from day one, but it by no means solves the above for you.
Apps don't normally have to care that deeply about UI latency, high cpu usage, various quirks of performance hits across platforms, VRAM, controller variations, achieving a solid 30-60 FPS minimum, battery life impact, etc. Unity, for example, is great at giving you the tools to do exactly what you suggest from day one, but it by no means solves the above for you.