It's a custom engine by nintendo, also seems to be user by other games like super mario odyssey.

Changing graphics settings would be accomplished using patches, more or less the same idea as making cheats in older games. Find the value somewhere in their code that corresponds to render distance and change that, as an example.

Emulators for older systems can do more impressive things. Graphics pipelines tend to be a certain shape and use certain data types, so once you're already emulating at the GPU level you can do things like upscale old textures (works great on cell-shaded games like megaman legends). For 2d games you can use a dedicated pixel art scaling algorithm.

Pretty much all 3D games can output at a higher resolution than the original hardware allowed, just due to how the hardware is set up. You're game isn't responsible for deciding the output resolution, the GPU is, essentially. Changes aspect ratios is much harder and often requires patching the games themselves to make it work properly.

Other common patches are things like higher FPS. By default the new zelda game plays at 20fps, but there are patches to play them at 30 and 60 fps for a smoother experience. Those are once again actually reverse engineering and patching the game files though.