If you have enough USB host controllers, you can pass one into a Windows VM and it will have direct access to the connected audio interface (no latency issues). And if on top of that you have two GPUs, you can pass one into Windows VM as well. If you have only one GPU, passing it around will be tricky (you need to terminate all apps working with GPU before taking it away).

If you plan to run a single application in VM, you can allocate just 3-4 Gb to it.