Because that means you are sharing kernel with the sandboxed agent. Virtualization presents an infinitely smaller attack surface.

If there is any attack surface within a properly-configured container, that's a kernel bug, right?

Probably. If I remember correctly, containers on Linux are implemented using the kernel's namespaces. The same ones which became famous for the vulnerabilities they surfaced in previously unexercised code.

Then use SmolVM

I'm already using virtdev, my own solution built on top of QEMU.