Putting it inside docker is probably fine for most use cases but it's generally not considered to be a safe sandbox AFAIK. A docker container shares kernel with the host OS which widens the attack surface.
If you want your agent to pull untrusted code from the internet and go wild while you're doing other stuff it might not be a good choice.
Could you point to some resources which talk about how docker isn't considered a safe sandbox given the network and file system restrictions I mentioned?
I understand the sharing of kernel, while I might not be aware of all of the implications. I.e. if you have some local access or other sophisticated knowledge of the network/box docker is running on, then sure you could do some damage.
But I think the chances of a whitelisted llm endpoint returning some nefarious code which could compromise the system is actually zero. We're not talking about untrusted code from the internet. These models are pretty constrained.
If you want your agent to pull untrusted code from the internet and go wild while you're doing other stuff it might not be a good choice.