My approach aswell. Lock down ssh-agent and restrict its usage as much as possible. Securing your keys is also very reasonable but it cant silence this naging voice in the back of my head that keeps reminding me of a compromised ssh-agent or shell, whenever i authorize privileged actions.
You can also do something similar with any computer that has a TPM. It's unfortunate that people don't really know about it, but I guess the tools available aren't that user friendly
It only supports sk-ecdsa-sha2-nistp256 key format, however that is widely supported currently.