The idea with HSM-backed keys is that even in case of compromise, you can clean up without having to rotate the keys. It also makes auditing easier as you can ensure that if your machine was powered down or offline then you are guaranteed the keys weren't used during that timeframe.