well once you "need" that swap, it will be writing pages across the network due to the storage being external to the physical server, so the latency is terrible
Latency of swap is always terrible in comparison to RAM. RAM vs disk is already something ~1000x right? I've never characterized EBS vs trad ssd, but I would be surprised if it's more than 10x.
I don't think using swap as "emergency RAM" makes a lot of sense in 2025. The arguments in favor of swap which I find convincing are about allowing the system to evict low use pages which otherwise would not be evictable.