The main "backdoor" to Signal is that having access to the phone can leak all of Signal's data. If the phone OS is backdoored, then Signal is already compromised. Anyway, the point is not to make it impossible to exfiltrate data, but to make it as hard as possible.