You’re not wrong. However, building from source wouldn’t have protected you against this specific backdoor. The upstream source tarball itself was compromised in a cleverly sneaky way.

You might read https://www.openwall.com/lists/oss-security/2024/03/29/4

"However, building from source wouldn’t have protected you against this specific backdoor." Depends on how exactly you build from source. A generic build was not the target. Andres Freund showed that the attack was targeted against a specific type of build system.

Building from git, or the github automatic tarball would have. The larger issue here is authenticating tarballs against the source.

There is no reason to believe the exploit would have been spotted earlier had the attacker included the final part in git.