Oh yes, I suppose a bad build script could undo you. In the land of iOS there are sometimes "libraries" that are just distributed as source files that you add to your own project, so I suppose that at least would qualify.
Additionally, are you willing to bet that there are no buffer overflows in gcc, sed, awk, and whatever other tools you use to build software from git on a regular basis?
Heck, some of the Google security guys have been discovering lately that you can pwn someone just by getting them to run less on a file. How many people start by doing "less README"?
Git should ideally not have this vulnerability, but panicking over this seems overkill. If you want to suck down and work with large amounts of code from a possibly malicious source, you get into virtual machines territory.