It's possible they could take that position, but I suspect you or your lawyer would be able to come up with a way out of most of it. For example, you could argue they were a series of interest-free loans, so there was no income to anyone (other than the forgiveness of any interest the IRS would impute to the loans). Or maybe there's an exception in the tax code for the situation you describe - I'm not a tax attorney :)

And as ebiester noted, there's an exemption for gifts under a certain amount.