They can't prevent you from entering the country. You do not have an unlimited right to bring items into the country with you, though. They can absolutely prevent you from bringing your phone across the border if you decline to unlock it
Customs law? You have an absolute right to return to the country as a citizen. You do not have an absolute right to bring whatever you want into the country, even as a citizen so they can search your belongings to see if you are carrying contraband. It's a different set of rules than if you're just on the street already in the country, crossing a border customs has a lot of leeway
I'm not saying that I agree with it, it's just the way it is
