"This is impolite" is not a truth statement, it is a value judgment. It can be impolite to do something even if someone has said that they don't mind.