His arguments can be validated only based on its internal logical structure. One should not try to invalidate it based on his claimed credibility. That's ad hominem.
A: "All rodents are mammals, but a weasel isn't a rodent, so it can't be a mammal."
B: "I'm sorry, but I'd prefer to trust the opinion of a trained zoologist on this one."
B's argument is ad hominem: he is attempting to counter A not by addressing his argument, but by casting doubt on A's credentials. Note that B is polite and not at all insulting.
Credibility is a heuristic to determine what arguments should be evaluated and which should be discarded. It is not reasonable to expect every person to critically evaluate the word of every person who has something to say on a subject.
