It is, and you must set aside your personal feelings and treat it as public info.
If people think these digits are like some secret password, they will be treated as such and used to gatekeep access to even more restricted info and accounts. Which would be a disaster because many people have had these last four digits exposed over time. Knowing them does not prove identity.
Like it or not, many banks and other industries use it as a verification question. A collections agent is going to be explicitly and strictly forbidden by policy from disclosing it to an unknown person on the phone, and that's a good thing.
Yes, I'm pretty sure everyone has my SSN and other details, but collections agents should absolutely follow the rules. The ones that don't tend to be abusive.
You might as well assume your entire SSN is public. Most of them have been leaked someplace or other, and for anyone who was an adult before about 2000 it was common to have them pre-printed on your personal checks. It's only in the past few decades that they have suddenly become "secret."
That someone who called you on the phone happens to know your SSN last four or even the entire number should not confer any trust on your part.
If people think these digits are like some secret password, they will be treated as such and used to gatekeep access to even more restricted info and accounts. Which would be a disaster because many people have had these last four digits exposed over time. Knowing them does not prove identity.
They are public.