That has everything to do with data collections. It won't always be something obvious as poor zip codes. When data of many people is collected it enables the establishment of certain behavioural types. Then when you come along with the few data points they know of you, the gaps can be filled. What goods you buy, travelling behaviour, waking hours can all be used to characterize you. Regardless of whether this labelling is fair.