In theory, yes. In practice it's just used as an epithet against specific groups. For instance, you almost never hear folks from South America being called colonizers despite how Spain and Portugal carved up that continent, meanwhile Jews are frequently called colonizers of their own ancestral homeland.