Curious if you could find errors by comparing the results from the different models. Places where models disagree with each other more often would be areas that I would want to target for error checking.
> Places where models disagree with each other more often would be areas that I would want to target for error checking.
This is a great idea if your goal is to maximize the rate at which things you look at turn out to be errors. (On at least one side.)
But it's guaranteed to miss cases where every model makes the same inexplicable-to-the-human-eye mistake, and those cases would appear to be especially interesting.
This is a good idea and there are actually 2 objectives when one wants to clean its dataset:
- you might want to optimize your time and correct as many errors as you can as fast as you can. Using several models will help you ion that case, adn that's actually what we've been focusing on so far.
- you might want to find the most ambiguous cases where you really need to improve your models as those edge cases are the ones causing the problems you have in production.
Those 2 objectives are quite opposite. In the first case, you want to find the "easiest" errors, while in the other one, you want to focus on edge cases and you then probably need to look at errors with intermediate scores, where nothing is really sure..
“Annotator agreement” is a measure of confidence in the correctness of labels. And you should always keep an eye out for how these are handled, when reading papers that present a dataset.
Saying we should start doing model agreement is a really good idea imho.
