This is brilliant misdirection. The crypto is great, but where and how your content is scanned was never the problem to begin with, or at least, a small part.
The accuracy of not just the detection rate (there are some outlandish claims of once in billion here), but also the accuracy of the NCMEC database are really the main concern, as well as Apple keeping this system limited to this specific scope.
Interesting aside: I once attempted to get access to PhotoDNA, essentially the only insurance against malicious actors abusing upload fields on your website to "digitally swat" you (as has happened to a twitch streamer with an open Dropbox folder), and there is no way you'll get access without a department of lawyers. Why is NCMEC is so protective of an API with rate limits and automated reporting features and then would let Apple ship a bloom filter.
I remember reading at one point that perceptual hashes used in PhotoDNA could be reversed to reconstruct the image, which is why the database is closely guarded. I assume the bloom filter implementation significantly raises the bar on reversing.
The accuracy of not just the detection rate (there are some outlandish claims of once in billion here), but also the accuracy of the NCMEC database are really the main concern, as well as Apple keeping this system limited to this specific scope.
Interesting aside: I once attempted to get access to PhotoDNA, essentially the only insurance against malicious actors abusing upload fields on your website to "digitally swat" you (as has happened to a twitch streamer with an open Dropbox folder), and there is no way you'll get access without a department of lawyers. Why is NCMEC is so protective of an API with rate limits and automated reporting features and then would let Apple ship a bloom filter.