AI doesn't know what either porn or children are. It finds correlations between aspects of inputs and the labels porn and children. Even if you did develop an advanced enough AI that could develop a good enough idea of what porn and children are, how would you ever verify that it is indeed capable of recognizing child porn without plugging in samples for it to flag?
LLMs don't "know" anything. But as you say, they can identify correlations between content "porn" and a target image; between content labeled "children" and a target image. If a target image scores high in both, then it can flag child porn, all without being trained on CSAM.
But things correlated with porn != porn and things correlated with children != children. For example, in our training set, no porn contains children, so the presence of children would mean it's not porn. Likewise all images of children are clothed, so no clothes means it's not a child. You know it's ridiculous because you know things, the AI does not.
Nevermind the importance of context, such as distinguishing a partially clothed child playing on a beach from a partially clothed child in a sexual situation.
So it is able to correlate an image as porn and also correlate an image as containing children. Seems like it should be able to apply an AND operation to this result and identify new images that are not part of the data set.
No, it found elements in an image that it tends to find in images labelled porn in the training data. It finds elements in an image it tends to find in images labelled child in the training data. If the training data is not representative, then the statistical inference is meaningless. Images that are unlike any in the training set may not trigger either category if they are lacking the things the AI expects to find, which may be quite irrelevant to what humans care about.
AI doesn’t understand context either — it can’t tell the difference between an innocent photo of a baby in a bathtub with a parent, a telehealth photo, or something malicious. Google is using AI in addition to hashing, and both systems can get it wrong. With AI you’re always dealing with confidence levels, not certainty. No model in history has ever had 100% confidence on anything.
A scanning system will never be perfect. But there is a better approach: what the FTC now requires Pornhub to do. Before an image is uploaded, the platform scans it; if it’s flagged as CSAM, it simply never enters the system. Platforms can set a low confidence threshold and block the upload entirely. If that creates too many false positives, you add an appeals process.
The key difference here is that upload-scanning stops distribution before it starts.
What Google is doing is scanning private cloud storage after upload and then destroying accounts when their AI misfires. That doesn’t prevent distribution — it just creates collateral damage.
It also floods NCMEC with automated false reports. Millions of photos get flagged, but only a tiny fraction lead to actual prosecutions. The system as it exists today isn’t working for platforms, law enforcement, or innocent users caught in the blast radius.
