I’m wondering if, like looking out from behind a blanket at horror movies, if getting a moderately blurred copy of images would reduce the emotional punch of highly inappropriate pictures. Or just scaled down tiny.
If it’s already bad blurred or as a thumbnail don’t click on the real thing.
This is more or less how police do CSAM classification now. They start with thumbnails, and that's usually enough to determine whether the image is a photograph or an illustration, involves penetration, sadism etc without having to be confronted with the full image.
We’re talking about Facebook here. You shouldn’t have the assumption that the platform should be “uncensored” when it clearly is not.
Furthermore, I’ll rather have the picture of my aunt’s vacation taken down by ai mistake rather than hundreds of people getting PSTD because they have to manually review if some decapitation was real or illustrated on an hourly basis.
Currently content is flagged and moderators decide whether to take it down. Using AI, it's easy conceive a process where some uploaded content is preflagged requiring an appeal (otherwise it's the same as before, a pair of human eyes automatically looking at uploaded material).
Uploaders trying to publish rule-breaking content would not bother with an appeal that would reject them anyway.
Because edge cases exist, and it isn't worth it for a company to hire enough staff to deal with them when one user with a problem, even if that problem is highly impactful to their life, just doesn't matter when the user is effectively the product and not the customer. Once the AI works well enough, the staff is gone and the cases where someone's business or reputation gets destroyed because there are no ways to appeal a wrong decision by a machine get ignored. And of course 'the computer won't let me' or 'I didn't make that decision' is a great way for no one to ever have to feel responsible for any harms caused by such a system.
This and social media companies in the EU tend to just delete stuff because of draconian laws where content must be deleted in 24 hours or they face a fine. So companies would rather not risk it. Moderators also only have a few seconds to decide if something should be deleted or not.
I already addressed this and you're talking over it. Why are you making the assumption that AI == no appeal and zero staff? That makes zero sense, one has nothing to do with the other. The human element comes in for appeal process.
> I already addressed this and you're talking over it.
You didn't address it, you handwaved it.
> Why are you making the assumption that AI == no appeal and zero staff?
I explicitly stated the reason -- it is cheaper and it will work for the majority of instances while the edge cases won't result in losing a large enough user base that it would matter to them.
I am not making assumptions. Google notoriously operates in this fashion -- for instance unless you are a very popular creator, youtube functions like that.
> That makes zero sense, one has nothing to do with the other.
Cheaper and mostly works and losses from people leaving are not more than the money saved by removing support staff makes perfect sense and the two things are related to each other like identical twins are related to each other.
> The human element comes in for appeal process.
What does a company have to gain by supplying the staff needed to listen to the appeals when the AI does a decent enough job 98% of the time? Corporations don't exist to do the right thing or to make people happy, they are extracting value and giving it to their shareholders. The shareholders don't care about anything else, and the way I described returns more money to them than yours.
> I am not making assumptions. Google notoriously operates in this fashion -- for instance unless you are a very popular creator, youtube functions like that.
Their copyright takedown system has been around for many years and wasn't contingent on AI. It's a "take-down now, ask questions later" policy to please the RIAA and other lobby groups. Illegal/abuse material doesn't profit big business, their interest is in not having it around.
You deliberately conflated moderation & appeal process from the outset. You can have 100% AI handling of suspect uploads (for which the volume is much larger) with a smaller staff handling appeals (for which the volume is smaller), mixed with AI.
Frankly if your hypothetical upload is still rejected after that, it 99% likely violates their terms of use, in which case there's nothing to say.
> it is cheaper
A lot of things are "cheaper" in one dimension irrespective of AI, doesn't mean they'll be employed if customers dislike it.
> the money saved by removing support staff makes perfect sense and the two things are related to each other like identical twins are related to each other.
It does not make sense to have zero staff in as part of managing an appeal process (precisely to deal with edge cases and fallibility of AI), and it does not make sense to have no appeal process.
You're jumping to conclusions. That is the entire point of my response.
> What does a company have to gain by supplying the staff needed to listen to the appeals when the AI does a decent enough job 98% of the time?
AI isn't there yet, notwithstanding, if they did a good job 98% of the time then who cares? No one.
> Their copyright takedown system has been around for many years and wasn't contingent on AI.
So what? It could rely on tea leaves and leprechauns, it illustrates that whatever automation works will be relied on at the expense of any human staff or process
> it 99% likely violates their terms of use, in which case there's nothing to say.
Isn't that 1% the edge cases I am specifically mentioning are important and won't get addressed?
> doesn't mean they'll be employed if customers dislike it.
The customers on ad supported internet platforms are the advertisers and they are fine with it.
> You're jumping to conclusions. That is the entire point of my response.
Conclusions based on solid reason and evidenced by past events.
> AI isn't there yet, notwithstanding, if they did a good job 98% of the time then who cares? No one.
Until you realize that 2% of 2.89billion monthly users is 57,800,000.
Then what is freedom of speech if every plattform deletes your content? Does it even exist? Facebook and co. are so ubiquitous, we shouldn't just apply normal laws to them. They are bigger than governments.
Freedom of speech means that the government can't punish you for your speech. It has absolutely nothing to do with your speech being widely shared, listened to, or even acknowledged. No one has the right to an audience.
This has always been the case. If the monks didn't want to copy your work, it didn't get copied by the monks. If the owners of a printing press didn't want to print your work, you didn't get to use the printing press. If Random House didn't want to publish your manifesto, you do not get to compel them to publish your manifesto.
The first amendment is multiple freedoms. Your freedom of speech is that the government shouldn't stop you from using your own property to do something. You are free to print out leaflets and distribute them from your porch. If nobody wants to read your pamphlets that's too damn bad, welcome to the free market of ideas buddy.
The first amendment also protects Meta's right of free association. Forcing private companies to platform any content submitted to them would outright trample their right. Meta has a right to not publish your work so that they can say "we do not agree with this work and will not use our resources to expand it's reach".
We have, in certain cases, developed system that treats certain infrastructure as a regulated pipe that is compelled to carry everything, like with classic telephone infrastructure. The reason for that, is it doesn't make much sense to require every company to put up their own physical wires, it's dumb and wasteful. Social networks have zero natural monopoly and should not be treated as common carriers.
Not if we retain control and each deploy our own moderation individually, relying on trust networks to pre-filter. That probably won't be allowed to happen, but in a rational, non-authoritarian world, this is something that machine learning can help with.
The solution to most social media problems in general is:
`select * from posts where author_id in @follow_ids order by date desc`
At least 90% of the ills of social media are caused by using algorithms to prioritize content and determine what you're shown. Before these were introduced, you just wouldn't see these types of things unless you chose to follow someone who chose to post it, and you didn't have people deliberately creating so much garbage trying to game "engagement".
There's a huge gap between "we will scan our servers for illegal content" and "your device will scan your photos for illegal content" no matter the context. The latter makes the user's device disloyal to its owner.
The choice was between "we will upload your pictures unencrypted and do with them as we like, including scan them for CSAM" vs. "we will upload your pictures encrypted and keep them encrypted, but will make sure beforehand on your device only that there's no known CSAM among it".
> we will upload your pictures unencrypted and do with them as we like
Curious, I did not realize Apple sent themselves a copy of all my data, even if I have no cloud account and don't share or upload anything. Is that true?
No. The entire discussion only applies to images being uploaded (or about to be uploaded) to iCloud. By default in iOS all pictures are saved locally only (so the whole CSAM scanning discussion would not have applied anyway), but that tends to fill up a phone pretty quickly.
With the (optional) iCloud, you can (optionally) activate iCloud Photos to have a photo library backed up in the cloud and shared among all your devices (and, in particular, with only thumbnails and metadata stored locally and the full resolution pictures only downloaded on demand).
These are always encrypted, with either the keys being with Apple ("Standard Data Protection") so that they're recoverable when the user loses phone or password, or E2E ("Advanced Data Protection") if the user so choses, thus irrecoverable.
It seems to me that in the latter case images are not scanned at all (neither on device nor in the cloud), and it's unclear to me whether they're scanned in the former case.
Apple doesn't do this. But other service providers do (Dropbox, Google, etc).
Other service providers can scan for CSAM from the cloud, but Apple cannot. So Apple might be one of the largest CSAM hosts in the world, due to this 'feature'.
> Other service providers can scan for CSAM from the cloud
I thought the topic was on-device scanning? The great-grandparent claim seemed to be that Apple had to choose between automatically uploading photos encrypted and not scanning them, vs. automatically uploading photos unencrypted and scanning them. The option for "just don't upload stuff at all, and don't scan it either" was conspicuously absent from the list of choices.
Why, do other phone manufacturers do this auto-upload-and-scan without asking?
I think FabHK is saying that Apple planned to offer iCloud users the choice of unencrypted storage with server-side scanning, or encrypted storage with client-side scanning. It was only meant to be for things uploaded to iCloud, but deploying such technologies for any reason creates a risk of expansion.
Apple itself has other options, of course. It could offer encrypted or unencrypted storage without any kind of scanning, but has made the choice that it wants to actively check for CSAM in media stored on its servers.
"In 2023, ESPs submitted 54.8 million images to the CyberTipline of which 22.4 million (41%) were unique. Of the 49.5 million videos reported by ESPs, 11.2 million (23%) were unique."
And, indeed, this is why we should not expect the process to stop. Nobody is rallying behind the rights of child abusers and those who traffic in child abuse material. Arguably, nor should they. The slippery slope argument only applies if the slope is slippery.
This is analogous to the police's use of genealogy and DNA data to narrow searches for murderers, who they then collected evidence on by other means. There's is risk there, but (at least in the US) you aren't going to find a lot of supporters of the anonymity of serial killers and child abusers.
There are counter-arguments to be made. Germany is skittish about mass data collection and analysis because of their perception that it enabled the Nazi war machine to micro-target their victims. The US has no such cultural narrative.
> And, indeed, this is why we should not expect the process to stop. Nobody is rallying behind the rights of child abusers and those who traffic in child abuse material. Arguably, nor should they.
I wouldn't be so sure.
When Apple was going to introduce on-device scanning they actually proposed to do it in two places.
• When you uploaded images to your iCloud account they proposed scanning them on your device first. This is the one that got by far the most attention.
• The second was to scan incoming messages on phones that had parental controls set up. The way that would have worked is:
1. if it detects sexual images it would block the message, alert the child that the message contains material that the parents think might be harmful, and ask the child if they still want to see it. If the child says no that is the end of the matter.
2. if the child say they do want to see it and the child is at least 13 years old, the message is unblocked and that is the end of the matter.
3. if the child says they do want to see it and the child is under 13 they are again reminded that their parents are concerned about the message, again asked if they want to view it, and told that if they view it their parents will be told. If the child says no that is the end of the matter.
4. If the child says yes the message is unblocked and the parents are notified.
This second one didn't get a lot of attention, probably because there isn't really much to object to. But I did see one objection from a fairly well known internet rights group. They objected to #4 on the grounds that the person sending the sex pictures to your under-13 year old child sent the message to the child, so it violates the sender's privacy for the parents to be notified.
If it's the EFF, I think they went out on a limb on this one that not a lot of American parents would agree with. "People have the right to communicate privately without backdoors or censorship, including when those people are minors" (emphasis mine) is a controversial position. Arguably, not having that level of privacy is the curtailment on children's rights.
The cultural narrative is actually extremely popular in a 10% subset of the population that is essentially fundamentalist christian who are terrified of the government branding them with "the mark of the beast".
The problem is that their existence actually poisons the discussion because these people are absurd loons who also blame the gays for hurricanes and think the democrats eat babies.
Apple is already categorizing content on your device. Maybe they don't report what categories you have. But I know if I search for "cat" it will show me pictures of cats on my phone.
No, they had backlash against using AI on devices they don’t own to report said devices to police for having illegal files on them. There was no technical measure to ensure that the devices being searched were only being searched for CSAM, as the system can be used to search for any type of images chosen by Apple or the state. (Also, with the advent of GenAI, CSAM has been redefined to include generated imagery that does not contain any of {children, sex, abuse}.)
That’s a very very different issue.
I support big tech using AI models running on their own servers to detect CSAM on their own servers.
I do not support big tech searching devices they do not own in violation of the wishes of the owners of those devices, simply because the police would prefer it that way.
It is especially telling that iCloud Photos is not end to end encrypted (and uploads plaintext file content hashes even when optional e2ee is enabled) so Apple can and does scan 99.99%+ of the photos on everyone’s iPhones serverside already.
> Also, with the advent of GenAI, CSAM has been redefined to include generated imagery that does not contain any of {children, sex, abuse}
It hasn’t been redefined. The legal definition of it in the UK, Canada, Australia, New Zealand has included computer generated imagery since at least the 1990s. The US Congress did the same thing in 1996, but the US Supreme Court ruled in the 2002 case of Ashcroft v Free Speech Coalition that it violated the First Amendment. [0] This predates GenAI because even in the 1990s people saw where CGI was going and could foresee this kind of thing would one day be possible.
Added to that: a lot of people misunderstand what that 2002 case held. SCOTUS case law establishes two distinct exceptions to the First Amendment – child pornography and obscenity. The first is easier to prosecute and more commonly prosecuted; the 2002 case held that "virtual child pornography" (made without the use of any actual children) does not fall into the scope of the child pornography exception – but it still falls into the scope of the obscenity exception. There is in fact a distinct federal crime for obscenity involving children as opposed to adults, 18 USC 1466A ("Obscene visual representations of the sexual abuse of children") [1] enacted in 2003 in response to this decision. Child obscenity is less commonly prosecuted, but in 2021 a Texas man was sentenced to 40 years in prison over it [2] – that wasn't for GenAI, that was for drawings and text, but if drawings fall into the legal category, obviously GenAI images will too. So actually it turns out that even in the US, GenAI materials can legally count as CSAM, if we define CSAM to include both child pornography and child obscenity – and this has been true since at least 2003, long before the GenAI era.
Thanks for the information. However I am unconvinced that SCOTUS got this right. I don’t think there should be a free speech exception for obscenity. If no other crime (like against a real child) is committed in creating the content, what makes it different from any other speech?
> However I am unconvinced that SCOTUS got this right. I don’t think there should be a free speech exception for obscenity
If you look at the question from an originalist viewpoint: did the legislators who drafted the First Amendment, and voted to propose and ratify it, understand it as an exceptionless absolute or as subject to reasonable exceptions? I think if you look at the writings of those legislators, the debates and speeches made in the process of its proposal and ratification, etc, it is clear that they saw it as subject to reasonable exceptions – and I think it is also clear that they saw obscenity as one of those reasonable exceptions, even though they no doubt would have disagreed about its precise scope. So, from an originalist viewpoint, having some kind of obscenity exception seems very constitutionally justifiable, although we can still debate how to draw it.
In fact, I think from an originalist viewpoint the obscenity exception is on firmer ground than the child pornography exception, since the former is arguably as old as the First Amendment itself is, the latter only goes back to the 1982 case of New York v. Ferber. In fact, the child pornography exception, as a distinct exception, only exists because SCOTUS jurisprudence had narrowed the obscenity exception to the point that it was getting in the way of prosecuting child pornography as obscene – and rather than taking that as evidence that maybe they'd narrowed it a bit too far, SCOTUS decided to erect a separate exception instead. But, conceivably, SCOTUS in 1982 could have decided to draw the obscenity exception a bit more broadly, and a distinct child pornography exception would never have existed.
If one prefers living constitutionalism, the question is – has American society "evolved" to the point that the First Amendment's historical obscenity exception ought to jettisoned entirely, as opposed to merely be read narrowly? Does the contemporary United States have a moral consensus that individuals should have the constitutional right to produce graphic depictions of child sexual abuse, for no purpose other than their own sexual arousal, provided that no identifiable children are harmed in its production? I take it that is your personal moral view, but I doubt the majority of American citizens presently agree – which suggests that completely removing the obscenity exception, even in the case of virtual CSAM material, cannot currently be justified on living constitutionalist grounds either.
My understanding was the FP risk. The hashes were computed on device, but the device would self-report to LEO if it detects a match.
People designed images that were FPs of real images. So apps like WhatsApp that auto-save images to photo albums could cause people a big headache if a contact shared a legal FP image.
No, the point of on-device scanning is to enable authoritarian government overreach via a backdoor while still being able to add “end to end encryption” to a list of product features for marketing purposes.
If Apple isn’t free to publish e2ee software for mass privacy without the government demanding they backdoor it for cops on threat of retaliation, then we don’t have first amendment rights in the USA.
You misunderstand me. The issue is that Apple is theoretically being retaliated against, by the state, if they were to publish non-backdoored e2ee software.
Apple does indeed in theory have a right to release whatever iOS features they like. In practice, they do not.
Everyone kind of tacitly acknowledged this, when it was generally agreed upon that Apple was doing the on-device scanning thing "so they can deploy e2ee". The quiet part is that if they didn't do the on-device scanning and released e2ee software without this backdoor (which would then thwart wiretaps), the FBI et al would make problems for them.
The same reason they made iMessage e2ee, which happened many years before CSAM detection was even a thing.
User privacy. Almost nobody trades in CSAM, but everyone deserves privacy.
Honestly, this isn’t about CSAM at all. It’s about government surveillance. If strong crypto e2ee is the hundreds-of-millions-of-citizens device default, and there is no backdoor, the feds will be upset with Apple.
This is why iCloud Backup (which is a backdoor in iMessage e2ee) is not e2ee by default and why Apple (and the state by extension) can read all of the iMessages.
I didn't ask why they would want E2EE. I asked why they would want E2EE without CSAM detection when they literally developed a method to have both. It's entirely reasonable to want privacy for your users AND not want CSAM on your servers.
> Honestly, this isn't about CSAM at all.
It literally is the only thing the technology is any good for.
Already happened/happening. I have an ex-coworker that left my current employer for my state's version of the FBI. Long story short, the government has a massive database to crosscheck against. Often times, the would use automated processes to filter through suspicious data they would collect during arrests.
If the automated process flags something as a potential hit, then they, the humans, would then review those images to verify. Every image/video that is discovered to be a hit is also inserted into a larger dataset as well. I can't remember if the Feds have their own DB (why wouldn't they?), but the National Center for Missing and Exploited Children run a database that I believe government agencies use too. Not to mention, companies like Dropbox, Google, etc.. all has against the database(s) as well.
