"“We take action against illegal content on X, including Child Sexual Abuse Material (CSAM), by removing it, permanently suspending accounts, and working with local governments and law enforcement as necessary,” X Safety said. “Anyone using or prompting Grok to make illegal content will suffer the same consequences as if they upload illegal content.”
How about not enabling generating such content, at all?
Given X can quite simply control what Grok can and can't output, wouldn't you consider it a duty upon X to build those guardrails in for a situation like CSAM? I don't think there's any grey area here to argue against it.
I am, in general, pretty anti-Elon, so I don't want to be seen as taking _his_ side here, and I am definitely anti-CSAM, so let's shift slightly to derivative IP generation.
Where does the line fall between provider responsibility when providing a tool that can produce protected work, and personal responsibility for causing it to generate that work?
It feels somewhat more clearcut when you say to AI, "Draw me an image of Mickey Mouse", but why is that different than photocopying a picture of Mickey Mouse, and using Photoshop to draw a picture of Mickey Mouse? Photo copiers will block copying a dollar bill in many cases - should they also block photos of Mickey Mouse? Should they have received firmware updates whenever Steamboat Willy fell into public domain, such that they can now be allowed to photocopy that specific instance of Mickey Mouse, but none other?
This is a slippery slope, the idea that a person using the tool should hold the tool responsible for creating "bad" things, rather than the person themselves being held responsible.
Maybe CSAM is so heinous as to be a special case here. I wouldn't argue against it specifically. But I do worry that it shifts the burden of responsibility onto the AI or the model or the service or whatever, rather than the person.
Another thing to think about is whether it would be materially different if the person didn't use Grok, but instead used a model on their own machine. Would the model still be responsible, or would the person be responsible?
> Where does the line fall between provider responsibility when providing a tool that can produce protected work, and personal responsibility for causing it to generate that work?
There's one more line at issue here, and that's the posting of the infringing work. A neutral tool that can generate policy-violating material has an ambiguous status, and if the tool's output ends up on Twitter then it's definitely the user's problem.
But here, it seems like the Grok outputs are directly and publicly posted by X itself. The user may have intended that outcome, but the user might not have. From the article:
>> In a comment on the DogeDesigner thread, a computer programmer pointed out that X users may inadvertently generate inappropriate images—back in August, for example, Grok generated nudes of Taylor Swift without being asked. Those users can’t even delete problematic images from the Grok account to prevent them from spreading, the programmer noted.
Overall, I think it's fair to argue that ownership follows the user tag. Even if Grok's output is entirely "user-generated content," X publishing that content under its own banner must take ownership for policy and legal implications.
This is also legally problematic: many jurisdictions now have specific laws about the synthesis of CSAM or modifying peoples likenesses.
So exactly who is considered the originator is a pretty legally relevant question particularly if Grok is just off doing whatever and then posting it from your input.
"The persistent AI bot we made treated that as a user instruction and followed it" is a heck of a chain of causality in court, but you also fairly obviously don't want to allow people to laundry intent with AI (which is very much what X is trying to do here).
Maybe I'm being too simplistic/idealistic here - but if I had a company that controlled an LLM product, I wouldn't even think twice about banning CSAM outputs.
You can have all the free speech in the world, but not with the vulnerable and innocent children.
I don't know how we got to the point where we can build things with no guardrails and just expect the user to use it legally? I think there should be responsibility on builders/platform owners to definitely build guardrails in on things that are explicitly illegal and morally repugnant.
>I wouldn't even think twice about banning CSAM outputs.
Same, honestly. And you'll probably catch a whole lot of actual legitimate usage in that net, but it's worth it.
But you'll also miss some. You'll always miss some, even with the best guard rails. But 99% is better than 0%, I agree.
> ... and just expect the user to use it legally?
I don't think it's entirely the responsibility of the builder/supplier/service to ensure this, honestly. I don't think it can be. You can sell hammers, and you can't guarantee that the hammer won't be used to hurt people. You can put spray cans behind cages and require purchasers to be 18 years old, but you can't stop the adult from vandalism. The person has to be held responsible at a certain point.
I bet most hammers (non-regulated), spray cans (lightly regulated) and guns (heavily regulated) that are sold are used for their intended purposes. You also don't see these tools manufacturers promoting or excusing their unintended usage as well.
There's also a difference between a tool manufacturer (hardware or software) and a service provider: once the tool is on the user's hands, it's outside of the manufacturer's control.
In this case, a malicious user isn't downloading Grok's model and running it on their GPU. They're using a service provided by X, and I'm of the opinion that a service provider starts to be responsible once the malicious usage of their product gets relevant.
> I don't know how we got to the point where we can build things with no guardrails and just expect the user to use it legally?
Historically tools have been uncensored, yet also incredibly difficult and time-consuming to get good results with.
Why spend loads of effort producing fake celebrity porn using photoshop or blender or whatever when there's limitless free non-celebrity porn online? So photoshop and blender didn't need any built-in censorship.
But with GenAI, the quantitive difference in ease-of-use results in qualitative difference in outcome. Things that didn't get done when it needed 6 months of practice plus 1 hour per image are getting done now it needs zero practice and 20 seconds per image.
> Where does the line fall between provider responsibility when providing a tool that can produce protected work, and personal responsibility for causing it to generate that work?
If you operate the tool, you are responsible. Doubly so in a commercial setting. If there are issues like Copyright and CSAM, they are your responsibility to resolve.
If Elon wanted to share out an executable for Grok and the user ran it on their own machine, then he could reasonably sidestep blame (like how photoshop works). But he runs Grok on his own servers, therefore is morally culpable for everything it does.
Your servers are a direct extension of yourself. They are only capable of doing exactly what you tell them to do. You owe a duty of care to not tell them to do heinous shit.
It's simpler to regulate the source of it than the users. The scale that genAI can do stuff is much, much different than photocopying + Photoshop, scale and degree matter.
So, back in the 90s and 2000s, you could get The Gimp image editor, and you could use the equivalent of Word Art to take a word or phase and make it look cool, with effects like lava or glowing stone, or whatever. The Gimp used ImageMagick to do this, and it legit looked cool at the time.
If you weren't good at The Gimp, which required a lot of knowledge, you could generate a cool website logo by going to a web server that someone built, giving them a word or phrase, and then selecting the pre-built options that did the same thing - you were somewhat limited in customization, but on the backend, it was using ImageMagick just like The Gimp was.
If someone used The Gimp or ImageMagick to make copyrighted material, nobody would blame the authors of The Gimp, right? The software were very nonspecific tools created for broad purposes, that of making images. Just because some bozo used them to create a protected image of Mickey Mouse doesn't mean that the software authors should be held accountable.
But if someone made the equivalent of one of those websites, and the website said, "click here to generate a random picture of Mickey Mouse", then it feels like the person running the website should at least be held partially responsible, right? Here is a thing that was created for the specific purpose of breaking the law upon request. But what is the culpability of the person initiating the request?
Anyway, the scale of AI is staggering, and I agree with you, and I think that common decency dictates that the actions of the product should be limited when possible to fall within the ethics of the organization providing the service, but the responsibility for making this tool do heinous things should be borne by the person giving the order.
I think yes CSAM and other harmful outputs are a different and more heinous problem, I also think the responsibility is different between someone using a model locally and someone promoting grok on twitter.
Posting a tweet asking Grok to transform a picture of a real child into CSAM is no different, in my mind, than asking a human artist on twitter to do the same. So in the case of one person asking another person to perform this transformation, who is responsible?
I would argue that it’s split between the two, with slightly more falling on the artist. The artist has a duty to refuse the request and report the other person to the relevant authorities. If that artist accepted the request and then posted the resulting image, twitter then needs to step in and take action against both users.
Even if you can’t reliably control it, if you make a tool that generates CSAM you’ve made a CSAM generator. You have a moral responsibility to either make your tool unavailable, or figure out how to control it.
I'm not sure I agree with this specific reasoning. Consider this, any given image viewer can display CSAM. Is it a CSAM viewer? Do you have a moral responsibility to make it refuse to display CSAM? We can extend it to anything from graphics APIs, to data storage, etc.
There's a line we have to define that I don't think really exists yet, nor is it supported by our current mental frameworks. To that end, I think it's just more sensible to simply forbid it in this context without attempting to ground it. I don't think there's any reason to rationalize it at all.
I think the question might come down to whether Grok is a "tool" like a paintbrush or Photoshop, or if Grok is some kind of agent of creation, like an intern. If I ask an art intern to make a picture of CSAM and he does it, who did wrong?
If Photoshop had a "Create CSAM" button and the user clicked it, who did wrong?
I think a court is going to step in and help answer these questions sooner rather than later.
Normalizing AI as being human equivalent means the AI is legally culpable for its own actions rather than its creators or the people using it, and not guilty of copyright infringement for having been trained on proprietary data without consent.
I happen to agree with you that the blame should be shared, but we have a lot of people in this thread saying "You can't blame X or Grok at all because it's a mere tool."
From my knowledge (albeit limited) about the way LLMs are set up, they most definitely have abilities to include guardrails of what can't be produced. ChatGPT has some responses to prompts which stops users from proceeding.
And X specifically: there have many cases of X adjusting Grok where Grok was not following a particular narrative on political issues (won't get into specifics here). But it was very clear and visible. Grok had certain outputs. Outcry from certain segments. Grok posts deleted. Trying the same prompts resulted in a different result.
From my (admittedly also limited) understanding, there’s no bulletproof way to say “do NOT generate X” as it’s not non-deterministic and you can’t reverse engineer and excise the CSAM-generating parts of a model. “AI jailbreak prompts” are a thing.
Well it’s certainly horrible that they’re not even trying, but not surprising (I deleted my X account a long time ago).
I’m just wondering if from a technical perspective it’s even possible to do it in a way that would 100% solve the problem, and not turn it into an arms race to find jailbreaks. To truly remove the capability from the model, or in its absence, have a perfect oracle judge the output and block it.
Again, I'm not the most technical, but I think we need to step back and look at this holistically. Given Grok's integration with X, there could be other methods of limiting the production and dissemination of CSAM.
For arguments sake, let's assume Grok can't reliably have guardrails in place to stop CSAM. There could be second and third order review points where before an image is posted by Grok, another system could scan the image to verify whether it's CSAM or not, and if the confidence is low, then human intervention could come into play.
I think the end goal here is prevention of CSAM production and dissemination, not just guardrails in an LLM and calling it a day.
Given how spectacular the failure of EVERY attempt to put guardrails on LLMs has been, across every single company selling LLM access, I'm not sure that's a reasonable belief.
The guardrails have mostly worked. They have never ever been reliable.
Yes, every image generation tool can be used to create revenge porn. But there are a bunch of important specifics here.
1. Twitter appears to be taking no effort to make this difficult. Even if people can evade guardrails this does not make the guardrails worthless.
2. Grok automatically posts the images publicly. Twitter is participating not only in the creation but also the distribution and boosting of this content. The reason why a ton of people doing this is not because they personally want to jack it to somebody, but because they want to humiliate them in public.
3. Decision makers at twitter are laughing about what this does to the platform and its users when they "post a picture of this person in their underwear" button is available next to every woman who posts on the platform. Even here they are focusing only on the illegal content, as if mountains of revenge porn being made of adult women isn't also odious.
> but output is directly connected to its input and blame can be proportionally shared
X can actively work to prevent this. They aren't. We aren't saying we should blame the person entering the input. But, we can say that the side producing CSAM can be held responsible if they choose to not do anything about it.
> Isn't this a problem for any public tool? Adversarial use is possible on any platform
Yes. Which is why the headline includes: "no fixes announced" and not just "X blames users for Grok-generated CSAM."
Grok is producing CSAM. X is going to continue to allow that to happen. Bad things happen. How you respond is essential. Anyone who is trying to defend this is literally supporting a CSAM generation engine.
It is trivially easy to filter this with an LLM or even just a basic CLIP model. Will it be 100% foolproof? Not likely. Is it better than doing absolutely nothing and then blaming the users? Obviously. We've had this feature in the image generation tools since the first UI wrappers around Stable Diffusion 1.0.
An analogy: if you're running the zoo, the public's safety is your job for anyone who visits. It's of course also true that sometimes visitors act like idiots (and maybe should be prosecuted), and also that wild animals are not entirely predictable, but if the leopards are escaping, you're going to be judged for that.
Maybe because sometimes they're kids? You gotta kid-proof stuff in a zoo.
Also, punishment is a rather inefficient way to teach the public anything. The people who come through the gate tomorrow probably won't know about the punishment. It will often be easier to fix the environment.
Removing troublemakers probably does help in the short term and is a lot easier than punishing.
If the personal accountability happened at the speed and automation level that X allows Grok to produce revenge porn and CSAM, then I'd agree with you.
Yep. "Oh grok is being too woke" gets musk to comment that they'll fix it right away. But turn every woman on the platform into a sex object to be the target of humiliation? That's just good fun apparently.
I even think that the discussion focusing on csam risks missing critical stuff. If musk manages to make this story exclusively about child porn and gets to declare victory after taking basic steps to address that without addressing the broader problem of the revenge porn button then we are still in a nightmare world.
Women should be able to exist in public without having to constantly have porn made of their likeness and distributed right next to their activity.
You always have liability. If you put something there you tell the court that you see the problem and are trying to prevent it. It often becomes easier to get out of liability if you can show the courts you did your best to prevent this. Courts don't like it when someone is blatantly unaware of things - ignorance is not a defense if "a reasonable person" would be aware of it. If this was the first AI in 2022 you could say "we never thought about that" and maybe get by, but by 2025 you need to tell the court "we are aware of the issue, and here is why we think we had reasonable protections that the user got around".
How about policing CSAM at all? I can still vividly remember firehose API access and all the horrible stuff you would see on there. And if you look at sites like tk2dl you can still see most of the horrible stuff that does not get taken down.
It's on X, not some fringe website that many people in the world don't access.
Regardless of how fringe, I feel like it should be in everyones best interests to stop/limit CSAM as much as they reasonably can without getting into semantics of who requested/generated/shared it.
> How about not enabling generating such content, at all?
Or, if they’re being serious about the user-generated content argument, criminally referring the users asking for CSAM. This is hard-liability content.
This is probably harder because it's synthetic and doesn't exist in PhotoDNA database.
Also, since Grok is really good in getting the context, something akin to "remove their T-shirt" would be enough to generate a picture someone wanted, but very hard to find using keywords.
IMO they should mass hide ALL the images created since then specific moment, and use some sort of the AI classifier to flag/ban the accounts.
Willing to bet that X premium signups have shot up because of this feature. Currently this is the most convenient tool to generate porn of anything and everything.
I don’t think anyone can claim that it’s not the user’s fault. The question is whether it’s the machine’s fault (and the creator and administrator - though not operator) as well.
The article claims Grok was generating nude images of Taylor Swift without being prompted and that there was no way for the user to take those images down
I don't know how common this is, or what the prompt was that inadvertently generated nudes. But it's at least an example where you might not blame the user
Yeah but “without being asked” here means the user has to confirm they are 18+, choose to enable NSFW video, select “spicy” in Grok’s video generation settings and then prompt “Taylor Swift celebrating Coachella with the boys”. The prompt seems fine but the rest of it is clearly “enable adult content generation”.
I know they said “without being prompted” here but if you click through you’ll see what the person actually selected (“spicy” is not default and is age-gated and opt-in via the nsfw wall).
Let’s not lose sight of the real issue here: Grok is a mess from top to bottom run by an unethical, fickle Musk. It is the least reliable LLM of the major players and musk’s constant fiddling with it so it doesn’t stray too far from his worldview invalidates the whole project as far as I’m concerned.
Isn't it a strict liability crime to posses it in the US? So if AI-generated apparent CSAM counts as CSAM legally (not sure on that) then merely storing it on their servers would make X liable.
You are only liable if you know - or should know - that you possess it. You can help someone out by mailing their sealed letter containing CSAM and be fine since you have no reason to suspect the sealed letter isn't legal. X can store CSAM so long as they have reason to think it is legal.
Note that things change. In the early days of twitter (pre X) they could get away with not thinking of the issue at all. As technology to detect CSAM marches on they need to use it (or justify why it shouldn't be used - too many false positives???). As a large platform for such content they need to push the state of the art in such detection.. At no point do they need perfection - but they need to show they are doing their reasonable best to stop this.
The above is of course my opinion. I think the courts will go a similar direction, but time will tell...
> You are only liable if you know - or should know - that you possess it.
Which he does and responded with “I will blame and punish users.” Which yeah, you should, but you also need to fix your bot. He’s certainly has no issue doing that when Grok outputs claims/arguments that make him look bad or otherwise engages in what he considers “wrongthink,” but suddenly when there are real, serious consequences he gets to hide behind “it’s just a user problem”?
This is the same thing YouTube and social media companies have been getting away with for so long. They claim their algorithms will take care of content problems, then when they demonstrably fail they throw their hands up and go “whoops! Sorry we are just too big for real people to handle all of it but we’ll get it right this time.” Rinse repeat.
Blame and punish should be a part of this. However that only works if you can find who to blame and punish. We also should put guard rails on so people don't make mistakes. (generating CSAM should not be an easy mistake to make when you don't intend it, but in other contexts someone may accidentally ask for the wrong thing)
I think platforms that host user-generated content are (rightly) treated differently. If I posted a base64 of CSAM in this comment it would be unreasonable to shut down HN.
The questions then, for me, are:
* Is Grok considered a tool for the user to generate content for X or is Grok/X considered similar to a vendor relationship
* Is X more like Backpage (not protective enough) than other platforms
I’m sure this is going to court, at least for revenge porn stuff. But why would anyone do this to their platform? Crazy. X/Twitter is full of this stuff now.
I don't think you can argue yourself out of "The Grok account is owned and operated by Twitter". In no planet is what it outputs user generated content since the content does not originate from the user, at most they requested some content from Twitter and Twitter provided it.
There's still a lot of of unanswered questions in that area regarding generated content. Whether the law deems it CSAM depends on if the image depicts a real child, and even that is ambiguous, like was it wholly generated or augmented. Also, is it "real" if it's a model trained on real images?
Some of these things are going into the ENFORCE act, but it's going to be a muddy mess for a while.
Getting off to images of child abuse (simulated or not) is a deep violation of social mores. This itself does indeed constitute a type of crime, and the victim is taken to be society itself. If it seems unjust, it's because you have a narrow view of the justice system and what its job actually is (hint: it's not about exacting controlled vengeance)
It may shock you to learn that bigamy and sky-burials are also quite illegal.
"“We take action against illegal content on X, including Child Sexual Abuse Material (CSAM), by removing it, permanently suspending accounts, and working with local governments and law enforcement as necessary,” X Safety said. “Anyone using or prompting Grok to make illegal content will suffer the same consequences as if they upload illegal content.”
How about not enabling generating such content, at all?