>This is the root of AI psychosis. There’s a lot of unpack here, and I won’t go too deep because you can’t really have a discussion with affected folks because their fundamental basis is not evidence, it’s belief. Treating it as if it is an intelligence is the problem.
The problem is that AI psychosis is fundamentally the belief that an LLM is "thinking" at all. Outputs are just believable word vomit which resembles factual information.
If "agency" is making decisions and performing corresponding actions in the real world, then LLMs most definitely LOOK LIKE they're making decisions (what's the next token? which tool to use? what's to say, in general? what idea to convey?) and performing actions (tool use). Can we tell whether they are ACTUALLY making decisions? Well, are the people around me "actually" making decisions? Or are they simply pushed around by circumstances and external forces?
Am I actually making decisions? Did I like DECIDE to write this comment? Maybe? I have no clue...
I think you're mildly obfuscating the issues at hand by diving too deeply into philosophical questions.
It's quite simple, the agency that the LLM appears to have is actually your own. Without a prompt an LLM does nothing. It has no thoughts between prompts about you or your problems.
Yes, I'm diving a bit too deeply because I don't really know what "thinking" is and therefore I don't understand how we can so confidently say that LLMs don't think, even though they definitely LOOK like they're thinking. They even have a "Thinking" section in their responses! If I say that a rock doesn't think, it's pretty convincing: does a rock look like it's thinking? No — it doesn't even do anything! But an LLM does look like it's thinking, at least while generating a response. When it's "offline" it's just a bunch of "dead" bytes, sure.
So when it's not active, not responding to a prompt, it's of course not thinking. I'm pretty sure nobody actually questions this. Is your computer "thinking" when it's powered off? Can a piece of metal think? Probably not. So there are no thoughts between prompts, this seems obvious.
Thus, this is a question of "discrete time vs continuous time". LLMs "live" from prompt to prompt. Humans are alive continuously. In some sense, we're prompted by a lot of things all the time. As I'm writing this, I'm seeing stuff, I'm hearing stuff, I can feel various parts of my body, I'm thinking about my problems, my goals, other people's problems and goals, etc. When I'm in a sensory deprivation tank, my brain keeps "entertaining" me by "self-prompting", like a recurrent neural network (I guess it literally is a massive RNN).
So it seems like your definition of "thinking" hinges upon the LLMs being discrete-time and single-threaded (can't think about multiple things in parallel).
IMO a more interesting question is whether an LLM is thinking WHILE IT'S GENERATING A RESPONSE, while it's "alive".
I want to say I really appreciate that you are putting a lot of thought into this, you certainly have interesting concepts here. However I think it seems a bit far off from the discussion I'm trying to have, and I do not have the bandwidth to fully understand and charitably respond to your points.
You are implying definitions that don't seem to be mainstream; thinking is internally manipulating information to reason, infer, plan, solve problems, and form judgments or beliefs. Also -- "Without a prompt an LLM does nothing. It has no thoughts between prompts about you or your problems." it sounds like you paint this like it's something fundamental? It isn't. Nothing is stopping you from streaming information to an LLM and letting it process this information, this is precisely what people are trying to build.
“It sounds like you believe” is a question, inviting your clarification. I will continue doing that because it’s perfectly reasonable. Also “machines have no driving force to act in the world” is also a mysterious statement but because you reacted so badly to anyone questioning you I will just leave it at that
That is called a leading question and it is not "perfectly reasonable." Resisting your attempts at bad faith discussion is not "reacting badly." I agree though that we should cease discussion.
A free will versus determinism argument doesn't really have a place here. Consider instead that humans factually have 'the illusion of agency.' The LLM does not even that have that. It cannot act on it's own, it has no ongoing drama or intention. It only reacts to prompts.
Wait, where are we assigning human-like agency in this case? Agency to me means the ability to do something by itself. Here the LLM is not doing anything, it is just responding with information to queries from people, that those people may then act on. (Which you can say about Google searches too, yet we don't ascribe agency to Google.)
You're confusing the training method with the internal process. If I had you repeatedly attempt to learn how to make believable completions of partial documents about a given topic, you would eventually learn things about that topic and could use your knowledge to create more believable completions of documents about that topic.
They do learn in context, and very sample efficiently. Continual learning is active area of research and we sort of already have something resembling it with persistent context. So yes they do learn.
Thank you for saying succinctly what I could not. If your consciousness and knowledge fundamentally does not change from your ongoing experience, then you are not learning. This is how the LLM currently functions.
You’re describing the problem of continual learning. As I said their “consciousness” for lack of a better term and knowledge does already change from ongoing experience in context which is another of saying for only a short window, today. They are ephemeral, sort of, but that’s a temporary limitation.
I think if your definition of consciousness can fit these things then you’re more open minded than I care to be. Consciousness isn’t really guessing the next thing to say- it’s hard to say what it is, obviously, but blindly feeling forwards with each new conversation doesn’t seem like consciousness or learning to me.
We aren't talking about consciousness, we're talking about learning.
> Consciousness isn’t really guessing the next thing to say-
I don't know what consciousness is either and these debates are a dumpster fire when they happen, but it sounds like you're pulling forward this "LLMs are just predicting the next token" (true by construction) implies that they can't learn or reason or be conscious (2/3 are wrong, the last one isn't falsifiable without a useful definition).
Yes, I think we simply disagree. I think you know what LLMs are based on our other thread and if you think something important happens when you get a large enough context vector I don’t think there’s much I can say to change your mind. It seems unlikely to me!
“They don’t understand” is a strong statement, maybe true but depends on what you mean by understand. What is your definition of this? I can’t think of a meaningful definition of “understand” that doesn’t apply to LLMs
The LLM itself doesn't, but agents can research, compare, add to their memory, and use that to narrow the results down to a probabilistically higher set of outputs; I have used an LLM for my own MRI results and it was nearly spot-on, verified by a subsequent visit to a specialist. YMMV as they say. But I do believe we are entering the era where LLMs are considering past interactions and long context windows to inform it of personal preferences and history in order to output more accurate results.
A very important callout. It's the crux of the whole thing really. Humans are easily susceptible to deception by statements that are structured to be believable.
Often times the words produced do have legitimate factual information though. It's less psychosis and more a confluence of well known human tendencies - salience bias, automation bias, etc.
The problem is that AI psychosis is fundamentally the belief that an LLM is "thinking" at all. Outputs are just believable word vomit which resembles factual information.