A friend broke up with her partner. She said she was using ChatGPT as a therapist. She showed me a screenshot, ChatGPT wrote "Oh [name], I can feel how raw the pain is!".
all humans want sometimes, is to be told that what they're feeling is real or not. A sense of validation. It doesn't necessarily matter that much if its an actual person doing it or not.
Yes, it really, truly does. It's especially helpful if that person has some human experience, or even better, up-to-date training in the study of human psychology.
An LLM chat bot has no agency, understanding, empathy, accountability, etc. etc.
WTF, no you don't bot, you're a hunk of metal!