by appplication 9 hours ago
This is the root of AI psychosis. There’s a lot of unpack here, and I won’t go too deep because you can’t really have a discussion with affected folks because their fundamental basis is not evidence, it’s belief.
It is weirdly religious in a way, because if you were to present contrary evidence (e.g. experts in a field weighing in about how plausible sounding responses are bunk), you would only be told you don’t believe enough in the long term potential and capabilities.
Don’t get me wrong, I think we all agree capabilities will eventually improve (and farther-future capabilities could reasonably surpass experts), but really is unclear if the current transformer architectures with their probabilistic/hallucinatory outputs will plateau before they surpass current experts abilities in all promised fields.
I was a very early adopter in my circles with AI and I shared it with many people. Strangely, I seem to be the most skeptical about AI in my circles as well, but because I was the gateway for a many folks, they want to come back and share their experiences with me.
And it's so much like listening to someone in a church congregation sharing their experiences with god. Clear and obvious gaps are hand-waved away exactly how you're describing.
>This is the root of AI psychosis. There’s a lot of unpack here, and I won’t go too deep because you can’t really have a discussion with affected folks because their fundamental basis is not evidence, it’s belief. Treating it as if it is an intelligence is the problem.
The problem is that AI psychosis is fundamentally the belief that an LLM is "thinking" at all. Outputs are just believable word vomit which resembles factual information.
That presumes that we have a definition of "thinking" or that we know that anything is "thinking" when in fact neither is true.
The problem is real but I don't think positing a philosophical root is helpful
The claim that we are assigning human-like agency to a machine with none is simple and factual.
What's "thinking"? What's "agency"? What's "human-like agency"?
If "agency" is making decisions and performing corresponding actions in the real world, then LLMs most definitely LOOK LIKE they're making decisions (what's the next token? which tool to use? what's to say, in general? what idea to convey?) and performing actions (tool use). Can we tell whether they are ACTUALLY making decisions? Well, are the people around me "actually" making decisions? Or are they simply pushed around by circumstances and external forces?
Am I actually making decisions? Did I like DECIDE to write this comment? Maybe? I have no clue...
I think you're mildly obfuscating the issues at hand by diving too deeply into philosophical questions.
It's quite simple, the agency that the LLM appears to have is actually your own. Without a prompt an LLM does nothing. It has no thoughts between prompts about you or your problems.
Yes, I'm diving a bit too deeply because I don't really know what "thinking" is and therefore I don't understand how we can so confidently say that LLMs don't think, even though they definitely LOOK like they're thinking. They even have a "Thinking" section in their responses! If I say that a rock doesn't think, it's pretty convincing: does a rock look like it's thinking? No — it doesn't even do anything! But an LLM does look like it's thinking, at least while generating a response. When it's "offline" it's just a bunch of "dead" bytes, sure.
So when it's not active, not responding to a prompt, it's of course not thinking. I'm pretty sure nobody actually questions this. Is your computer "thinking" when it's powered off? Can a piece of metal think? Probably not. So there are no thoughts between prompts, this seems obvious.
Thus, this is a question of "discrete time vs continuous time". LLMs "live" from prompt to prompt. Humans are alive continuously. In some sense, we're prompted by a lot of things all the time. As I'm writing this, I'm seeing stuff, I'm hearing stuff, I can feel various parts of my body, I'm thinking about my problems, my goals, other people's problems and goals, etc. When I'm in a sensory deprivation tank, my brain keeps "entertaining" me by "self-prompting", like a recurrent neural network (I guess it literally is a massive RNN).
So it seems like your definition of "thinking" hinges upon the LLMs being discrete-time and single-threaded (can't think about multiple things in parallel).
IMO a more interesting question is whether an LLM is thinking WHILE IT'S GENERATING A RESPONSE, while it's "alive".
I want to say I really appreciate that you are putting a lot of thought into this, you certainly have interesting concepts here. However I think it seems a bit far off from the discussion I'm trying to have, and I do not have the bandwidth to fully understand and charitably respond to your points.
We don't know what thinking is but pattern matching is definitely a big part of it. That's why people see Jesus on a piece of burnt toast.
You are implying definitions that don't seem to be mainstream; thinking is internally manipulating information to reason, infer, plan, solve problems, and form judgments or beliefs. Also -- "Without a prompt an LLM does nothing. It has no thoughts between prompts about you or your problems." it sounds like you paint this like it's something fundamental? It isn't. Nothing is stopping you from streaming information to an LLM and letting it process this information, this is precisely what people are trying to build.
The machines have no driving force to act in the world. That is fundamental for humans.
Twice in your comment you suggest things that you think that I believe, please do not do this.
“It sounds like you believe” is a question, inviting your clarification. I will continue doing that because it’s perfectly reasonable. Also “machines have no driving force to act in the world” is also a mysterious statement but because you reacted so badly to anyone questioning you I will just leave it at that
That is called a leading question and it is not "perfectly reasonable." Resisting your attempts at bad faith discussion is not "reacting badly." I agree though that we should cease discussion.
Wait, where are we assigning human-like agency in this case? Agency to me means the ability to do something by itself. Here the LLM is not doing anything, it is just responding with information to queries from people, that those people may then act on. (Which you can say about Google searches too, yet we don't ascribe agency to Google.)
The idea that humans have agency is supernatural thinking imo
A free will versus determinism argument doesn't really have a place here. Consider instead that humans factually have 'the illusion of agency.' The LLM does not even that have that. It cannot act on it's own, it has no ongoing drama or intention. It only reacts to prompts.
You're confusing the training method with the internal process. If I had you repeatedly attempt to learn how to make believable completions of partial documents about a given topic, you would eventually learn things about that topic and could use your knowledge to create more believable completions of documents about that topic.
LLMs do not learn. You put it out to pasture and create a new one. "Memory" in a session is essentially a context window party trick.
They already learned. A lot or basically everything evern written and available digital.
And context window work very well. You can 'teach' an llm a new programming lanuage and other things through it.
They learn during training, which is what we're talking about.
>which is what we're talking about.
You are anyway, I don't see anyone up the chain saying that.
They do learn in context, and very sample efficiently. Continual learning is active area of research and we sort of already have something resembling it with persistent context. So yes they do learn.
I consider that to be the illusion of learning. You are not wrong, I think they may actually learn in the future though. But not today.
That’s strange to me, what would you define as learning?
To acquire new knowledge and build your understanding. They don’t understand so they can’t learn
Thank you for saying succinctly what I could not. If your consciousness and knowledge fundamentally does not change from your ongoing experience, then you are not learning. This is how the LLM currently functions.
You’re describing the problem of continual learning. As I said their “consciousness” for lack of a better term and knowledge does already change from ongoing experience in context which is another of saying for only a short window, today. They are ephemeral, sort of, but that’s a temporary limitation.
I think if your definition of consciousness can fit these things then you’re more open minded than I care to be. Consciousness isn’t really guessing the next thing to say- it’s hard to say what it is, obviously, but blindly feeling forwards with each new conversation doesn’t seem like consciousness or learning to me.
We aren't talking about consciousness, we're talking about learning.
> Consciousness isn’t really guessing the next thing to say-
I don't know what consciousness is either and these debates are a dumpster fire when they happen, but it sounds like you're pulling forward this "LLMs are just predicting the next token" (true by construction) implies that they can't learn or reason or be conscious (2/3 are wrong, the last one isn't falsifiable without a useful definition).
“They don’t understand” is a strong statement, maybe true but depends on what you mean by understand. What is your definition of this? I can’t think of a meaningful definition of “understand” that doesn’t apply to LLMs
The LLM itself doesn't, but agents can research, compare, add to their memory, and use that to narrow the results down to a probabilistically higher set of outputs; I have used an LLM for my own MRI results and it was nearly spot-on, verified by a subsequent visit to a specialist. YMMV as they say. But I do believe we are entering the era where LLMs are considering past interactions and long context windows to inform it of personal preferences and history in order to output more accurate results.
believable != true
A very important callout. It's the crux of the whole thing really. Humans are easily susceptible to deception by statements that are structured to be believable.
This is what Stephen Colbert called "truthiness". People want to believe what they feel is true even if it is directly contradicted by evidence.
Sure. But that's not the subject.
Please stop trying to police what the subject is to suit your own arguments.
Often times the words produced do have legitimate factual information though. It's less psychosis and more a confluence of well known human tendencies - salience bias, automation bias, etc.
The big problem is often times they don't as well. That's why we can't rely on them.
Same with humans? Doctors, scientists...if a tool has any error rate above zero its not reliable?
I don’t think they will improve, there is too much incentive to poison the datasets going forward.
A lot of the models up to this point have been benefitted - like Google did - from essentially ‘pre SEO’ internet.
Now the same tools are being used to generate nigh infinite good sounding bullshit, which poisons the dataset in all sorts of hard to detect ways.
To add insult to injury, the human experts are also not as. Naive, and have many incentives to poison their own input in subtle ways too.
I seriously doubt that data set poisoning will be a real limiter in model performance.
For one, if your website/book is poisoned, who is going to trust it for anything at all, much less for training models?
For two, all the major AI labs hire or contract for subject matter experts to create curated data sets, evaluate model performance, etc.
Unless they hire malicious experts, this will provide a growing, high quality data set that should drown out any poisoned pretraining data.
There's a post every other month where some dude who put nonsense information online celebrates because it actually ended up in some frontier models weights.
If it's easy enough that some randos can do it for fun, what do you think happens when there's commercial interest behind it?
Obviously companies are going try nudging AI towards recommending whatever they're selling. It's a logical extension of SEO - and that's a 100 billion USD industry.
Additionally, if I believed myself to be in some sort of spending - err - AI race, I'd try to poison the data sets of my competitors by putting crap out there for others to ingest.
It's not really a problem. We're out of natural tokens anyway. The future is synthetic verifiable traces (already the way we train coding agents).
> synthetic verifiable traces
What does it mean, Is it like when somebody used some coding agent to develop a feature and later input prompts and a resulting PR can be used for training by a presumption that final PR was a correct implementation of a prompt?
Yea it’s rejection sampling, so you have an agent, you take a verifiable problem (people use lots of different verification signals but say unit tests etc) and have the agent attempt it K times. You accept the trajectories (all context, tool use etc, the entire log) that are positively verified and use these as training examples.
The trick is to find the examples that are just in between too difficult and too easy for the existing agent, these have the strongest training signals
There are so many better data sources that AI labs can use here that this argument really holds no water at all.
Peer reviewed journals, textbooks, in-house teams of experts, trusted news publications, etc.
The whole idea of scraping large swaths of the internet for training data has always been pretty dubious due to the variable data quality.
I mean, just look at the early Google models that told people to put glue in their pizza due to a joke in the training set. Garbage in, garbage out.
This is one of the first and most obvious problems all of these labs have run into, and countermeasures are only going to improve.
But they don’t, generally. Which is why it is a great argument, because it’s easy to falsify - and see it is what is actually happening.
Also, those other sources are getting buried in AI slop too.
The question is not whether it has happened or will continue to happen. Of course it will always be a problem to some extent.
Your original claim is that this will be enough of a problem to prevent models from improving in expert level knowledge. I completely disagree with this premise.
If the models fail to improve, it will likely be due to limitations in the transformer architecture rather than poisoned training data.
And even then, I doubt that the transformer is the best architecture we will ever come up with.
Clearly it doesn’t learn or think like a human does, since humans don’t need many gigabytes of text samples to learn to talk, so there is some room for improvement.
https://arstechnica.com/science/2025/01/its-remarkably-easy-...
Great, an article about Llama 2 from early 2025. That doesn’t at all invalidate what I said.
While completely ignoring the fundamental reason. Whoosh.
Not sure what point you’re trying to make.
Do you have examples of such celebrations?
They already are, It has become a real problem in Reddit. Especially with the latest in pseudo-science crap like peptides.
I think you underestimate just how much money is being poured into LLM SEO at the moment. It's real quiet because they don't want to draw attention and countermeasures from the frontier labs, but this is getting huge investment, and they will have a monomaniac focus on juicing product results whereas the attention of the labs necessarily has to be spread out.
Data curation is important and expensive and frontier labs can afford to do it right. Natural data isn't the limitation, we are already literally out of tokens. It doesn't matter how much you poison things it's not going to stop the progress train.
Who's doing llm seo right now? How does that work when you only gets feedback every few months when a new model is out?
I'm pretty sure the Optimization part is just ... not present at all.
This is how we get LLM summaries presenting something mentioned once by some nutjob in a reddit thread as bona fide FACT
Look at G2.com - they found their website is highly references by AIs and they are leaning into it hard.
Pretty easy to display one thing to verified browsers (just latest few user-agents from the 10ish different mainstream browsers on the 3 main OSes) and another to anything else.
Yes AI scrapers can easily spoof user-agent, but they fall out of date as the browser updates.
Bit harder to catch them in tarpits and then serve nonsense to whoever ever triggered the tarpit.
>Yes AI scrapers can easily spoof user-agent, but they fall out of date as the browser updates.
It’s a hell of a lot easier for a company to ensure that its scrapers all report the latest user agent string than it is to get everyone and their mother to update their browsers in a timely fashion.
yeah but unless everyone is checking the version, if it's just a handful of websites checking it, they don't.
and browsers forcibly auto-update
Human doctors use LLMs to diagnose too
OpenEvidence claims
"More than 40% of U.S. physicians use it daily, and it handled around 20 million clinical consultations per month. Over 100 million Americans were treated by a doctor using it in 2025."
This is a very misleading statement; most of those physicians are using LLMs to transcribe notes from visits and/or for billing purposes (e.g., proper billing codes).
The problems isnt LLMs per se, it is the shift to trusting the output of the machine coupled with a decline in verifying that the output is reasonable. It's basically what your teachers warned you about with wikipedia in eight grade except applied to all areas of life, including medicine. Dictation is already high-stakes and LLMs do not automatically reduce that risk.
Here is an example. My provider sent me this note. I'm quoting verbatim here from my MyChart record:
"Your liver enzymes are high, I would like to order acetaminophen containing medication like Tylenol, I would like to order liver ultrasound I placed ultrasound order in the system, make an appointment for radiology, I would like you to get hepatitis panel lab work done, obtain blood work order, please schedule a well visit to get it done"
When I queried it, this is what I got back. It was a dictation error. You could almost hear the panic in the message:
"Sorry for wrong message earlier, I was dictated message- so could not realize that it was written to take Tylenol type of medicines- I DO NOT RECOMMEND ACETAMINOPHEN CONTAINING MEDICINE - LIKE TYLENOL AND ALCOHOL DUE TO ELEVATED LIVER ENZYMES."
Again the problem is not dictation, or LLMs. The problem is humans ignoring their responsibility to check the output of a machine.
> Again the problem is not dictation, or LLMs. The problem is humans ignoring their responsibility to check the output of a machine.
100%. Also, management.
I wish someone would go ahead and coin an AI version of Amdahl's law that states the work speedup from AI is dependent on amount of unverified AI output used.
Iow, if you 1:1 verified everything, there would be no time savings.
Ergo, you get management saying (1) we demand time savings due to AI & (2) we demand you fully check anything you use AI for.
End result? People skip (2) to hit (1).
Then management burns anyone at the stake whenever inevitable mistakes happen.
But that’s trivially false. There is an entire category of work where it is hard to come up with an answer and easy to verify the answer, which means that if you verified everything there would still be a large time savings.
I would question whether that holds in the practical LLM automation space.
Can you think of any real life examples where an LLM is likely to be used?
I think in practice what you're saying is there are problems where there exist efficient deterministic verification methods, and I'm sure that's true.
But that's not the bulk of everyday work LLMs are being asked to do nowadays across industry.
Which is itself a problem as (in my partners evaluations as an optometrist), LLMs used for clinical notes has a bad habit of dropping clinically important information, and the biggest providers don’t give you a copy of the raw transcript or a recording
Which means she ends up spending just as much time as if she’d done it herself as it needs to be verified for accuracy every time…
OpenEvidence is specifically meant to help clinicians make evidence-based decisions in the diagnosis and treatment of patients, not note transcription.
It does both: https://www.openevidence.com/user-guide/visits-overview
Ignoring the fact that this number comes from a company press release, it doesn’t say anything about the number of doctors using it to diagnose, just that they use it.
If a physician uses Google to search for a dosage chart for some drug they rarely prescribe, you wouldn’t say they are using Google to diagnose the patient. You wouldn’t say that either if they used Google to search for the most recent studies on a topic.
To me this is like a good software engineer using AI.
The fact that they use it doesn't make what the result is any worse or less trustworthy - arguably it makes it better.
It only becomes a problem if they offload all of the thinking to AI.
Human expertise is also improving all the time and not limited to just connecting dots. When AI seems to surpass a particular human, it's just because the human lacks broader knowledge and fails to investigate further.
An expert already knows they don't know everything. That was never the point. Critical thinking cannot be delegated to AI any more than it can be delegated to a book. There is nothing new going on here.
> There’s a lot of unpack here, and I won’t go too deep because you can’t really have a discussion with affected folks
Do you think it is any more possible to have a proper discussion with someone who preemptively paints the other person as mentally ill? Or someone who preemptively victimizes themselves?
Cause I don't think these are the hallmarks of an honest discussion. See also the entire past decade of political discourse.
Like, consider this:
> It is weirdly religious in a way, because if you were to present contrary evidence (e.g. experts in a field weighing in about how plausible sounding responses are bunk), you would only be told you don’t believe enough in the long term potential and capabilities.
A trivial counter to this is that you can just be an expert at something (e.g. your own work), use the damn thing yourself (professionally), and evaluate the outcomes for yourself. Then maybe remark "LLM good".
Now you come and remark "LLM bad", and point at random "evidence", either of outright other workloads, or even the one at hand: you're asking someone to reject the reality they've already experienced, entirely based on the assumption that they're "merely religious" or "in psychosis". You tell me if that's any more epistemically rigorous and sensible than their story.
Why is it psychosis and not lower standards?
While I can understand being skeptical of non-experts' claims that such answers are enough, I don't understand why you call it "psychosis" and not simply naivety or lack of expertise.
At the same time, the new so-called "models" haven't been pure transformer-based LLMs, but entire systems with tools (with access to the Internet), data storage, and the options to trigger additional instances for different tasks.
Because some people develop actual psychosis. They go down some rabbit hole with an LLM until the LLM makes them believe they invented new kind of physics that makes them go harassing experts who obviously try to ignore them because its all nonsense.
For me, what others said and literally showed with Claude Code, et al, and what I’ve been experiencing with it, clearly signal way lower standards. But this was true even before LLMs.
Reminds me of that clip of Travis Kalanick, sexual deviant and harasser of women, talking about "discovering new physics."
The Uber guy? Yeah that was a painful watch.
Graciously diagnosed for them by random unqualified people on the internet with an agenda, frequently before even any relevant interaction:
"Oh you like LLMs? You must in AI psychosis!"
Let's not pretend it is anything more than the run of the mill wet fart of a culture war label. It's quite literally the "TDS" of the anti-AI crowd.
That's really not the argument being made here, and you're panning it further by claiming this is staunchly anti-LLM.
The idea here is to signal that you can absolutely use LLMs to help you figure something out. But also, they're wrong a lot. So use your own brain too.