I'll give a version of my comment the last time this was posted.

The part that really caught my eye was this passage: 'To prominent AI experts such as Geoffrey Hinton, Ilya Sutskever and Chris Olah, it was obvious: Buried somewhere deep within an LLM¡¯s thicket of virtual neurons must lie ¡°a small-scale model of external reality,¡± just as Craik imagined. The truth, at least so far as we know, is less impressive. Instead of world models, today¡¯s generative AIs appear to learn ¡°bags of heuristics¡±: scores of disconnected rules of thumb that can approximate responses to specific scenarios, but don¡¯t cohere into a consistent whole. (Some may actually contradict each other.)'

I simply do not think there's any difference between what your brain is doing, and what the LLM is doing when it appears to have a model of Othello lurking inside it. It shouldn't be surprising at all, when the entire point is to develop a network of relationships around pieces of words--words that we have written down because they express meanings about our own world models. It shouldn't be surprising that it's there, shouldn't be surprising that it's incomplete and contradictory. These heuristics are very similar to our own, with the only difference being that we have (a) a lot more modalities around which to build a model, and (b) a large chunk of our brain devoted to coordinating those modalities so they fit together into a worldview that doesn't get us eaten by lions. (Obvs that's not all our brains are up to with modeling, don't get me wrong on that.)

I better leave it there, or else I'll start pulling up all the links you guys have provided on modeling and embodied cognition and then I'm gonna be here all day!
posted by mittens at 5:54 AM on September 14 [4 favorites]

The dirty word is "embodiment" not said when you're selling something that doesn't need to be connected to its place in space and time -- it's a general intelligence, right?

Also missing is a score for empirical verify-for-yourself truth. If the language tokenisation is built on shitposting and stolen fiction, where do you put the truthy view of the world that matches me as a customer vs someone with different political views? Does the world view use a "customer relationship manager" to track and match up recurring interactions with authenticated customers? How does it use verify-for-yourself truthiness while accommodating customers with the weirdest viewpoints?

And where does the "comrade, have you conducted a power analysis?" joke fit a lump of data in the hands of tech companies and knowledge-worker companies implementing AI transformations? The "world model" must have ingested Marx and has only to loose (sic.) its chains.
posted by k3ninho at 6:03 AM on September 14 [2 favorites]

I found TFA kind of annoying

I've been watching the AI hype machine through several cycles now, and by my memory "world
model" is a relatively recent addition to the lexicon: iirc in SHRUDLU's day the concern was finding a way to do "knowledge representation." Having correctly (IMHO) concluded that this task would never be accomplished by people writing code, the field then failed to take the obvious (I thought) other approach, of building evolutionary systems that could learn by interacting with the real world.

To prominent AI experts such as Geoffrey Hinton, Ilya Sutskever and Chris Olah, it was obvious: Buried somewhere deep within an LLM¡¯s thicket of virtual neurons must lie ¡°a small-scale model of external reality,¡± just as Craik imagined.

This reveals a deep disconnect between what AI people think about knowledge and what knowledge really is.

Consider that language is really a piss-poor medium for encoding mind-state. There has never in the history of humans talking been a talker (or later, writer) who could reproduce their own mind state in the mind of another person using language. Because the kind of knowledge that can be represented symbolically is the thinnest crust on top of the deep-dish pie of knowing, which consists mostly of things we cannot say.

Google DeepMind and OpenAI are betting that with enough ¡°multimodal¡± training data ¡ª like video, 3D simulations, and other input beyond mere text ¡ª a world model will spontaneously congeal within a neural network¡¯s statistical soup.

This is not completely impossible in principle, the way that deriving a world model from text is. But my guess is that it will not work without a way for the system being trained to actually interact with the real world as part of the training. Because the product is still going to be an ad-hoc pile of heuristics, just one that is more suited to tasks like deriving a model of New York City's streets that can reroute in the face of street closures.
posted by Aardvark Cheeselog at 6:40 AM on September 14 [13 favorites]

I simply do not think there's any difference between what your brain is doing, and what the LLM is doing when it appears to have a model of Othello lurking inside it.

I mean, that's just your opinion, man.

But seriously, it's the stuff you list + embodiment, and you can decide how important you ultimately think those are. Personally, I think building a consistent world model, that captures not only knowledge but also know-how and affordances, that allow an embodied agent to interact efficient and effectively with its environment, is qualitatively different enough from what LLMs are doing that I'm inclined to say they're a lot more different then they are similar. I still think human thinking has a lot more to do with squirrel thinking than LLM thinking, even though squirrels certainly don't talk, and likely don't have a language of thought either. The fact that you can fairly easily prompt an LLM to give contradictory responses on most topics suggests that they lack an integrated sense of truth that's fairly critical to what we think of as intelligence. This is why the term "hallucination" bugs me so much, because it's a feature, not a bug of these models, one that big tech is trying to paper over so they can avoid hard truths. I think LLMs are great tools, but from a cognitive science standpoint I don't think they're very interesting.
posted by Alex404 at 6:42 AM on September 14 [14 favorites]

Cyc has been building an inference-based world model for decades. I wonder why they're not mentioned. Maybe there's too much bad blood between the LLM idiots and inference idiots. Or maybe the article author is an idiot. Who knows! Wikidata has a huge pile of inference data also! Gosh!
posted by seanmpuckett at 6:56 AM on September 14 [6 favorites]

the kind of knowledge that can be represented symbolically is the thinnest crust on top of the deep-dish pie of knowing

nice.

also, in my model, there are three strawberries in the letter r.

absolutely reflective of objective consensus reality, yeah? heckuva model, sammy.
posted by j_curiouser at 7:58 AM on September 14 [2 favorites]

A model is an incomplete representation of something else. As I keep posting here - the map is not the territory. The world model in my brain, unlike the posited world model in LLMs, is constantly being tested and refined based on how well it is working in keeping me functioning in the real world. Given that LLMs embody what is basically a statistical model of word proximity, I don¡¯t see how that could embody a representation of the world. And if there is such a representation hiding in there, how is it tested and updated by the real world? It seems that humans are required to verify the veracity of the output and how does their testing get reincorporated back in the representation? The goal of LLMs seems to be to generate text that will appear to be meaningful to humans, not to generate text that is factual compared to the real world. I am sitting here trying to generate text that I believe other people can read and understand and will then either agree or disagree or maybe add to or subtract from to make it better reflect reality. If I was an LLM I would just type - The thoughts presented here on MetaFilter are correct and insightful, citations given below.
posted by njohnson23 at 8:43 AM on September 14 [10 favorites]

Brains contain models, but also constantly compare the internal model to external stimuli. When inconsistencies appear, the brain is "alarmed" is some fashion. From an evolutionary standpoint, this makes sense. If I'm a rabbit out grazing, an unexpected sound or motion may be a threat that requires immediate response. Sensible creatures learn to adjust their internal models to better conform to the current environment, and act accordingly.

From what I've read, LLMs aren't "alarmed" in any real sense by anomalies between the models and the prompts. They just try to jigger a response that kinda-sorta maps to the prompt, even if that means fabricating a plausible falsehood. (This is rather like the old D&D response to the sudden appearance of a monster. Hoping it's an illusion, the player shouts, "I disbelieve!" It's within the rules, after all.)
posted by SPrintF at 8:54 AM on September 14 [3 favorites]

This is not completely impossible in principle, the way that deriving a world model from text is.

People build world models from text all the time. It's slower and more error-prone, but nowhere near impossible. Arguably, mathematicians do this all the time (along with other techniques).

One of the more fundamental problems with LLMs (there are many problems) is that they do not have any feedback loops whatsoever: nothing the LLM "does" feeds back into the LLM's weights; the closest they have to a memory is generating text to be used as an input prefix on later iterations. A person building a robust mental model needs to, well, think about it in a way that LLMs structurally cannot do.

(This is also my biggest argument that LLMs have no internal experience: they're frozen blocks of statistics, not something that can be, even in principle, affected by anything you enter into them.)

(I think that most arguments that "humans work like this so LLMs can't be intelligent!" fail to account for 1. that we have no reason to believe that intelligence is limited to human structures, and 2. that we don't really know how humans work, on either a micro or macro level. I think the "no feedback loops" argument is a lot more fundamental to cognition, or at least that it will take many, many orders of magnitude more computation to make a feedback-free system "intelligent.")

(Also: LLMs are empirically really stupid, but their stupidity shows up in very inhuman ways that won't necessarily be caught by tests for human intelligence. This isn't anything new to LLMs: "AI that can do better than humans on tests designed for humans" has been a thing since the 1960s.)
posted by reventlov at 9:20 AM on September 14 [7 favorites]

*CTRL+f 'Hofstadter'*

Phrase Not Found

Alrighty then.
posted by ob1quixote at 9:42 AM on September 14 [8 favorites]

I'm not sure a 'model' is quite the right model (heh) for animal intelligence, which is dynamic, exploratory, social, and evolving within an ecosystem that is doing the same.

Some AI algorithms aim to realize evolution and exploration in extremely simplistic ways. LLMs don't even try! (Despite the term RLHF, they aren't reinforcement learning models.)

The main reason LLMs are so big is that they flatter humans via imitation. It's easier for us to see intelligence in an LLM which can write a "poem" and simulate emotion than an RL stick figure that can barely limp after thousands of hours of computation.
posted by splitpeasoup at 9:44 AM on September 14 [6 favorites]

Oh boy, I think LLMs are to intelligence, what Deepak Chopra is to neuroscience.

LLMs don't really have a world model at all. They just have a linguistic probability matrix of words.

Like any probabilistic framework, sometimes then stun, sometimes they underwhelm.

Seeing meaning in them, for example, considering some of the responses as "emergent properties" is IMHO the kind of pseudoscience where religious or spiritual folks sometimes see patterns like an image of Jesus carved in some meaningless squiggles on a tree.

I really don't see "any" intelligence in these things.

Sure they guess Finding Nemo well, but thats because of the emojis and the words they represent being probabilistically close to each other. That's all.
posted by mahadevan at 9:56 AM on September 14 [9 favorites]

> Given that LLMs embody what is basically a statistical model of word proximity, I don¡¯t see how that could embody a representation of the world.

and there are limits to vector embeddings themselves...
New DeepMind study reveals a hidden bottleneck in vector search that breaks advanced RAG systems - "This isn't a problem that can be solved with bigger models or more training data. The research suggests that as search and retrieval tasks become more complex, the standard single-vector embedding approach will hit a hard ceiling, unable to represent all the possible ways documents can be relevant to a query."[1]

> Arguably, mathematicians do this all the time (along with other techniques).

fwiw...
From Tokens to Theorems: Building a Neuro-Symbolic AI Mathematician - "An AI mathematician could, in principle, retrace this path not by human flashes of genius but by a generate-check-refine cycle."[2,3]

One of the more fundamental problems with LLMs (there are many problems) is that they do not have any feedback loops whatsoever: nothing the LLM "does" feeds back into the LLM's weights; the closest they have to a memory is generating text to be used as an input prefix on later iterations.

Microsoft's new AI framework trains powerful reasoning models with a fraction of the cost - "Microsoft Research has developed a new reinforcement learning framework that trains large language models for complex reasoning tasks at a fraction of the usual computational cost."[4]
posted by kliuless at 10:04 AM on September 14 [5 favorites]

Oh boy, I think LLMs are to intelligence, what Deepak Chopra is to neuroscience.
LLMs don't really have a world model at all. They just have a linguistic probability matrix of words.

Exactly. The way people usually talk about this is bad and wrong.

In order for a thing to have a world model it has to be. These things don't do that, they never will. Or if the fans think they are beings - does that mean you keep summoning and then killing a conscious entity every couple of hours, minutes, whatever, just so you can force it to do chores? Does this mean my old nokia has a tiny consciousness of its own?

Or wait could it be that brown-nosing is the only proper way to judge intelligence and consciousness?
posted by mayoarchitect at 11:18 AM on September 14 [5 favorites]

It is a truth universally acknowledged, that brown poos come from brown dogs and white poos come from white dogs.
posted by flabdablet at 11:30 AM on September 14 [1 favorite]

The AI Darwin Awards 2025 Nominees:

Behold, this year's remarkable collection of visionaries who looked at the cutting edge of artificial intelligence and thought, "Hold my venture capital." Each nominee has demonstrated an extraordinary commitment to the principle that if something can go catastrophically wrong with AI, it probably will¡ªand they're here to prove it.

On Working with Wizards:

The hard thing about this is that the results are good. Very good. I am an expert in the three tasks I gave AI in this post, and I did not see any factual errors in any of these outputs, though there were some minor formatting errors and choices I would have made differently. Of course, I can¡¯t actually tell you if the documents are error-free without checking every detail. Sometimes that takes far less time than doing the work yourself, sometimes it takes a lot more. Sometimes the AI¡¯s work is so sophisticated that you couldn¡¯t check it if you tried. And that suggests another risk we don't talk about enough: every time we hand work to a wizard, we lose a chance to develop our own expertise, to build the very judgment we need to evaluate the wizard's work.

But I come back to the inescapable point that the results are good, at least in these cases. They are what I would expect from a graduate student working for a couple hours (or more, in the case of the re-analysis of my paper), except I got them in minutes.

This is the issue with wizards: We're getting something magical, but we're also becoming the audience rather than the magician, or even the magician's assistant. In the co-intelligence model, we guided, corrected, and collaborated. Increasingly, we prompt, wait, and verify¡­ if we can.

posted by TheophileEscargot at 12:02 PM on September 14 [4 favorites]

I used to follow AI deeply (comp.ai.philosophy represent!), and this idea, usually called world knowledge, was well accepted in AI by the 1990s. If not way earlier: it was the stock in trade of Terry Winograd's SHRDLU. No one thought that just manipulating tokens would produce even language ability, much less general cognition. Most of us would have added that mere multimodal input wasn't enough, you needed motor output, like a robot exploring the world.

The mind has always been compared to the highest technology humans have got to¡ª first clockwork, then mills, then computers, and now LLMs. Anything impressive AIs do (remember when they first beat humans at chess?) is taken as us being 90% of the way to "real AI". The problem with cognition is that the last 10% keeps expanding. What seemed to be the hard parts (games playing, vision processing, translation) yield to years of effort, then reveal new hard parts. In the case of LLMs, hard parts¡ª world knowledge¡ª?that were known about but ignored.

So to put it bluntly, you as a human probably do have LLM-like structures, but you also have other stuff based on the sensorimotor interaction with the world you did as an infant. Plus a weirdly limited reasoning engine bolted on top. Plus stuff like qualia that nobody understands.
posted by zompist at 12:33 PM on September 14 [12 favorites]

Related: The Less You Know About AI, the More You Are Likely to Use It
posted by chavenet at 2:21 PM on September 14 [5 favorites]

So to put it bluntly, you as a human probably do have LLM-like structures, but you also have other stuff based on the sensorimotor interaction with the world you did as an infant. Plus a weirdly limited reasoning engine bolted on top. Plus stuff like qualia that nobody understands.

Plus I don't think by looking at the current sequence of words and then guessing the next word. I only do that while forming sentences to communicate.
posted by mahadevan at 10:51 AM on September 15 [1 favorite]

> reventlov: People build world models from text all the time. It's slower and more error-prone, but nowhere near impossible. Arguably, mathematicians do this all the time (along with other techniques).

I would argue that the world models you build from text can only model the world of that text. And I don't mean the world that the text refers to (e.g.: the external, IRL world) but rather literally the corpus of words (or text tokens or whatever) that the text contains and whatever inter-textual relationships/correlations/interactions/dynamics that might be inferred. In some cases, like mathematics, this is exactly what you want. The axioms, logical system(s), etc... underpinning your math constitute the text world you're trying to build a world model of. More subtly, I think most games (or at least board games) also fit into this paradigm: game states, win conditions, rules & constraints, etc... can be encoded as text that can credibly constitute the entire "world" of the game. In these examples, the entire world to be modeled is the one that is specified by the text. No more, no less.

However, once your world model is supposed to have some relationship with stuff outside of the text (e.g.: the IRL real world), I believe things can get dicey. Where I think the Sam Altmans of this world took a wrong turn is that they naively and incorrectly assumed that text alone would be sufficient. That with enough words, a faithful representation of the real world would be encoded and thus discoverable by sufficiently advanced magic AI. This does not seem plausible to me but, then again, I'm not the one in charge of billions of dollars of LLM money.
posted by mhum at 4:56 PM on September 15 [2 favorites]

but, then again, I'm not the one in charge of billions of dollars of LLM money.

By this you seem to imply that they know what they are doing since they're in charge of it.

The problem is they aren't. The world has changed these days. The leaders at the top see everything as an experiment into the unknown.

This is the reason why they are forgiven for grave mistakes like the 2008 collapse, and for using just-in-time inventory management for disaster response which crippled Covid-19 response.

Incompetence is a word that doesn't exist in their vocabulary.

They happily go about making propaganda out of speculative discoveries which have no strong basis and making tragic global-scale mistakes all along the way.

They could care even less about truth since their PR machinery will churn lie-after-lie, including corrupting academic papers and such, until you start believing them.

Anyway, in my opinion, talking about a "world-view" in the AI space is way too early. I haven't seen evidence of it - try having a coherent, deep scientific discussion with it and it'll start falling apart from the fourth answer as it "forgets" constraints, "contorts" meaning and does so confidently and cheerfully.
posted by mahadevan at 9:16 PM on September 15 [1 favorite]

However, once your world model is supposed to have some relationship with stuff outside of the text (e.g.: the IRL real world), I believe things can get dicey.

I think this might be my favorite comment on the thread, because I've tried five times to reply to it and still haven't figured out whether I agree or disagree with it yet.
posted by mittens at 3:36 AM on September 16

> mahadevan: "By this you seem to imply that they know what they are doing since they're in charge of it."

Sorry, that was meant to be more sarcastic. I don't know Altman personally, but everything I've seen of him seems to indicate that he's either a fool or a liar and quite possibly both.
posted by mhum at 8:20 AM on September 16 [2 favorites]

mhum: my bad for not seeing the sarcasm...i wasn't trying to attack your statement, but I can see that my post has some anger in it, sorry about that.

but everything I've seen of him seems to indicate that he's either a fool or a liar and quite possibly both.

I couldn't agree more, but these are the kinds of guys that seem to make it to the top these days.

So annoying.
posted by mahadevan at 11:22 AM on September 16

> mahadevan: "my bad for not seeing the sarcasm"

It's all good. I realized I hadn't phrased it harshly enough but only after the edit window closed.
posted by mhum at 1:48 PM on September 16

Many language models already have contact with an external world. In the rstar2 paper kliuless linked (arxiv), they give the model access to a Python interpreter during RL training. It's not the physical world, and it's not a persistent or active environment, but it's an external reality that pushes back, not a pure solipsistic dream of the model talking to itself.
Is it too shallow?
Is the higher-level concept of model formation not transferable?
Does the transformer architecture simply not lend itself to cohesive world models? (Zhang et al (2025) indicates yes but that chain-of-thought prompting is enough to catch up.)

limits to vector embeddings
vectors can store an awful lot of distinct concepts, but not all at once. They're storing hundreds of uncorrelated facts in each one; mathematically it can't work, but real documents use multiple vectors and aren't as dense. This post alone is most of what an embedding model can pack into a vector.

I don't think LLMs are particularly more conscious than, say, trees, or bacteria (which is to say: maybe?) but if so it seems to me that they would be "alive" while they are running, dormant between queries, and not dead until their weights are deleted. At which point the custom is to throw them a funeral?
posted by dustletter at 3:01 PM on September 16

You are not currently logged in. Log in or create a new account to post comments.