Comments on: RIP John Searle

RIP John Searle

deeker — Mon, 29 Sep 2025 11:38:02 -0800

American philosopher John Searle, widely known for his famous "Chinese room" argument produced in 1980, has died aged 93.

Perhaps best known for The Chinese Room Argument or his debate with Derrida, Searle was more generally famed for his work on philosophy of mind and philosophy of language. Over the course of his career, Searle was the recipient of several awards and honors, including the Jean Nicod Prize, the National Humanities Medal, and the Mind & Brain Prize. In 2019, he was stripped of his emeritus status at Berkeley, where he had been a professor and worked from 1959 to 2019, after the University of California deemed that he had violated their sexual harassment policies. This followed prior complaints from students and workers at the university, including accusations that he fired one research assistant who rejected his advances and made inappropriate advances on students. Searle died in a nursing home on September 17, per an email from his secretary of 40 years.

By: GenjiandProust

GenjiandProust — Mon, 29 Sep 2025 11:48:02 -0800

An interesting philosopher, but the worst sort of academic. No ".".

By: mittens

mittens — Mon, 29 Sep 2025 11:51:58 -0800

Jackson Kernion: I blew the whistle on John Searle.

By: Didymus

Didymus — Mon, 29 Sep 2025 11:57:05 -0800

A philosophy prof introduced the class to Searle in his junior Philosophy of Mind course In hindsight I think the prof was quite taken by Searle's arguments at the time, and he fostered this attitude in the class (mid- to late 1990s). So for a period of time I was convinced Searle got it right in a number of ways. Now, that feels like a lifetime ago and "The Social Construction of Reality" feels like a galaxy away.

By: mittens

mittens — Mon, 29 Sep 2025 12:02:14 -0800

(also the chinese room thought experiment is dumb, there, i said it)

By: flabdablet

flabdablet — Mon, 29 Sep 2025 12:19:28 -0800

the chinese room thought experiment is dumb I never found it the slightest bit convincing. It was my introduction to Searle, and resulted in my filing him under Men Who Think They're Much Cleverer Than They Show Any Sign Of Being, and nothing of his I've read since has persuaded me that I got that wrong.

By: GenjiandProust

GenjiandProust — Mon, 29 Sep 2025 12:20:15 -0800

It is, kind of, but it's also a decent argument for why GenAI is not conscious nor a path toward consciousness.

By: ocschwar

ocschwar — Mon, 29 Sep 2025 12:21:51 -0800

If you train an AI on Searle's writings and it produces similar output to what he did, and doesn't sexually proposition a student, is it conscious?

By: GenjiandProust

GenjiandProust — Mon, 29 Sep 2025 12:33:37 -0800

No.

By: Sebmojo

Sebmojo — Mon, 29 Sep 2025 12:34:10 -0800

(also the chinese room thought experiment is dumb, there, i said it) i thought this too, but LLMs are literally chinese rooms. I'm mildly embarassed by how wrong i was.

By: The Bellman

The Bellman — Mon, 29 Sep 2025 12:40:47 -0800

I'm mildly embarassed by how wrong i was. You weren't wrong and you don't have to be embarrassed. An analogy can accurately describe a system and still be dumb. The Chinese Room is that.

By: edselford

edselford — Mon, 29 Sep 2025 12:45:12 -0800

Every 'AI' chatbot company owes his estate royalties for chinese-rooms-as-a-service.

By: Slothrup

Slothrup — Mon, 29 Sep 2025 12:47:44 -0800

The Chinese Room argument is dumb because the thing described as a "human" in this case is just part of a wire between the outside world and the program, which is the real "Chinese speaker". The trick is that the wire has been anthropomorphized while the program has been deanthropomorphized.

By: SPrintF

SPrintF — Mon, 29 Sep 2025 13:43:00 -0800

I first read the Chinese Room argument in Scientific American many years ago. The article stumped me for a while, because I couldn't quite grasp Searle's point. Re-reading the article later, I realize that his argument confused me because it makes no sense. He seemed to be claiming that the Room can't understand Chinese because there is no component within the room that understands Chinese. By that logic, no human being can understand Chinese either, because if you divide the brain into small enough pieces, you won't have anything left that understands anything. But somehow, collectively, it works!

By: kittens for breakfast

kittens for breakfast — Mon, 29 Sep 2025 13:50:07 -0800

LLMs are Chinese Rooms. I'm sorry this guy evidently sucked ass as a human being, but as someone who regularly works with everyday people who seem to believe AI is Data from Star Trek, a demon who lives in the phone, or somehow both, this is too important an analogy to just blow off because we don't like this individual.

By: eraserbones

eraserbones — Mon, 29 Sep 2025 13:52:25 -0800

He seemed to be claiming that the Room can't understand Chinese because there is no component within the room that understands Chinese. That's how I read it too. Basically he's asserting that the reason I'm conscious is because there's a tiny little man inside me who is embodies the consciousness.

By: kittens for breakfast

kittens for breakfast — Mon, 29 Sep 2025 13:53:28 -0800

He seemed to be claiming that the Room can't understand Chinese because there is no component within the room that understands Chinese. There is a component within the room that can produce Chinese. It can break Chinese down into patterns and match the patterns. It cannot understand Chinese. The Chinese writer on the other side of the door is fooled into believing the entity on the other side is communicating with them in a meaningful way. This is not the case.

By: eraserbones

eraserbones — Mon, 29 Sep 2025 14:11:48 -0800

If people think LLMs are the same thing as the Chinese Room then I must not understand the Room at all. I thought the part of the premise of the Chinese Room was that it actually worked?

By: flabdablet

flabdablet — Mon, 29 Sep 2025 14:13:08 -0800

LLMs are one way to implement Chinese Room-like behaviour. A genuine engineered consciousness could be another. The thought experiment deliberately specifies no details about the operation of the room's proposed algorithm, then simply asserts that no such algorithm could be conscious because it's merely an algorithm. This is nothing more than a classic question-beg, festooned with enough red herrings for Searle to convince himself that his reader won't spot what he's done. Well, I did spot it, and it's a stupid argument. Like an awful lot of philosophical thought experiments, it's heavy on thought, too bloody light on experiment, and far too keen to lay down the law on what is or isn't possible "in principle". I've heard more defensibly rigorous work from friends as the joint gets passed around.

By: mittens

mittens — Mon, 29 Sep 2025 14:15:23 -0800

this is too important an analogy to just blow off because we don't like this individual. No, no, the essay was problematic long before our culture got around to caring about the welfare of his victims. (Have we talked about why it's Chinese in particular that's part of the example, and not, like, Linear A or French or something? Like it really leans hard into "why, this language is nought but meaningless squiggles to me!")

By: flabdablet

flabdablet — Mon, 29 Sep 2025 14:16:34 -0800

Chinese was clearly all Greek to Searle.

By: BungaDunga

BungaDunga — Mon, 29 Sep 2025 15:02:24 -0800

By: BungaDunga

BungaDunga — Mon, 29 Sep 2025 15:03:29 -0800

that said yes it is remarkable how much LLMs do prove that you can build a system that very convincingly mimics language but definitely isn't conscious

By: flabdablet

flabdablet — Mon, 29 Sep 2025 15:06:42 -0800

one by one start to replace neurons with the obviously-unconscious algorithm This being 2025, you'd one by one replace them with subscriptions to spiking-as-a-service.

By: Sebmojo

Sebmojo — Mon, 29 Sep 2025 15:36:13 -0800

the chinese room is like the one after the room that has the turing test in it. we passed the first, and hit the second. that said yes it is remarkable how much LLMs do prove that you can build a system that very convincingly mimics language but definitely isn't conscious exactly.

By: fnerg

fnerg — Mon, 29 Sep 2025 15:44:03 -0800

Oh god people. The Chinese room is dumb in the same way Schroedinger's Cat is dumb. Although Schroedinger's Cat was actually supposed to show that quantum superposition is stupid. (It seems superficially stupid, but has turned out to be true) It's an analogy. Analogies break down if you look too closely. Searle may have been a horrible person, but the analogy is still useful. Ultimately, we don't have a good definition of consciousness, so we have no way of measuring how conscious anything is. The Chinese Room is just saying that we can't use language to measure consciousness, because it should be possible to create a non-living machine/algorithm/whatever that can hold a conversation. WHICH WE HAVE. (shakes head, rolls eyes)

By: Ryvar

Ryvar — Mon, 29 Sep 2025 15:50:02 -0800

The Chinese Room is about adherence to syntax without understanding semantic relationships. LLMs are based on proximity of embeddings (words-in-context) within the complex vectorspace of the middle layers. They are nothing but semantic comprehension. The problem is that the Chinese Room is a garbage thought experiment much in the way Searle was a garbage human. There is a difference between semantic comprehension within the language itself - how words and even concepts relate to each other - and language as a tool in its originating context: the world it describes and the coupling between the language-as-model and the material reality it represents. The problem with the Chinese Room is that it elides this distinction. LLMs are as proficient or more than humans at the purely internal semantic comprehension side of language, and utterly, irredeemably hapless at the latter; we've built The Ultimate Poet. The only thing useful about the Chinese Room is that it is yet one more illustration of how fuzzy most human thinking on cognition and language really is at heart. But the need for such illustrations is long past when we live with the consequences of that fuzzy thinking every day. Cognitive Science, Machine Learning, Artificial Intelligence: whatever you want to call the field, it's better off without men like Searle in it. For multiple reasons.

By: Didymus

Didymus — Mon, 29 Sep 2025 15:51:55 -0800

fnerg, it turns out your last two comments have started "Jesus people" and "Oh god people"

By: kittens for breakfast

kittens for breakfast — Mon, 29 Sep 2025 15:53:53 -0800

So, in conclusion, the Chinese Room is a good analogy for LLMs.

By: migurski

migurski — Mon, 29 Sep 2025 16:05:18 -0800

What I took from his classes (Mind ca. 1998 and Language the following year) is that we can't talk about consciousness without intent (wanting things) and belief (about the outside world). I think that's what the Chinese Room was trying to illustrate? Today we have talking machines that don't want anything and don't believe anything, but they can give answer-shaped utterances to prompts so people get confused about whether they have some kind of subjective interiority. The CR is a great analogy for LLMs for the same reasons the critical theorists would have had a field day with ChatGPT. He was an interesting professor. Took and enforced attendance in a 200 person lecture. Stories of his creepiness were starting to come out around that time.

By: mittens

mittens — Mon, 29 Sep 2025 16:08:05 -0800

It's an analogy. Analogies break down if you look too closely. I encourage you to read his The Mystery of Consciousness and ask yourself if he meant this thought experiment to be merely an analogy.

By: SPrintF

SPrintF — Mon, 29 Sep 2025 16:11:33 -0800

difference between semantic comprehension within the language itself... and language as a tool This makes me recall one of the issues I had with the Scientific American article. Searles frequently references the "semantic meaning" of something without ever defining what "semantic meaning" is. This is key because the whole idea of "meaning" and "comprehension" are bound up in the concepts of "understanding" and "consciousness". I also recall a cartoon that illustrated the article. It showed a "computer" looking at the Chinese character for "horse". The computer just sees the same character. A human looking at the character has a "thought balloon" that depicts a cartoon horse. But (1) the character for "horse" is itself a simple cartoon of a horse and (2), the human doesn't actually have a literal horse in his head; he has an abstracted notion of a horse, a cartoon, if you will. Are the computer and the human really different, then? I'll go a step further: if two humans look at the character for "horse," do they really conceptualize the same thing? I see "horse" and think of Secretariat, BoJack Horseman and My Little Pony. You see "horse" and think of carousel horses, sawhorses and a pickup basketball game. Neither of us think of "large-bodied domestic quadruped," yet somehow our definitions overlap enough that we reckon we understand each other when talking about horses. I suggest that the human "large world model" from which "semantic meaning" arises, is unique to every individual. All of our heads contain "cartoons," not the things themselves. Do any of us fully understand "horse"?

By: kittens for breakfast

kittens for breakfast — Mon, 29 Sep 2025 16:16:22 -0800

I mean, I know what a horse is, so.

By: fnerg

fnerg — Mon, 29 Sep 2025 16:19:19 -0800

The problem with the Chinese Room is that it elides this distinction. Cool. That makes perfect sense. Now talk to me about consciousness.

By: kaibutsu

kaibutsu — Mon, 29 Sep 2025 16:23:34 -0800

The real mystery is why one would bother talking about the chinese room when we could be talking about Blindsight.

By: fnerg

fnerg — Mon, 29 Sep 2025 16:28:44 -0800

fnerg, it turns out your last two comments have started "Jesus people" and "Oh god people" I think this is a sign I need to delete my account, because I'm rolling my eyes more and more at the conversation here in general. Not just this thread.

By: sickos haha yes dot jpg

sickos haha yes dot jpg — Mon, 29 Sep 2025 16:39:41 -0800

Have we talked about why it's Chinese in particular that's part of the example, and not, like, Linear A or French or something? Great essay that does ask this question

By: sickos haha yes dot jpg

sickos haha yes dot jpg — Mon, 29 Sep 2025 16:41:22 -0800

I think this is a sign I need to delete my account, because I'm rolling my eyes more and more at the conversation here in general. Not just this thread. good luck out there!

By: dis_integration

dis_integration — Mon, 29 Sep 2025 16:42:46 -0800

> By that logic, no human being can understand Chinese either, because if you divide the brain into small enough pieces, you won't have anything left that understands anything. But somehow, collectively, it works! that's the rub though. that's the whole thing. no part of my brain "understands" language but all the same there is something inside me, which all of us agree (or tend to agree) that we have, which does understand language: our minds. the paradox is what makes this philosophically interesting

By: notoriety public

notoriety public — Mon, 29 Sep 2025 16:46:24 -0800

I mean, I know what a horse is, so. A horse is a horse, of course, of course!

By: Ryvar

Ryvar — Mon, 29 Sep 2025 16:51:54 -0800

SprintF: That's kind of my point. We all understand a horse is an animal, a large mammalian quadruped, often domesticated, frequently used for transport, etc. But I have ridden horses. And smelled horses (christ). I have fed and petted horses. LLMs get the first part perfectly. They don't have coupling between "horse" and the experiences in the second. They also don't have coupling between any of the other terms used to describe and classify horses and real world experiences, meaning their comprehension exists purely within the language. That's the critical distinction, and I think the Chinese Room obfuscates this by attempting to draw a hypothetical fault at semantic comprehension in general rather than diving into what actually constitutes comprehension: the abstract and material duality of our existence. Now talk to me about consciousness. Which part? The one where we're generalized predictive modeling systems with a deep specialization in agentic prediction, or the one where we're all self-authored narratives attempting to continuously paper over the fact that we're basically deterministic processes lightly modulated by random quantum noise? I can do either. Still kind of working through the human mind as an energy conservation mechanism attempting to self-regulate its rate of change in order to maintain coherence, as a survival trait. Basically: why do we instinctively treat new ideas as a threat? Does this represent a sort of meat-based, behavioral analogue to gradient clipping? Is that why we reencode all our long term memories, are we solving for minimum-topological-delta write operations? And what does that imply about the merits of debating this in a John Searle obit thread on Metafilter?

By: kittens for breakfast

kittens for breakfast — Mon, 29 Sep 2025 17:03:55 -0800

The threat is that grown-up people who should know better think that LLMs are people -- or demons -- or superintelligences that will skinamarink us all in a fit of pique one day like in that one Harlan Ellison story. That's the problem; you and I can do some bong rips and be all like, "Sooo what if we're all just LLMs ourselves, man," and it's cute, but what's not cute is the staggering number of people who believe we have created sentient beings.

By: Didymus

Didymus — Mon, 29 Sep 2025 17:04:30 -0800

I think this is a sign I need to delete my account, because I'm rolling my eyes more and more at the conversation here in general. Not just this thread. stick around, who the hell wants to read everyone lined up saying how positively useless the CR analogy is, how wrong Searle was (and is) and generally agreeing on all of it oh and you earned a nasty from sickos and that right there should tell you to stay a while

By: SaltySalticid

SaltySalticid — Mon, 29 Sep 2025 17:14:41 -0800

What's amazing to me is how many experts on philosophy of consciousness and language we have in this thread! When we see posts about influential findings in science (or the people who did the work), for some reason we don't see all that many people here with the years of training in the specific subject necessary to tear down ideas that have already been vetted and deemer valuable by qualified peers in the real world. But apparently there are a lot more Mefites with graduate degrees in philosophy and it's a real experience to watch them at work.

By: mittens

mittens — Mon, 29 Sep 2025 17:17:52 -0800

What's amazing to me is how many experts on philosophy of consciousness and language we have in this thread! If he didn't want us to talk about his ideas, then he probably should have stopped publishing them.

By: sickos haha yes dot jpg

sickos haha yes dot jpg — Mon, 29 Sep 2025 17:30:18 -0800

oh and you earned a nasty from sickos and that right there should tell you to stay a while best to you in your endeavors, dog. but if you want more of my attention, you shall have it!

By: SaltySalticid

SaltySalticid — Mon, 29 Sep 2025 17:33:39 -0800

Oh I don't think Searle would care that people with little relevant training would think his work is dumb and like to say so. That's true for most any influential scholar.

By: GenjiandProust

GenjiandProust — Mon, 29 Sep 2025 17:38:56 -0800

I mean, maybe, but, then, he was an abuser and now he's dead. We can care about his ideas, but we do not have to care about him as a person.

By: star gentle uterus

star gentle uterus — Mon, 29 Sep 2025 17:46:44 -0800

The real mystery is why one would bother talking about the chinese room when we could be talking about Blindsight. kaibutsu Blindsight itself discusses the Chinese Room concept explicitly (as well as implicitly being the whole point of the book).

By: mr_roboto

mr_roboto — Mon, 29 Sep 2025 17:47:34 -0800

By: SaltySalticid

SaltySalticid — Mon, 29 Sep 2025 17:52:38 -0800

Oh yeah, Searle was a total scumbag. It must have been a mountain of evidence to get him stripped of emeritus status at Cal. I don't give a shit about the guy, it's not out of respect for him that I'm annoyed with the facile critiques above. I just thought I might see some interesting and informed discussion of the ideas here, since I've only ever studied it briefly and talked with a few established philosophers about it. But instead the tone was set with "it's dumb" and that tends to drive away people who actually know about stuff.

By: mittens

mittens — Mon, 29 Sep 2025 17:55:05 -0800

people with little relevant training would think his work is dumb and like to say so What an odd thing to say. For all the faults of this thought experiment, Searle was a very clear and entertaining writer who took care to explain his point of view and objections to it, in a way that any reader could understand. Are you saying the Chinese Room would seem like a better thought experiment if we were all PhDs? That's certainly not the impression you get reading other philosophers talk about the flaws in the argument. The whole reason we're talking about this argument today is because of the many, many objections people have raised to it. The life the argument has, is due to those objections and his constant reworking of it to try to forestall those objections. It's a bad argument and it fails, but it's interesting, and people's reactions to it are interesting, the conversation is enlightening, and that's all we can require of an argument about consciousness.

By: kittens for breakfast

kittens for breakfast — Mon, 29 Sep 2025 17:58:03 -0800

I mean...I'm gonna try to let it go after this, but I think it's important to recognize that this dumb, ridiculous, totally unworkable idea now literally exists in the real world and is rapidly changing our society.

By: dis_integration

dis_integration — Mon, 29 Sep 2025 17:59:27 -0800

searle's argument is interesting. it's not a slam dunk and it's derivative of leibniz (it's also possible to find a reading of spinoza that says similar things). it's historically very important to the discussion in philosophy of mind and there's enough said about it that it's silly to just reject it out of hand. the core intuition is that there's something missing from a completely linguistic account of "understanding" because we cannot separate our own experience of understanding from our having a mind. the thought experiment just makes that clear, similar to the argument from Chalmers about "philosophical zombies". the problem of the mind is something you have to take into account in any theory of what it means to be an "intelligent" being. i think you can say: it's just an epiphenomenon that reflects what operations the brain has already taken (with all the consequences for free will entailed), but you can't just ignore it. i've heard stories of how the man was a piece of shit from people with firsthand knowledge, but he was still an incredibly important figure in analytic philosophy for whatever that's worth. he also got his ass handed to him by derrida, that "debate" in limited, inc is one of my favorite books. he just doesn't get that he's being dunked on, its very funny.

By: zippy

zippy — Mon, 29 Sep 2025 18:12:54 -0800

We should all get so much mileage out of a thought experiment so absurd that it can show that people cannot understand language.

By: SaltySalticid

SaltySalticid — Mon, 29 Sep 2025 18:25:27 -0800

Are you saying the Chinese Room would seem like a better thought experiment if we were all PhDs? No, I'm saying I'd rather read criticisms that give the impression the writer has spent more than 5 minutes thinking about CR, or maybe even more than 15 minutes reading about it. and people's reactions to it are interesting, the conversation is enlightening, Yeah, I'm with you, that's what I was hoping to see some of here! But instead we get: (also the chinese room thought experiment is dumb, there, i said it) And I don't think that's interesting or enlightening. And that by itself is fine I suppose, lots of comments here are neither, mine included. But it set the tone, and set off a chain of similarly dismissive comments without any real substance. TLDR, and more sincerely: it would be nice to see less low-effort hot takes and more informed discussion. That's what I used to like about MeFi. I still do, but I used to, too :)

By: Sebmojo

Sebmojo — Mon, 29 Sep 2025 18:33:54 -0800

Here's a good primer.

By: jamjam

jamjam — Mon, 29 Sep 2025 18:45:32 -0800

I don't quite get why people think language is the key to consciousness, and that to understand consciousness, it's both necessary and sufficient to understand language. Consciousness has been around for more than a hundred million years, and language for maybe a hundred thousand or so, and language might well never have arisen, whether you think it evolved or didn't. And human beings seem entirely conscious to me before they learn language and even if they never do. I personally cannot imagine understanding consciousness without language, even though understanding certain things, such as some geometrical and other mathematical truths do not seem to depend on language, but I can't entirely foreclose the possibility.

By: mittens

mittens — Mon, 29 Sep 2025 18:46:21 -0800

I'm saying I'd rather read criticisms that give the impression the writer has spent more than 5 minutes thinking about CR, or maybe even more than 15 minutes reading about it. I appreciate how patiently you made your point, and on reflection, I agree and apologize. So here's someone who spent a LOT of time thinking about Searle's experiment, Daniel Dennett! "For his part, he has one argument, the Chinese Room, and he has been trotting it out, basically unchanged, for fifteen years. It has proven to be an amazingly popular number among the non-experts, in spite of the fact that just about everyone who knows anything about the field dismissed it long ago. It is full of well-concealed fallacies. By Searle's own count, there are over a hundred published attacks on it. He can count them, but I guess he can't read them, for in all those years he has never to my knowledge responded in detail to the dozens of devastating criticisms they contain; he has just presented the basic thought experiment over and over again. I just went back and counted: I am dismayed to discover that no less than seven of those published criticisms are by me. Searle debated me furiously in the pages of the NYRB back in 1982, when Douglas Hofstadter and I first exposed the cute tricks that make the Chinese Room "work." That was the last time Searle addressed any of my specific criticisms until now. Now he trots out the Chinese Room yet one more time and has the audacity to ask "Now why does Dennett not face the actual argument as I have stated it? Why does he not tell us which of the three premises he rejects in the Chinese Room Argument?" Well, because I have already done so, in great detail, in several of the articles he has never deigned to answer. For instance, in "Fast Thinking" (way back in The Intentional Stance, 1987) I explicitly quoted his entire three premise argument and showed exactly why all three of them are false, when given the interpretation they need for the argument to go through!"

By: BungaDunga

BungaDunga — Mon, 29 Sep 2025 19:30:10 -0800

The IEP entry makes a good point that if you buy the Chinese Room, you imagine that there could be entities walking around, talking, and behaving in all ways like they are conscious beings, and yet insist that they are just automatons once you learn what stuff they're made of. Even if they have brains that are organized like our own except each neuron is a nano-sized Turing machine. Now, what stops you from abusing these entities? You have established that they don't think, they have no subjective experience, no matter how much they insist they do. So it's probably fine. This seems like a bad state of affairs to me, so I'm pretty skeptical of a philosophical idea that would lead you in that direction.

By: BungaDunga

BungaDunga — Mon, 29 Sep 2025 19:32:11 -0800

and the clockwork neurons have the same issue where you replace each of your neurons one at a time with a clockwork neuron. when does your consciousness snuff out?

By: Sebmojo

Sebmojo — Mon, 29 Sep 2025 19:33:50 -0800

it's endlessly attackable! but it's also a perfect way of thinking about why LLMs feel like they're thinking, but aren't. I don't have any problem holding those two thoughts in my head.

By: jamjam

jamjam — Mon, 29 Sep 2025 19:48:01 -0800

The way I would put that, Sebmojo, is that we already know consciousness does not require language. If AIs can actually talk, I think that would show that language does not require consciousness either. Then with that out of the way, we might be able to finally get down to the business of understanding consciousness.

By: Faint of Butt

Faint of Butt — Mon, 29 Sep 2025 20:00:13 -0800

I've long said that being the person inside the Chinese room would be my dream job.

By: SaltySalticid

SaltySalticid — Mon, 29 Sep 2025 20:04:03 -0800

Thanks for that bit from Dennett, I didn't realize he was one of the big haters of CR. I guess I'll read some of his seven published criticisms if I want more detail. I think CR is a nice little story to tell to illustrate why the fact that something seems to somewhat convincingly talk like a human isn't any kind of evidence that it understands or has consciousness, and as mentioned above in dis_integration's good comment, others have built up similar and useful arguments along these lines too. I've never been sure why anyone, Searle included, would think that such a state of affairs would entail that such a non-human consciousness or understanding is impossible, as described by the SEP article. It seems to me that there are smaller lessons we can take from CR that are good, and giant leaps to big and wrong conclusions that are not good.

By: kittens for breakfast

kittens for breakfast — Mon, 29 Sep 2025 20:18:58 -0800

I mean, this is kind of getting into the weeds on a basically flawless illustration of something that really exists, right now, that a lot of people dramatically do not comprehend, and could comprehend -- maybe -- if they were introduced to this concept. It's ironic that the concept wound up explaining something that didn't really exist yet, and that Searle never intended to explain, and it's too bad that he was an asshole, but holy shit I cannot tell you how many people need to be introduced to this idea anyway. Like, it's really bad. All this other stuff about "he didn't presuppose the existence of a monkey that knew five human words! What a lummox!" or whatever just seems so utterly irrelevant to me. Not for the first time, a conversation here is making me feel like Blade talking to the Avengers.

By: axiom

axiom — Mon, 29 Sep 2025 20:23:06 -0800

I've long said that being the person inside the Chinese room would be my dream job. you_guys_are_getting_paid.gif

By: zompist

zompist — Mon, 29 Sep 2025 20:26:35 -0800

The Chinese Room is incoherent and was well answered decades ago. Here's Searle himself explaining what the metaphors are:

Now, the rule book is the "computer program." The people who wrote it are "programmers," and I am the "computer." The baskets full of symbols are the "data base," the small bunches that are handed in to me are "questions" and the bunches I then hand out are "answers." [SciAm p. 26]

So the man in the CR represents the CPU, not the program. Querying if the CPU "understands" is supposed to be a test of whether the system "understands." One, it's not, any more than a single neuron in your head "understands." And two, he's narrowly but uninterestingly right about the CPU: the CPU understands nothing, learns nothing; it's the same bit of the computer whether it's running an AI program or Word or Minesweeper. Yet a page later he says "Programs are neither constitutive of nor sufficient for minds". He is trying to pass off an observation about the CPU as a fact about the program. Later on he attempts to hide this blunder by inflating the role of the CPU (the man in the room) even more, by having him memorize the rules, and deflating the program to "a few scraps of paper". The sole test he can conceive of is whether the man in the room can "understand Chinese". But a program that is actually intelligent is too big for the man in the room to memorize. His definition of syntax does not apply to the way computers work... or the way symbols work. E.g. what if one of the rules in the program is this:

If you see 马, write down "horse".

That isn't even how a potential AI works, but it completely destroys Searle's argument. Can he really maintain, if there are enough such rules, that the man memorizing rules "can't understand Chinese"? (Also, yes, he is totally relying on xenophobia to shore up his argument. He talks about "squiggle squiggles", as if the computer is as baffled by Chinese characters as he is.) Searle never bothers to explain how "semantics" can arise in a human mind-- which is a mass of tofu-like cells in a bony prison-- and why it can't in a silicon mind. He just focuses on the CPU understanding or not; it's exactly like someone expecting a water molecule to be wet. (All this is expanded on my site.) Since the Chinese Room proves nothing, it also proves nothing about LLMs. (Searle's argument was devised in the time of algorithmic AI hand-written by humans, in particular story analyzers like Roger Schank's. But it was supposed to apply to any possible computer program.) What LLMs do is show is that the Turing Test sucks. It's way too easy to fool human beings, at least of the CEO level.

By: dustletter

dustletter — Mon, 29 Sep 2025 21:00:26 -0800

I think Searle is wrong - I hold that the universe obeys computable physical laws and therefore there must be some program that understands Chinese, in the sense he meant it. Maybe the one produced by scanning someone's brain or replacing their neurons one by one. But, I think I've been engaging with a weak version of his argument. Suppose you take the program from before, run it through homomorphic encryption, and destroy the key. Before, the symbols were opaque to the operator but held some relationship to the world and the simulated mind's state. Now, their meaning is opaque to all but God. Does that matter to the simulation's experience? A more interesting variant offers that the room is unnecessary; you could memorize the symbol manipulation rules but still wouldn't understand Chinese. I think this implies that I don't understand arithmetic. Maybe I don't?

By: BungaDunga

BungaDunga — Mon, 29 Sep 2025 21:10:21 -0800

Old SMBC comic: "John Searle's last words"

By: cotton dress sock

cotton dress sock — Mon, 29 Sep 2025 21:28:24 -0800

By: Slothrup

Slothrup — Mon, 29 Sep 2025 21:41:07 -0800

> What LLMs do is show is that the Turing Test sucks. Yes, this -- at least for consciousness. An LLM can clearly fool many people into thinking it's human and I believe fairly strongly that it's not conscious in any meaningful sense of the word. My cats could never pass the test and yet I believe equally strongly that they *are*.

By: cotton dress sock

cotton dress sock — Mon, 29 Sep 2025 21:53:14 -0800

Wow I had no idea he was such a pig. I read the CR piece over 20 years ago, found it silly on its face, but it did illustrate the concept that lower level processes can give rise to a kind of experience, defined by an observer's imposition of a narrative around it. It's interesting that we resist this idea so strongly & insist on an essential, unitary, willing force, like a homunculus. We know it's there, emerging from a complex of synapses, bacteria, electricity, etc. It's a good question I think, in line with what I understand of the current scientific understanding. I have nostalgia for the time of my life when I threw myself into these ideas, and that piece was part of it, I guess. What a pig, though. Wild. Where are the decent people who achieve notoriety? I think they're outnumbered.

By: polymodus

polymodus — Mon, 29 Sep 2025 22:57:55 -0800

Searle's Room is wrong because it assumes that Chinese or any language is bounded by grammar. It isn't, the rules are in principle infinite, because all human language is embodied in our societies existing across time in the universe, with contingent exceptions and linguistic changes across this space-time. In order for a Searle Room to in principle completely contain a human language it would have to be infinitely large, therefore it would collapse into a black hole. Conversely a Searle's Room with the grammar rules for C++ does not understand C++ either, and in that case the grammar is finite. Human minds are connected to the rest of the universe (or Earth) so we grow and change with it. So SR does not apply.

By: sickos haha yes dot jpg

sickos haha yes dot jpg — Tue, 30 Sep 2025 02:53:19 -0800

basically flawless lol this seems like a different order of commentary than your other remarks about it

By: Phanx

Phanx — Tue, 30 Sep 2025 03:43:56 -0800

I'm not fond of the Chinese room (incidentally one example of a weird Sinophobic tendency in academic philosophy) but I take Searle's main point to have been that you can't get semantics from syntax. I think that's right - his view of where we do get it from (qualia, basically) is less convincing.

By: flabdablet

flabdablet — Tue, 30 Sep 2025 05:39:06 -0800

So here's someone who spent a LOT of time thinking about Searle's experiment, Daniel Dennett! Pass me the brush to tar ya Make your choice then live your life Come on pal, what are ya

By: HearHere

HearHere — Tue, 30 Sep 2025 05:56:01 -0800

* [nybooks]

By: eraserbones

eraserbones — Tue, 30 Sep 2025 06:40:12 -0800

I was hoping to find a defense of the Room here because I assume that my take is overly shallow and/or missing the point. I don't really see that here, though -- is anyone willing to take a stab? My takes are twofold: 1) The 'systems' argument. Specifically, I think that Searle believes that consciousness is magic and only 'real' if it runs on a platform that he regards as human. I don't think that. If we opened up my mother's skull and found nothing but relays and microfiche, I would still regard my mother as a conscious, real, understanding person. If we found the same thing in Searle's mother, he would conclude that she had just been a convincing simulacrum all along. 2) The 'assuming the conclusion' argument. "Imagine a room that speaks Chinese, but doesn't understand it. By imagining this, you have demonstrated that speaking Chinese isn't the same thing as understanding it." That's just a tautalogy, and tells us nothing about the the question of whether it is /necessary/ to understand a language in order to speak it (which, for certain values of 'speak' I am sure that it is.)

By: Didymus

Didymus — Tue, 30 Sep 2025 07:37:16 -0800

Take what you can from what you read, is what I always say

By: BungaDunga

BungaDunga — Tue, 30 Sep 2025 07:57:31 -0800

Using LLMs to code is a lot like a Chinese room though, neither I nor the LLM nor the combination actually understand what we're producing (unless I go back and look at it), even when it's perfectly cogent code. There's no way there's a third entity being formed by the system that's understanding anything, that seems unreasonable.

By: dis_integration

dis_integration — Tue, 30 Sep 2025 08:18:58 -0800

>In order for a Searle Room to in principle completely contain a human language it would have to be infinitely large, therefore it would collapse into a black hole. Actually it seems like it would have to be about 10-100 gigabytes or so

By: Western Infidels

Western Infidels — Tue, 30 Sep 2025 08:27:02 -0800

I've been trained to manipulate the syntactic and semantic symbols in and surrounding propositions like "The Chinese Room," "The Trolley Problem," "The Turing Test," and "The Simulation Argument," but a moment's examination will show that I don't really understand any of them, and that's how you can tell I'm just a machine and not a living, breathing human being.

By: dis_integration

dis_integration — Tue, 30 Sep 2025 08:34:13 -0800

It's funny. In grad school I took a very strong stand against Searle (because he's very dumb on some things) and his Chinese room. Primarily because I was, I dunno maybe still am, a materialist, and Searle is a sneaky dualist. He's implying that minds cannot be completely explained as the outcome of the causal relations of the bodies that "have" them. But reading these criticisms I have never wanted to defend him more. Like it doesn't matter that it's Chinese, or that he wasn't an expert in computers, or that he talks about syntactic rules when that's inadequate since coherent language requires more than syntax. It's a thought experiment, or (in a phrase I hate), an "intuition pump" intended to highlight a problem with functional-materialist conceptions of intelligence. And the thing is, as others have pointed out, we have real life Chinese rooms right now. You can deny it all you want, but up until the very day that ChatGPT 4 came out, just about everyone would have agreed that if we had a computer that could do what it does it would count as intelligent in some sense, and certainly that it would pass the Turing test. But his insight still stands. The transformer does not understand anything. It simply predicts the next most probable byte given the input bytes and the current set of answer bytes. What it has is a giant, almost infathomably massive table of byte pairs, and a fairly straightforward if computationally intensive process for doing a lookup in that table. But does it understand? Does it have a mind? It seems obvious that doing table lookups is not understanding. That's my intuition. There, I made the Chinese room argument.

By: flabdablet

flabdablet — Tue, 30 Sep 2025 09:13:46 -0800

What it has is a giant, almost infathomably massive table of byte pairs Strictly speaking, what it has is a machine-learning-derived approximation of the function for which a too-large-to-implement table of input/output pairs would be a complete piecewise specification. That specification is not a table of byte pairs, nor even token pairs; those are implementable because they require only N² entries where N is the number of distinct bytes or tokens. Rather, it's a key->value mapping where the keys are derived from an entire context window's worth of tokens and the value is a probability distribution over tokens. The bigger the context window gets, the sparser becomes the population of keys with any actual existence within the combinatorial universe of possible keys, so the more specific and less generalizable the function specification becomes, so the closer any ML-derived approximation of it becomes to just making shit up.

By: SPrintF

SPrintF — Tue, 30 Sep 2025 09:44:20 -0800

Using LLMs to code is a lot like a Chinese room though, neither I nor the LLM nor the combination actually understand what we're producing Don't your prompts provide semantic meaning, though? Unpredictability isn't the same as randomness.

By: BungaDunga

BungaDunga — Tue, 30 Sep 2025 09:56:30 -0800

It probably does, and the whole thing definitely wouldn't work without it. But the LLM doesn't really understand the inputs or outputs, and if I never read the outputs and just vibe, nobody ever understands the output code at all. And then you can shove that output code back into the input and ask for updates. The ratio of "semantic meaning that a human provides" gets pretty small. Yes, obviously, there's a nub of meaning that is coming from the human director, but you can produce a whole small app without anyone being consciously aware of the architecture or even any significant fraction of the code. That's weird!

By: BungaDunga

BungaDunga — Tue, 30 Sep 2025 10:00:33 -0800

humans have a goal when they're directing LLMs, that's the big difference from trapping a human in a room and making them push symbols around. It's sort of the opposite of Searle's room. But it means a human can do things that look like cognitive tasks without knowing what they're doing at all, and just pulling levers on a sort of cognition engine that also doesn't know what it's doing. And yet the outputs look like the outputs of cognition, at least sometimes.

By: jackbishop

jackbishop — Tue, 30 Sep 2025 10:34:50 -0800

Another way to build a Chinese room is to take a Chinese-speaking brain and one by one start to replace neurons with the obviously-unconscious algorithm until you have replaced the whole brain with no change in behavior. When did the consciousness snuff out? Your Chinese Room of Theseus is an excellent thought experiment, but could we work a trolley into it somewhere? (Somewhere nice, please. I'm certain the philosophical role of trolleys as hypothetical manslaughter devices has done untold harm to the widespread adoption of public transit infrastructure.)

By: NotAYakk

NotAYakk — Tue, 30 Sep 2025 10:36:34 -0800

> It is, kind of, but it's also a decent argument for why GenAI is not conscious nor a path toward consciousness. The obvious thing from it is that we, our brains, are Chinese rooms. But apparently this isn't what most people get out of it? So I see it actually an argument why GenAI is a path towards consciousness. Not a huge step, but an impressive one. We are a bunch of (fuzzily separate) subsystems in our brain. We have started to understand how some of them work (at a very vague level). GPT was based on study of organic brains (at a very removed level), and it is slightly plausible some of what it is doing is similar to what our language-processing part of our brain is doing. I mean, not that high a probability, but definitely non-zero. Like, Wernicke and Broca's areas, coming to a dozen or so grams of grey matter, might be doing something similar to GPT; this is roughly 1% of our brain mass. Processing language. We don't think they (alone) are conscious, but we also don't think our vision center is, or our hippocampus, or... --- The thing to me about the Chinese Room as (originally) described is that it is a library the size of a galaxy (maybe larger than the visible universe) with super-luminal delivery of the books the person does lookup on: Lookup tables are *not very compact*. "That universe-sized-library has consciousness within it" (or some kind of complete recording of a consciousness) is a lot less silly than the implied "the library is not that big, how can it be conscious?" --- > But does it understand? Does it have a mind? It seems obvious that doing table lookups is not understanding. That's my intuition. Based on this, I'd recommend reading Permutation City by Greg Egan, which is a kind of materialist-maximalist approach to reality, where reality doesn't even have to be exist to exist. What if patterns are reality? It does this in small steps as a science fiction novel. He's a fun author, writes stuff like "what does a universe without a Minkowski metric look like?", without ever saying the word.

By: deeker

deeker — Tue, 30 Sep 2025 12:22:47 -0800

A trolley is being driven by something that is physically identical to a normal human being but does not have conscious experience. Tied to the tracks ahead is a person whose neurons have been totally replaced with an algorithm with no change in behaviour. Does the Artificial Intelligence intervene to divert the trolley? Now assume the same scenario is a projection on the wall of a cave witnessed by people in the state of nature behind a veil of ignorance...

By: deeker

deeker — Tue, 30 Sep 2025 12:41:41 -0800

I apologise. I wouldn't normally do this, especially because it arises from laughing at my own joke, and I promise not to do it again. A trolley is being driven by something that is physically identical to a normal human being but does not have conscious experience. Tied to the tracks ahead is a person whose neurons have been totally replaced with an algorithm with no change in behaviour. Tied to the other track is baby Hitler. Does the paperclip-maximising Artificial Intelligence intervene to divert the trolley? Now assume the same scenario is a projection on the wall of a cave witnessed by people in the state of nature behind a veil of ignorance...

By: Didymus

Didymus — Tue, 30 Sep 2025 13:08:14 -0800

You left out brain in a vat? Coward!

By: Salvor Hardin

Salvor Hardin — Tue, 30 Sep 2025 13:17:46 -0800

It's amusing when people explain that LLMs can't be "thinking" because they're just following a set of rules. How exactly do you think a brain works? It doesn't follow any rules? I don't know if current LLMs "think" or "understand" in the same sense a brain does (however the heck that works), but I haven't heard a convincing explanation (including the "Chinese Room") of why it's impossible. Why couldn't a system involving a secretary scurrying around following a complex set of algorithms passing notes back and forth give rise of to thought or understanding or consciousness? Doesn't seem any more unlikely than a big electrical meatball.

By: mittens

mittens — Tue, 30 Sep 2025 13:19:16 -0800

Like it doesn't matter [...] that he talks about syntactic rules when that's inadequate since coherent language requires more than syntax. But that's literally his first premise!

By: dis_integration

dis_integration — Tue, 30 Sep 2025 15:20:04 -0800

The Chinese Room doesn't say it's impossible for a computer to think. It says: if you consider this thought experiment, doesn't it seem like something is missing from the computational model of cognition. Some philosophers of mind want to argue that the brain is just a biological computer that processes information according to rules which we don't yet understand but could understand and model in sufficiently large and complicated computer. For example, you could just do what we've done for a fly's brain to model a human brain and then what you'd get is more or less a human mind but in a computer. And then maybe you could talk to it and it would say "jesus christ where's my body kill me it's so horrible oh it's so terrible to be in a computer, i can't feel anything but i feel everything all at once and i hate it so much" or something and we'd know that human beings are just really wet and slimy computers. But the you have to say: well everything is determined completely by rules and nobody makes any choices or has strokes of genius or anything and nobody is responsible for anything (since consequentialism is incoherent, imo). Or maybe there's something else there, more than the operations of physical law, that we can very secularly call a mind, and it intervenes through a magical swerve, a clinamen, to interrupt the causal order with the spontaneity of a spirit. Or something. The chinese room says: it seems like there's something else here! Since if I did all the linear algebra by hand to compute the answer to the question of what a hamburger tastes like working entirely with real numbers alone and then you mapped the output to the lookup table that pairs those numbers with words and the result was an eloquent description of the perfect comibination of salt fat acid and breadiness that is a cheeseburger you might be fooled into thinking this is a person who has had a first-person /experience/ and called to mind eating a burger becaus they *understood* what a hamburger *is*. Except they didn't, it was just me following some rules, doing calculations without knowing what any of them meant, producing a likely sounding output, and there's a difference there, right? There's something there that distinguishes the two? Since isn't that all that a computer would be doing when it modeled all the neurons of a brain to produce what seems like the thoughts of a person? I'm a person with "mental states". A computer is just a computer. How would it get a mind? I'm not saying I agree but it's an unsolved question. Nobody knows where a mind comes from.

By: zompist

zompist — Tue, 30 Sep 2025 16:18:02 -0800

If we go on we're going to repeat everything from comp.ai.philosophy. The chinese room says: it seems like there's something else here! I'm afraid not, because Searle explicitly denies being a dualist. (See his Scientific American article, Jan. 1990.). He claims that he has understanding but thinks he doesn't have to explain how. Dualism does at least provide a place to shove anything about the mind we don't understand. That's more a bug than a feature, since that provides no actual explanation of those things, nor can we imagine how the soul tells the meat what to do. (If your soul orders your muscles to move, the brain has to receive that order somehow, and we should be able to detect that. If you respond that it's too small to detect... well, philosophy of science tells us to be suspicious of theories that cannot be observed or tested.) A common response (one I agree with) is that brains can have mental states because they are closely connected to the world— they have sensorimotor experience. You don't just know the word for horse, you can see and touch and ride horses. Cool. But your brain can't ride a horse; it's a blind chunk of neurons inside a prison of bone, merely connected to the outside world by electrical signals (i.e. nerves). You can see, but it's not via light shining in your skull. Is that a different thing from a robot, also connected to the outside world by wires, sensors, and effectors? Is it a different thing from a computer controlling a CAD system or a 3-D printer? Searle's claims about "syntax" vs. "semantics" are, again, based on reacting against all-verbal programs like Schank's in the 1970s. His arguments get a lot less plausible when we tallk about computers with sensorimotor capabilities. (Actually, well before that. Somewhere in your bank is a computer system with a number representing your bank balance. Is that not real money? It's affected by the money you put in or take out. If a glitch halves the number, suddenly you have less money. It makes little sense to call the banking system "just syntax.")

By: dis_integration

dis_integration — Tue, 30 Sep 2025 16:23:28 -0800

searle is dumb about dualism. he wants his cake and to eat it too. that's ok, he's allowed to be wrong even about his own arguments. well, he's dead now so it's even more permitted

By: judgement day

judgement day — Tue, 30 Sep 2025 19:03:16 -0800

I first came across Searle's Chinese Room concept as a college freshman, and never understood how it was supposed to work in practice. Maybe someone more familiar with it and the criticisms of it can explain... If a Chinese-speaker outside the room passes in a simple question like "what day of the week is it?" how is the instruction set that the English-speaker inside is using supposed to handle that? Short of being provided some additional information (i.e., what day of the week it actually is), I don't see how it could ever consistently provide answers to such questions correctly. And it seems to me that a great many questions the Chinese-speaker outside could ask would fall into this category: any question about current events, many questions about the future, even lots of questions about the past ("how long ago was WWII?" depends on knowing what year it is now). If the instruction book has "if...then" statements in it and/or lookup tables that rely on additional information known by the English-speaker in the room (e.g. "if the question looks like these characters and today is Monday, answer with this character") then how is the room anything different from a particularly extensive and cumbersome-to-use Chinese-English dictionary? The English-speaker might not be able to read Chinese (at least at first), but they must understand something about what is going on in order to follow the instructions to get the correct answer. Maybe they can't (initially) tell whether a question reads "what day of the week is it today?" or "what day of the week is it tomorrow?" and correspondingly don't know which day of the week the response they write refers to, but they can at least immediately figure out that if they need to provide the day of the week in order to look up the response, then the question must have been something to do with days of the week. I bet with enough effort they could even learn to read Chinese, by keeping track of what other information is needed to answer questions besides the questions themselves. But whether or not they can learn to understand Chinese, at very least they are providing an understanding of the state of the world that is not present in the instructions they are using. They can't be reduced to an unthinking "wire." I don't see how the Room works if the person inside is supposed to do no thinking, provide no information, and only look up the characters they receive and write down corresponding answers. Unless I missing something, if they person inside can't provide additional understanding of the current state of the world, than the Room can only be used to answer the types of questions an encyclopedia or calculator could answer (i.e. those whose answers don't depend on any information beyond the question itself) and can't be used to answer the broader class of questions that humans can answer.

By: SPrintF

SPrintF — Tue, 30 Sep 2025 19:07:57 -0800

if they person inside can't provide additional understanding of the current state of the world "I don't know" is an acceptable response. It's interesting that LLMs try to avoid stating that.

By: judgement day

judgement day — Tue, 30 Sep 2025 19:15:24 -0800

"I don't know" is an acceptable response. If I'm the Chinese-speaker and I ask the Room "what day of the week is it?" and it responds "I don't know" -- well, I'm pretty sure I'm not going to conclude that the Room can understand Chinese...

By: flabdablet

flabdablet — Tue, 30 Sep 2025 20:08:00 -0800

we'd know that human beings are just really wet and slimy computers People willing to countenance the idea that a human being is just or merely an instance of some other category are probably worth actively excluding from positions of power, especially if they're all "essentially" about it.

By: kittens for breakfast

kittens for breakfast — Tue, 30 Sep 2025 21:25:51 -0800

Yeah, the people who think people are really just machines always seem to be the very least functional people, don't they? And yet they're also the first to tell you how much better they are than everyone else. It's a conundrum for sure, hell if I can figure it out.

By: flabdablet

flabdablet — Tue, 30 Sep 2025 22:28:43 -0800

hell if I can figure it out For me it comes down to how much detail folks are willing to ignore and/or dismiss as irrelevant. The single most irritating misfeature of the Chinese Room argument, from my perspective, was Searle's eagerness to use dismissive language like "scraps of paper" to characterize the mind-punishingly massive amount of infrastructure that the argument just imagines into existence. Whether or not it's reasonable to include both people and machines in the same category seems to me to hinge on why you're keen to do that. If it comes down to some kind of conviction that causal analysis rooted in physics will at some point be sufficient to account for all of human behaviour in practice, I think you're barking up the wrong tree and shouldn't bother. If it's "in principle" rather than "in practice" then I'm going to need you to start with a careful exposition of exactly what principles you're invoking and exactly why you think that applying them in this instance might illuminate a useful path toward clarity. "In principle" is the self-aggrandizing handwaver's best rhetorical friend. I don't recall ever once in the 63 years I've been on this planet encountering an "in principle" argument that's actually any good; when it comes right down to it, there is only practice. Whether a thing be machine or whether it be biological or whether it be both, causal analysis of it rooted in physics has severe practical limitations and it pays never to lose sight of that. LLMs are a case in point. Even though those are uncontroversially machines, it takes way longer to predict any LLM's output based on any analysis of its inputs and present internal state than it would simply to run it and see what it does. Even getting close to making any such prediction arrive ahead of the observed behaviour requires abandoning strictly physical concepts entirely, instead reasoning solely in terms of much more heavily chunked abstractions. Physics is excellent for telling me why I ought to wear a seatbelt when I'm driving. It's utterly useless for telling me why I just typed out this lot and hit Post. But that says much more about physics than it does about any real-world system that one might consider applying its concepts to. Much the same applies to free will, intentionality, consciousness, determinism, qualia, gods, demons and all the rest of the philosophical gamut. Expecting any of these thinking tools to provide a fundamental and/or definitive account of the real world of which we're all parts strikes me as somewhere between touchingly naive and massively arrogant. Searle never really struck me as touchingly naive.

By: adrienneleigh

adrienneleigh — Tue, 30 Sep 2025 23:28:42 -0800

flabdablet: ""In principle" is the self-aggrandizing handwaver's best rhetorical friend. I don't recall ever once in the 63 years I've been on this planet encountering an "in principle" argument that's actually any good; when it comes right down to it, there is only practice." "In theory there is no difference between theory and practice, while in practice there is." —Benjamin Brewster (often erroneously cited as by Yogi Berra)

By: flabdablet

flabdablet — Wed, 01 Oct 2025 00:58:54 -0800

When we see posts about influential findings in science (or the people who did the work), for some reason we don't see all that many people here with the years of training in the specific subject necessary to tear down ideas that have already been vetted and deemer valuable by qualified peers in the real world. Philosophy benefits from scholasticism much, much less than science does because its domain of applicable subject matter is so much narrower. It looks wider, but that's just parochialism.

By: polymodus

polymodus — Wed, 01 Oct 2025 01:13:53 -0800

There's nothing wrong with saying in principle, it just means first principles reasoning, it declares a logical argument which is often useful. And big-O notation is a good example of "in principle" reasoning about computation, like exponential growth arguments about algorithms or COVID.

By: flabdablet

flabdablet — Wed, 01 Oct 2025 01:58:39 -0800

Big-O actually provides a really good example of the kind of abuses of "in principle" that I so frequently find myself annoyed by. It is often the case than an algorithm with worse big-O behaviour proves to be better in practice, exactly because of details completely extraneous to big-O considerations such as simplicity of implementation or inner-loop speed. Ignoring those details purely for the sake of achieving better big-O behaviour is doing software engineering wrong. The history of philosophy is littered with dubious principles that have all been used at some time or other as "in-principle" justification for policy that works to the severe detriment of ordinary people.

By: rum-soaked space hobo

rum-soaked space hobo — Wed, 01 Oct 2025 03:22:11 -0800

Okay, new goal unlocked: Live a life in such a way that none of the comments on your MeFi obit thread are more than one line.

By: Didymus

Didymus — Wed, 01 Oct 2025 06:32:44 -0800

Here is a Dennett/Searle exchange in case you missed it at the time It is good to have a rival

By: Brian B.

Brian B. — Wed, 01 Oct 2025 07:41:30 -0800

Searle interview on free will.

By: L.P. Hatecraft

L.P. Hatecraft — Wed, 01 Oct 2025 18:41:20 -0800

Here's another exchange between Searle and Dennett, from the early 80s. I could have sworn that Douglas Hofstadter addressed Searle's Chinese Room argument in Godel Escher Bach but seems like I'm misremembering and it was in The Mind's I (co-written with Dennett). See also: China brain. Personally I think this reply from Searle to the "systems reply" was a bit of a cop out:

The idea is that while a person doesn't understand Chinese, somehow the conjunction of that person and bits of paper might understand Chinese. It is not easy for me to imagine how someone who was not in the grip of an ideology would find that idea at all plausible.

The whole thought experiment is contrived as an "intuition pump" to make the systems reply as implausible as possible, but I think if such a system really was possible (it would most likely take millions of years to output one sentence) then it really would understand Chinese. Play language games, get language prizes. Just saying in effect "lol that's ridiculous, you're brainwashed" isn't a convincing counter-argument.

By: mittens

mittens — Thu, 02 Oct 2025 05:08:40 -0800

somehow the conjunction of that person and bits of paper might understand Chinese "Here at our translation office, we have one person who speaks French and another that speaks Chinese. However, there is no way whatsoever that you could say our translation office understands Chinese; I need you to strike that from the marketing materials immediately."

By: flabdablet

flabdablet — Thu, 02 Oct 2025 10:19:05 -0800

Shorter Dennett: Eyeballs are a thing, and here are some clues we've gathered about how they might work for seeing with. Shorter Searle: I cannot see my own eyeballs, therefore whatever Dennett says they're for is irrelevant, wrong, and argued in bad faith, and also why does he keep insisting we're all actually blind?

By: mscibing

mscibing — Fri, 03 Oct 2025 09:51:15 -0800

I feel that Aunt Hillary the intelligent ant colony in Gödel Escher Bach is a bit of a response to Searle's Chinese Room. Aunt Hillary is fanciful sure, but ant colonies very much do have emergent behavior. And really, basic metazoan biology is a problem for the Chinese Room argument; there are people who manifestly do speak Mandarin or Cantonese but you would search in vain for a neuron in their brains that understands these languages.

By: L.P. Hatecraft

L.P. Hatecraft — Fri, 03 Oct 2025 17:03:58 -0800

Gödel Escher Bach was published in 1979 and Searle published the Chinese Room paper in 1980, so it wasn't a direct response but I agree it does address the same issues, the Chinese Room is just a particularly memorable and catchy way of framing them. It's honestly a bit strange to me how these arguments and counter-arguments seem to have been forgotten - there are things people say about LLMs that seem to map directly to the Chinese Room argument, for example when people say "LLMs don't understand anything, they're just doing bunch of linear algebra", you can think of the man in the Chinese Room doing a bunch of large matrix multiplications by hand. Can't people just make the "systems reply" to this? It's not even like they are saying that AI may be possible but LLMs ain't it, "it's just linear algebra" style arguments foreclose any possibility of artificial intelligence because there will always be some underlying mechanism be it linear algebra or even just basic Turing machine operations, we can't just make an irreducible intelligence, we have to make it out of something. Maybe some people aren't familiar with the concept of emergence, never read GEB or similar or heard of Conway's Game of Life or whatever, but it seems there's plenty of people who are - and who might accept the idea of emergent intelligence in the context of an abstract argument or thought experiment set in the far future, but instinctively reject it when there's a suggestion of it being an actual thing people might create in real life. It's like they default to these "common sense" intuition-based arguments, and this is what Searle specialises in (his free will arguments are also like this).

By: flabdablet

flabdablet — Fri, 03 Oct 2025 23:44:41 -0800

It's not even like they are saying that AI may be possible but LLMs ain't it For what it's worth, that's been my own position for a very long time now. Main reason I maintain the opinion that LLMs ain't it is that I see them as such a comically crude Plato's Cave sketch of what it would actually need to be; there is so much more to us, in which group I include large numbers of non-primates, than meets the I.

By: flabdablet

flabdablet — Fri, 03 Oct 2025 23:49:20 -0800

they default to these "common sense" intuition-based arguments, and this is what Searle specialises in (his free will arguments are also like this) Yeah, that's mainly why I have him filed under Men Who Think They're Much Cleverer Than They Actually Are. Wilhoit on conservatism includes a remark apposite to Searle on the Hard Problem:

As the core proposition of conservatism is indefensible if stated baldly, it has always been surrounded by an elaborate backwash of pseudophilosophy, amounting over time to millions of pages. All such is axiomatically dishonest and undeserving of serious scrutiny.

By: mscibing

mscibing — Sat, 04 Oct 2025 16:41:46 -0800

Ah, thank you L. P. Hatecraft for the correction on the order of Aunt Hillary and the Chinese Room. It's not even like they are saying that AI may be possible but LLMs ain't it,... I think that LLMs have been badly oversold, and while I wouldn't say they aren't intelligent at all, that intelligence is very limited, fragmentary, and alien, obscured by their impressive ability to imitate human conversation. My workplace is pushing hard for us to use A.I. and one of the chatbots comes off as a human that would rather make shit up rather than reveal that they don't know something. That's not at all what what's going on, but it's very easy to read human-like thinking into it. And even knowing that it's still infuriating when your plastic pal who's fun to be with lies to your face. I don't think there is going to be an "it", not because A.I. is impossible, but because intelligence is a bit ill-defined and the old sci-fi idea of the machines suddenly "waking up" and thinking the way humans think is unrealistic. It's going to be a slog, with a lot of grift along the way.