The argument originally appeared in Searle's 1980 article "Minds, Brains, and Programs." \citet*{searle_1980a}. The argument is summarized as follows (from \citet*{hauser_2001}):
Against "strong AI," Searle (1980a) asks you to imagine yourself a monolingual English speaker "locked in a room, and given a large batch of Chinese writing" plus "a second batch of Chinese script" and "a set of rules" in English "for correlating the second batch with the first batch." The rules "correlate one set of formal symbols with another set of formal symbols"; "formal" (or "syntactic") meaning you "can identify the symbols entirely by their shapes." A third batch of Chinese symbols and more instructions in English enable you "to correlate elements of this third batch with elements of the first two batches" and instruct you, thereby, "to give back certain sorts of Chinese symbols with certain sorts of shapes in response." Those giving you the symbols "call the first batch 'a script' [a data structure with natural language processing applications], "they call the second batch 'a story', and they call the third batch 'questions'; the symbols you give back "they call . . . 'answers to the questions'"; "the set of rules in English . . . they call 'the program'": you yourself know none of this. Nevertheless, you "get so good at following the instructions" that "from the point of view of someone outside the room" your responses are "absolutely indistinguishable from those of Chinese speakers." Just by looking at your answers, nobody can tell you "don't speak a word of Chinese." Producing answers "by manipulating uninterpreted formal symbols," it seems "[a]s far as the Chinese is concerned," you "simply behave like a computer"; specifically, like a computer running Schank and Abelson's (1977) "Script Applier Mechanism" story understanding program (SAM), which Searle's takes for his example. But in imagining himself to be the person in the room, Searle thinks it's "quite obvious . . . I do not understand a word of the Chinese stories. I have inputs and outputs that are indistinguishable from those of the native Chinese speaker, and I can have any formal program you like, but I still understand nothing." "For the same reasons," Searle concludes, "Schank's computer understands nothing of any stories" since "the computer has nothing more than I have in the case where I understand nothing" (1980a, p. 418). Furthermore, since in the thought experiment "nothing . . . depends on the details of Schank's programs," the same "would apply to any [computer] simulation" of any "human mental phenomenon" (1980a, p. 417); that's all it would be, simulation. Contrary to "strong AI", then, no matter how intelligent-seeming a computer behaves and no matter what programming makes it behave that way, since the symbols it processes are meaningless (lack semantics) to it, it's not really intelligent. It's not actually thinking. Its internal states and processes, being purely syntactic, lack semantics (meaning); so, it doesn't really have intentional (i.e., meaningful) mental states.
Why is so much confusion caused by the Chinese Room Argument (CRA)? I think the answer is simple: Searle offers an intuitively convincing argument against 'Strong AI' which on closer examination is seriously flawed. The crucial step in presentations of the CRA is the moment when the reader is asked (or told as the question is usually rhetorial): 'Simply looking up responses like that isn't understanding Chinese is it?' This combined with the suggestive construction of the story's narrative normally elicit a gut response of no.
But why do we consider that 'room' understands Chinese? One obvious reason is that we have been told that the man in the room does not. But this is a crux of the have your cake and eat it part of the argument. Are we defining understanding by external performance or not? If we are then the man does understand Chinese. If we are not (which is what Searle is explicitly saying) then the argument is vacuous. It is vacuous because having implicitly defined understanding as in some way more than performance, and more than purely syntactic manipulation, it is not surprising that the room fails to display understanding.1 Notice also the sleight of hand here whereby understanding is implicitly asserted to be 'what humans do' without ever making the claim explicit.
What do we mean even for a human to understand something? One is that innate question of understanding (the a-ha) and the other is directly related to performance. e.g. ability to explain to solve relevant problems. Given that the whole question is about applying the concept of understanding to non-human entities, in particular machines, using a concept of understanding strongly based on subjective human experience distorts the discussion and misses the point. It also confutes the role of the likelihood of system working with argument against understanding. It is in fact very unlikely that basic syntax system (i.e. match against input and then use a huge lookup table) is likely to work. It wouldn't scale and can't deal with the fact that the problem space is practically infinite.
Consider now the formalization of the argument proposed by Searle \citet*{searle_1990} which provides an approach to the problem shorn of the striking details, which while they enliven the argument serve to beguile the reader and obstruct a clear analysis (again from \citet*{hauser_2001}):
(A1) Programs are formal (syntactic).
(A2) Minds have mental contents (semantics).
(A3) Syntax by itself is neither constitutive of nor sufficient for semantics.to the conclusion:
(C1) Programs are neither constitutive of nor sufficient for minds.
Searle then adds a fourth axiom (p. 29):
(A4) Brains cause minds.
from which we are supposed to "immediately derive, trivially" the conclusion:
(C2) Any other system capable of causing minds would have to have causal powers (at least) equivalent to those of brains.
whence we are supposed to derive the further conclusions:
(C3) Any artifact that produced mental phenomena, any artificial brain, would have to be able to duplicate the specific causal powers of brains, and it could not do that just by running a formal program. (C4) The way that human brains actually produce mental phenomena cannot be solely by virtue of running a computer program.
Having put the argument into this form it is clear that the essential steps/assumptions are A2 and A3. And, that the setup of the Chinese Room is primarily directed towards convincing the reader of A3. It is at this point that we should step back and remember the reason we have the CRA in the first place. It is there as an explicit attack on the Turing Test as a validation of intelligence/understanding.2 The whole intention of the Turing Test, to my mind at least, is to get rid of endless debate about the internals and to focus on something which can be objectively evaluated, namely performance. Yet the whole structure of this debate is about internals. A3 explicitly makes claims a priori about whether what is necessary for semantics (for which read understanding). That is we are discarding an objective approach (necessarily grounded in external observables) for a priori arguments about the internal workings of the entity. Even were we to go along with this approach, it begs several questions:
From \citet*{searle_1980a}:
As regards the first claim, it seems to me quite obvious in the example that I do not understand a word of the Chinese stories. I have inputs and outputs that are indistinguishable from those of the native Chinese speaker, and I can have any formal program you like, but I still understand nothing.
NB: the Turing Test is but the modern formulation of a conception extending back at least to Descartes of a way of distinguishing unthinking automata and intelligent thinking entities (beings - if i want to be provocative). Moreover its approach is Behaviouralist/Functionalist one. However while i do think this is a correct approach to take for a test I wish to make it clear that this does not imply an embrace of these positions. One can think that a test based on external observables is important without thinking that is all there is (which I take to be the cariacature of the Behaviouralist position).↩