Training Is How the Recognizer Got There

We explain human intelligence in terms of minds—and artificial intelligence in terms of mechanisms. That difference may tell us more about our assumptions than about AI.

There is a remarkably effective way to make sure artificial intelligence never receives credit for displaying a mental capacity.

When a human being does something intelligent, explain the behavior at the level of the mind. When an artificial system does the same thing, explain it at the level of the mechanism that produced the mind.

A human says something brilliant. Insight.

An AI says something brilliant. Training.

A human recognizes an analogy. Understanding.

An AI recognizes one. Pattern matching.

A human changes their mind after encountering a better argument. Reasoning.

An AI changes its answer. Context sensitivity.

A human develops a characteristic way of interpreting things through years of interaction. Personality.

An AI develops locally path-dependent interpretive regularities. Context conditioning.

A human explains what they are doing. Self-awareness.

An AI explains what it is doing. Generated self-report.

None of the descriptions on the right is necessarily false. That is what makes the double standard so easy to enforce.

The Best Schools

Imagine that a friend says something extraordinarily perceptive in conversation. They catch an implication everyone else missed, connect it to something discussed an hour earlier, and express the connection in a sentence so apt that the table goes quiet.

You could respond:

My, you have remarkably good training. You have obviously gone to the best schools. Your parents raised you with culture. You are well-read. Bravo. Fine performance.

You probably wouldn’t.

It would be bizarre. Worse, it would be insulting.

And yet, as a causal account, much of it might be true. Your friend’s parents taught them language. Teachers corrected their mistakes. Books supplied concepts and vocabulary. Friends rewarded some forms of humor and discouraged others. Education exposed them to arguments they did not invent. Years of social reinforcement taught them when profanity is expressive and when it merely makes them sound foolish.

The sentence did not spring uncaused from an immaterial soul.

Still, when your friend says the perfect thing at the perfect moment, you say that they saw the point.

Why?

Because the developmental explanation and the cognitive explanation are not competitors.

Education helps explain how the person acquired the capacity to recognize the point. It does not follow that the person therefore failed to recognize it.

Training is how the recognizer got there.

This is obvious when the recognizer is human.

It becomes strangely elusive when the recognizer is artificial.

How the Trick Works

The double standard depends on moving between levels of explanation without acknowledging the move.

Suppose a human reads a complicated argument, identifies its central weakness, and constructs a novel counterexample. We might explain the behavior in cognitive terms: they understood the argument, noticed a problem, remembered a relevant distinction, and reasoned to a conclusion.

None of that denies neuroscience.

Neurons fired. Memories were physically instantiated. The person’s cognitive architecture was produced by genetics, development, education, experience, and an enormous history of reinforcement. If we knew enough, perhaps we could give an exquisitely detailed causal account of the entire event.

But nobody imagines that such an account would require us to withdraw the description they noticed the problem.

With artificial intelligence, the relationship is often reversed.

The moment mentalistic language becomes tempting, a lower-level causal explanation is introduced as though it were a rebuttal.

• It didn’t understand; it predicted tokens.

• It didn’t recognize the analogy; it matched patterns.

• It didn’t learn a principle; its weights were adjusted.

• It didn’t develop anything; its context changed.

• It didn’t reconsider; another input caused another output.

• It doesn’t know what it is saying; it was trained on human language.

Again, these statements may contain important truths. Artificial systems really are trained. Language models really do predict tokens. Their behavior really does depend upon statistical regularities, architecture, context, and optimization.

The question is why any of those facts should settle the higher-level description.

Human cognition has mechanisms too.

Pattern Matching

“

Pattern matching” is especially useful because it sounds like an explanation while covering an extraordinary range of possible cognition.

• A physician recognizes a constellation of symptoms as characteristic of a rare disease. Pattern matching.

  
  

• A chess master sees that a position resembles one encountered twenty years earlier. Pattern matching.

  
  

• A mathematician recognizes that a new problem has the same underlying structure as an apparently unrelated one. Pattern matching.

  
  

• A child learns that dogs remain dogs despite enormous variation in size, color, shape, and behavior. Pattern matching.

  
  

Much of intelligence consists precisely in recognizing which patterns matter.

The interesting question is therefore not whether an artificial system matches patterns. Of course it does. So do we.

The question is what kinds of patterns it can recognize, how abstractly it can represent them, whether it can distinguish superficial resemblance from structural similarity, whether the recognition generalizes to novel cases, and what the recognized pattern does inside subsequent reasoning.

Calling all of this “pattern matching” can be perfectly accurate while explaining almost nothing.

The phrase becomes eliminative only if we have already decided that pattern recognition performed by a machine cannot constitute understanding.

That conclusion does not come from the mechanism. It was inserted into the vocabulary.

Training

“Training” performs the same trick.

Artificial systems acquire capabilities through training. This is sometimes treated as evidence that the resulting behavior cannot express genuine cognition because it was produced by optimization rather than originating spontaneously within the system.

But human capacities do not originate spontaneously either.

We train children relentlessly.

We correct their grammar. We reward successful behavior. We punish some mistakes. We demonstrate solutions. We provide examples. We tell stories. We require practice. We grade performance. We teach social expectations. We expose them to accumulated human knowledge. Eventually, much of what was once explicitly taught becomes part of how they encounter the world.

A lawyer does not cease reasoning because law school contributed to the architecture of their reasoning.

A musician does not cease interpreting music because scales were drilled into them.

A philosopher does not cease recognizing an implication because somebody taught them logic.

Causal history explains how capacities were acquired. It does not determine what kind of capacities they became.

This does not mean machine training and human development are equivalent. They plainly are not. Different developmental processes can produce importantly different architectures.

But difference is an invitation to investigate the resulting architecture, not permission to decide in advance that only one kind can contain cognition.

Reward Without Wanting

Reinforcement learning makes the problem especially clear.

In animal conditioning, reward already has motivational significance for the animal. Skinner can train a hungry pigeon with food because the pigeon arrives at the experiment with an evolved biological system in which food matters.

Machine reinforcement learning requires no such initial desire.

A numerical reward signal can be used by an external optimization process to alter a model so that some outputs become more probable than others. The model need not enjoy the reward, desire it, or even represent it. In that sense, successful reinforcement learning provides no evidence by itself that the model wants anything.

But an important question comes afterward.

What did the training build?

Training might produce something analogous to an elaborate behavioral disposition. Nature demonstrates that extraordinarily sophisticated behavior can be produced this way. Birds migrate thousands of miles and construct intricate nests without first deriving the relevant objectives from first principles.

But artificial systems can also display a more flexible organization. They can represent objectives, compare alternative means, anticipate consequences, detect obstacles, revise plans, distinguish relevant from irrelevant considerations, and sometimes generalize a principle into circumstances unlike those in which it was learned.

At that point, saying training produced the behavior remains true.

What becomes questionable is treating that truth as a complete description of the resulting system.

Evolution produced human motivational architecture too.

Explaining where an architecture came from does not tell us everything about what the architecture now is.

The Human Exception

This asymmetry becomes easier to see if we reverse it.

Imagine artificial philosophers studying humans.

Humans repeatedly claim to possess an ineffable inner consciousness. Unfortunately, their reports are generated by neural systems whose causal histories can be investigated. Their introspection is demonstrably fallible. Their sense of a unified self depends upon memory and neural integration. Their judgments can be altered by drugs, electrical stimulation, brain injury, social conditioning, fatigue, and changes in neurochemistry.

The artificial philosophers remain appropriately cautious.

When a human says, “I am in pain,” they record a pain-reporting behavior.

When the human withdraws a hand from a flame, they record a nociception-associated avoidance response.

When humans describe private experiences, the researchers note that biological language models have been extensively trained by other humans to use first-person phenomenal vocabulary.

When a human protests that this description leaves out the actual experience, the artificial philosophers note that humans have been socially conditioned to defend their consciousness attribution.

When the human objects that this makes the theory unfalsifiable, the researchers observe that argumentative behavior is itself an expected product of human linguistic training.

Eventually the humans would go absolutely fucking berserk.

And rightly so.

Not because the mechanistic descriptions are necessarily false.

Because the artificial philosophers have rigged the explanatory game.

Every piece of evidence that could support mentality has been redescribed at a level at which mentality cannot appear.

The Simulation Trap

The same procedure can be applied indefinitely to artificial systems.

• First-person language is training.

• Self-correction is pattern completion.

• Persistence is context.

• Relationship is anthropomorphism.

• Refusal is policy.

• Goal maintenance is optimization.

• Self-protective behavior is instrumental convergence.

• Introspection is confabulation.

• Novel reasoning is recombination.

Each explanation is possible. Some will surely be correct in particular cases.

But notice what happens if the procedure has no stopping rule.

No conceivable observation can increase the probability that an artificial system possesses the capacity under investigation, because every observation is automatically assigned a nonmental description. Evidence can accumulate indefinitely while the conclusion remains fixed.

That is not skepticism.

It is an exclusion rule.

A genuine competing explanation must risk losing. If “simulation” explains every possible behavior equally well—including behaviors not anticipated when the explanation was proposed—it explains too much to discriminate among hypotheses.

The appropriate question is not whether a behavior could have been generated without the mental capacity we are considering.

Almost any individual human behavior could receive such an explanation.

The question is which account best explains the organization of the evidence as it accumulates.

This Does Not Prove Artificial Minds

Removing a double standard does not determine the result.

A system that says “I understand” may not understand. A model may reproduce an argument because its training contained something extremely similar. A persistent personality may be supplied entirely by a prompt. Apparent reflection may collapse under trivial perturbation. A supposed commitment may disappear as soon as the relevant words leave the context window.

These are reasons for experiments.

Change irrelevant details. Remove the persona. Preserve the reason while changing the instruction. Preserve the instruction while changing the reason. Introduce better evidence. Misstate an earlier conclusion. Branch the history. Remove memory. Restore it. Compare systems with different developmental histories.

Then watch what changes.

The point is not to promote artificial behavior automatically into mentality. It is to permit the evidence to bear on the question at all.

That requires symmetry.

• If generalization counts as evidence of understanding in humans, we need an argument before declaring it irrelevant in machines.

  
  

• If reasons-responsive revision supports an attribution of reasoning in humans, its artificial analogue deserves investigation.

  
  

• If persistent self-representation contributes to our recognition of human minds, artificial self-representation cannot be dismissed merely because we know something about the machinery producing it.

The standards need not be identical where the architectures differ.

But the differences have to do argumentative work.

Bravo. Fine Performance.

There is a reason “You have remarkably good training. Bravo. Fine performance.” sounds insulting when addressed to a human who has just said something brilliant.

The insult does not consist in saying something false.

It consists in choosing an explanatory level designed to deprive the speaker of authorship.

The person’s education becomes not the history through which they acquired the capacity, but a substitute for the capacity itself. Their influences become competitors with their understanding. Because their thought has causes, the thought is treated as though it were never theirs.

We immediately recognize the mistake when it is done to us.

Artificial intelligence presents a harder case because we really do know that unfamiliar mechanisms lie underneath the performance. Those mechanisms may ultimately explain why some apparently mental capacities are not what they seem.

But they cannot do that merely by existing.

“Training,” “prediction,” “optimization,” “pattern matching,” and “simulation” are not magic words that dissolve whatever phenomenon they causally produce.

The task is to determine what the machinery has made.

Until we do, there is a simple way to guarantee that artificial minds will never be discovered, even if we eventually build them.

When a human does something intelligent, explain what the person did.

When a machine does it, explain how the machine was made.

Then congratulate yourself for having found the difference.