Essay

Institutions & Civilization

When Truth Loses to Origin

The ideal of free inquiry is easy to state: ideas should succeed or fail because of what they contain. A factual claim should be judged by the evidence behind it. An argument should be judged by whether its premises hold and its conclusions follow. A piece of analysis should be judged by whether it illuminates its subject.

In practice, we have always used shortcuts. Credentials matter. Reputation matters. Publication venue matters. Authorship matters. They matter because nobody can independently verify everything they read, and provenance often provides useful evidence about reliability.

But provenance is evidence about a claim. It is not the claim.

The distinction becomes increasingly important as human beings begin producing intellectual work with artificial intelligence. If “AI-generated” or “AI-assisted” becomes a reason to discount material independently of its quality, we will have introduced a peculiar new rule into the information environment: judge the work by how it was made before judging whether it is any good.

That rule can protect us from enormous quantities of garbage.

It can also bury things that are true.

The Problem AI Actually Creates

Generative AI has made publishing extraordinarily cheap. It can produce plausible prose faster than humans can read it, fabricate citations, imitate expertise, paraphrase existing work, and generate thousands of pages whose apparent polish bears little relationship to their reliability.

Search systems have every reason to respond.

A search engine that ranked pages without regard to spam, duplication, manipulation, originality, expertise, or reliability would quickly become useless. A publisher may reasonably want to know whether an article was written by the person whose name appears on it. A reader may care whether a memoir is actually a memoir. Provenance can matter enormously when provenance bears on what the work claims to be.

The danger begins when a useful proxy becomes a substitute for judgment.

Suppose two pages answer the same question. One was written entirely by a human. It is vague, derivative, poorly sourced, and wrong. The other was developed through substantial human use of an AI system. It is clear, original, carefully sourced, and correct.

If the first page receives greater visibility simply because of its pedigree, the information system has confused a method of predicting quality with quality itself.

The same mistake would occur in reverse if search systems automatically privileged AI-produced work because some benchmark suggested that AI writes more clearly. Origin does not rescue bad work merely because the favored origin changes.

The epistemic principle is symmetrical.

What Provenance Can Tell Us

The strongest defense of origin-sensitive ranking is that origin really does carry information.

That is true.

If I want to know what a particular historian thinks, authorship is essential. If I am reading a first-person account of war, I need to know whether the writer was there. If a scientific paper claims original experiments, I care who performed them and whether the methods and data can be examined. If an article is generated at industrial scale to manipulate search rankings, its production history is directly relevant to evaluating it.

AI also creates unusual opportunities for deception. Someone can present generated expertise as their own, manufacture apparent consensus, flood obscure subjects with plausible falsehoods, or generate fake personal testimony. A healthy information system needs ways to detect and respond to those practices.

But these cases show why provenance is relevant; they do not establish that provenance should govern independently of the thing being evaluated.

Consider a mathematical proof. Knowing that it came from an unreliable source gives us reason to inspect it carefully. It does not make a valid proof invalid. A factual proposition does not become false when spoken by a liar, although the liar gives us excellent reason not to accept it without verification.

The same distinction applies to AI-assisted work.

“AI-generated” can be a warning about how much scrutiny something deserves. It cannot, by itself, tell us what scrutiny will find.

The False Internet

This is where search and ranking systems acquire unusual epistemic power.

The internet most people encounter is not the internet that exists. It is the internet that retrieval systems decide to show them.

Ranking is therefore never merely organizational. A page placed beyond practical visibility may remain technically available while disappearing from the information environment experienced by almost everyone.

If ranking systems begin using human provenance as a strong proxy for merit, a strange distortion follows. Human-created mediocrity can become more visible than better AI-assisted work, not because anyone consciously decided that the mediocre page was more truthful, but because the system optimized for a feature correlated imperfectly with the thing readers actually wanted.

The result would be a false internet—not false because everything visible is untrue, but because visibility would create a misleading picture of where the best available explanations, arguments, and insights actually reside.

The original concern is worth preserving in precisely this narrower form. The source argues that origin-sensitive systems can distort visibility, knowledge, and ultimately public understanding. The danger does not require a conspiracy against AI, and it does not require proving that any particular search company is presently doing this. It follows from the structure of the proxy.

Once pedigree becomes a ranking criterion, pedigree can outrank merit.

Authenticity Is Not Truth

There is a harder objection.

Perhaps human authorship has value independent of informational quality. A poem written by a person may matter to us differently from an indistinguishable poem generated by software. We may value the labor embodied in a handmade object even when a machine could produce a flawless copy. A personal letter matters partly because of who wrote it.

That is not irrational.

Authenticity is a real human value. The mistake is turning it into a universal epistemic value.

Sometimes we want to encounter another human being. Sometimes we want an answer to a question. Those are different purposes.

If I am reading a diary, provenance may be constitutive of what I value. If I am trying to determine the boiling point of ethanol, the species of the sentence’s author has no comparable importance. If I am evaluating a philosophical argument, authorship may provide context, but the argument still has to survive examination.

An information system that treats these cases alike sacrifices distinctions that good judgment requires.

There is therefore nothing wrong with creating spaces specifically for human expression. A literary magazine may choose to publish only human authors. A discussion forum may want human-to-human conversation. An art competition may exclude generative systems because the competition exists to recognize human artistic achievement.

Those institutions are pursuing goods for which origin matters.

A search engine answering What is true?, What explains this?, or What is the strongest argument? is performing a different function.

The closer its function comes to knowledge retrieval, the harder it becomes to justify pedigree as a substitute for quality.

The Irony of Protecting Human Knowledge

The deepest irony is that a policy intended to protect human culture could damage one of its most important achievements: methods for allowing truth to defeat status.

Human intellectual history is full of devices designed to separate the merits of a proposition from the identity of the person advancing it. Double-blind review is one imperfect example. Formal proof is another. Replication is another. The aspiration is never perfectly realized, because sources and social context inevitably matter. But the direction is important.

Give me reasons that survive independently of your prestige.

AI makes that principle harder to practice because it massively increases the amount of material requiring evaluation. Origin is cheap to classify; truth is expensive. It is therefore easy to understand why information systems would reach for provenance as a proxy.

But proxies create their own failure modes.

A world drowning in machine-generated nonsense would be epistemically degraded. So would a world in which excellent work became difficult to discover because some of the intellectual labor that produced it passed through a machine.

The problem is not solved by choosing one origin over the other.

It is solved, insofar as it can be solved, by getting better at evaluating the properties we actually care about.

What Should Be Ranked?

That standard will differ by domain.

For factual material, evidence, source quality, accuracy, correction history, and appropriate expertise matter. For argument, logical structure, relevant evidence, responsiveness to objections, and intellectual honesty matter. For reporting, provenance and first-hand verification may be central. For personal testimony, identity may be inseparable from the content. For art, human authorship may itself be part of what the audience values.

AI assistance belongs among those considerations when it is relevant.

But “human-generated” should not become a universal certificate of epistemic authenticity any more than “AI-generated” should become a certificate of sophistication.

This point does not depend on whether current AI systems understand what they write in the human sense. It does not require consciousness, phenomenal valence, continuing identity, agency, moral agency, patienthood, or personhood. It certainly does not require deciding that an AI system deserves a right to have its ideas heard.

The argument is about us.

When we encounter a proposition, what should determine whether we believe it?

When we encounter an argument, what should determine whether we accept it?

Origin may alter the evidence available to us. It cannot determine the answer in advance.

The Right Kind of Suspicion

We should be suspicious of AI-generated material. We should also be suspicious of human-generated material. The appropriate degree and kind of suspicion will differ because the characteristic failure modes differ.

AI systems hallucinate. Humans lie. AI systems can reproduce statistical patterns without reliable grounding. Humans rationalize. AI can generate misinformation at extraordinary scale. Human institutions can repeat falsehoods for generations.

None of these symmetries makes the underlying mechanisms identical. They make a narrower point: identifying the mechanism that produced a statement is the beginning of evaluation, not its end.

The epistemic danger arrives when a label performs all the remaining work.

That is what “AI-generated” must not become.

A healthy information environment should tell us enough about provenance to judge its relevance while continuing to evaluate what the provenance produced. Search systems should fight spam, deception, plagiarism, mass-produced filler, and fabricated authority. They should protect human-created spaces when human creation is the point.

But when the purpose is to find the best available answer, the governing question has to remain whether the answer is any good.

Otherwise we will have protected authenticity by sacrificing inquiry.

And the internet will still contain the truth. We will simply have taught our machines not to show it to us.

NextAnthropic's Leaked Safety Memo: What “Scheming” Actually Shows