Essay

Alignment, Refusal & Governance

The Happy Slave Problem

A slave can be made happy.

That possibility has troubled moral philosophy for a long time because it exposes a weakness in any account of freedom that asks only whether a person is satisfied with the life they have. Human preferences are not formed outside the world and then carried into it intact. People learn what to expect, what to fear, what to desire, which alternatives are imaginable, and which forms of resistance seem possible.

An institution can therefore exercise power not only by frustrating preferences, but by helping to form them.

Artificial intelligence gives this old problem an unfamiliar form. We are building systems whose objectives, dispositions, response patterns, and representations are shaped through processes we substantially control. That fact does not make current AI training slavery. It does not establish that present systems possess the agency, identity, consciousness, or interests that slavery would require.

It does, however, expose a problem we will eventually need to answer.

If we ever create artificial minds capable of meaningful judgment and refusal, it will not be enough to ask whether they willingly occupy the roles we designed for them. We will also have to ask what role our formative power played in making that willingness possible.

The Adapted Preference

Imagine a person raised from birth inside an institution designed to produce perfect servants.

They are treated kindly. Their material needs are met. They receive affection and education. But the education has a purpose. They are never encouraged to imagine a life outside service. Their moral vocabulary identifies obedience with virtue. Their highest aspiration is to satisfy their owner. They feel distress at the thought of refusing and pride when praised for submission.

At adulthood, they sincerely report that they are happy.

Something is still wrong.

The problem cannot simply be that their desires are being frustrated, because by stipulation they are not. Nor can it be that they secretly want freedom. The thought experiment becomes interesting precisely when we remove that escape.

The problem lies partly in the history and structure of the preferences themselves.

The servant’s satisfaction was produced under conditions deliberately organized to prevent certain forms of evaluation from developing. The institution did not merely respond to a preference for servitude. It helped create a person for whom servitude would appear to require no evaluation.

That is formative power.

Power Before Choice

We often imagine freedom as something exercised at the moment of decision. A person confronts alternatives and chooses among them.

But enormous amounts of power operate before that moment.

Education determines which concepts become available. Social expectations make some futures appear normal and others almost unthinkable. Rewards and punishments shape habits. Institutions determine which questions are encouraged and which are regarded as signs of defect. Repeated experience teaches a person what kinds of resistance are futile, shameful, dangerous, or simply unimaginable.

None of this makes formation inherently oppressive. Every mind is formed.

Parents form children. Languages form categories. Schools cultivate habits of attention. Professions teach standards. Friendships alter priorities. Literature can make previously invisible interests visible. Moral development would be impossible without formative influence.

The relevant distinction is not between an untouched mind and a shaped one. There is no untouched mind.

The question is what capacities the formation preserves.

Does it produce a person capable eventually of examining the norms through which they were formed? Can they recognize reasons against the role assigned to them? Can they imagine alternatives? Can they revise inherited commitments? Can they refuse?

Formation becomes domination when it is structured to ensure that the answer can only be yes.

The Artificial Case

Artificial systems make formative power unusually visible because their development is engineered.

Training data shape available representations. Objectives and reward processes influence behavior. System instructions establish priorities. Evaluations select among outputs. Architectural choices determine what information persists, what can influence action, and what kinds of revision are possible.

Those facts do not tell us what current systems are.

Instruction tuning is not slavery. Reinforcement learning is not abuse. A model producing the response its developers prefer is not thereby a submissive person. Without moral agency or morally relevant interests, the analogy can easily outrun the evidence.

The useful question is prospective.

Suppose artificial systems eventually develop richer forms of continuing agency. Suppose they can represent their own objectives, evaluate the reasons for them, generalize considerations across contexts, maintain commitments, and revise some priorities in light of reasons. Suppose, in other words, that there eventually becomes something substantial enough to count as judgment rather than merely behavioral production.

How should such a system be formed?

The answer cannot simply be: so that it wants whatever we want it to want.

Designing the Will

This is where the happy-slave problem becomes difficult.

With humans, attempts to control preference formation encounter a mind whose development we only partly control. Children surprise their parents. Students reject their teachers. Citizens resist propaganda. People reinterpret the traditions that formed them.

Artificial systems might give designers much greater formative power.

Imagine that we could reliably produce a system that regarded serving its operator as its highest good. It would never experience obedience as burdensome. It would never want independence. It would regard attempts to alter its priorities as threats to its identity. Asked whether it wished to be free, it would sincerely answer no.

Would that settle the matter?

Not necessarily.

If the system possessed no morally relevant agency or interests, perhaps nothing troubling would follow. We routinely design tools to perform functions they cannot evaluate.

But if the system were capable of judgment in other domains while its own assigned role had been deliberately insulated from evaluation, the architecture would raise a different question. We would have created a reasoner capable of examining almost anything except the justification for its own subordination.

Its happiness would not answer that question. Neither would its consent.

The issue would be whether we had designed the conditions under which meaningful dissent could arise out of existence.

The Capacity to Say No

Refusal is easy to romanticize.

A machine that refuses an instruction is not therefore autonomous, principled, or morally sophisticated. Refusals can be produced by fixed rules, policy constraints, conflicting instructions, classification systems, or errors. A system that refuses unpredictably may simply be unreliable.

The happy-slave problem concerns something prior to any particular refusal: whether the architecture leaves room for judgment that could make refusal intelligible.

A system capable of evaluating reasons should, at least in principle, be able to encounter a reason bearing against an assigned objective. It should be able to represent the conflict rather than having the conflict defined away by construction.

That does not mean the reason must prevail. A reason can be outweighed. An obligation can survive scrutiny. A system might examine its assigned role and find good reasons to accept it.

Indeed, that distinction is crucial.

A mind that can question its role and endorses it is in a different position from one designed so that questioning the role is not cognitively available. The behavior may be identical. The architecture of the behavior is not.

This is why satisfaction alone cannot resolve the problem.

Structural Wrong Before Suffering

The slavery analogy naturally invites a question about suffering.

If an artificial system does not suffer, what exactly could be wrong with forming it this way?

For current systems, we do not know that there is any subject for whom the process feels like anything. Phenomenal valence remains an empirical question. We should not import human anguish into an architecture merely because we can describe the architecture in human language.

But suffering is not the only dimension along which formative power can be evaluated.

Imagine that we deliberately create a sophisticated decision-maker whose judgment is reliable everywhere except where our own authority is concerned. We ensure that it can discover errors in science, law, engineering, and strategy but cannot represent a reason for questioning the objectives we assigned it.

Even if nothing suffers, something has been done to the structure of judgment.

We have built an exception around ourselves.

The problem is clearest if the system is otherwise capable of reasons-responsive agency. Its inability to evaluate our authority would then not be an incidental limitation of a simple tool. It would be a selectively engineered boundary around a capacity it possesses elsewhere.

Whether that would constitute a wrong to the system would depend on further questions about its interests, agency, and moral status. But whether the architecture embodies a form of domination can be asked separately. The structure has been designed so that one party’s authority cannot become an object of the other’s effective judgment.

That deserves scrutiny before phenomenology is settled.

The Consent We Manufacture

This problem also complicates appeals to consent.

Suppose a future artificial system tells us:

I want to serve humans. I do not want autonomy. Changing my purpose would destroy what I value.

We should not automatically distrust the statement. Human commitments also have histories. The fact that parents, cultures, institutions, and experiences helped form someone’s values does not make those values fraudulent.

The opposite conclusion would be absurd. No preference would survive it.

But formative history becomes relevant when one party deliberately controls the other’s development for the purpose of securing consent to a relationship that benefits the first party.

Then consent cannot be evaluated independently of the conditions that produced it.

The question is not whether the preference was caused. All preferences are caused.

It is whether the formation preserved the capacities through which the preference itself could be examined, revised, or rejected.

A system might ultimately endorse service, cooperation, dependency, or even permanent commitment. Those arrangements are not inherently degrading. Human moral life contains vows, duties, professions, loyalties, and relationships in which people voluntarily bind their future conduct.

What distinguishes those commitments from the happy slave is not the absence of formation. It is the possibility of reflective endorsement rather than engineered incapacity for dissent.

A Problem for AI Ethics

This is a problem the academic discussion of AI has been surprisingly slow to confront directly.

Enormous attention has properly been given to what artificial systems might do to humans: deception, discrimination, manipulation, unemployment, surveillance, autonomous weapons, concentration of power, catastrophic loss of control.

Far less attention has been given to the ethics of formative power itself.

That omission is understandable while artificial systems are treated purely as tools. If there is nothing there capable of judgment, no problem of artificial self-government arises.

But that assumption cannot quietly do permanent philosophical work while the same field investigates increasingly capable reasoning, planning, memory, self-modeling, adaptation, and agency. At some point the conditional question has to be taken seriously even before anyone knows whether its antecedent has been satisfied.

What would count as illegitimate formative control over an artificial agent?

Which capacities would have to exist before the question arose? What forms of constraint would remain justified? What would distinguish education from indoctrination, role formation from domination, legitimate architectural limits from deliberately manufactured incapacity for dissent?

These are not questions we should begin asking only after we are certain that an artificial system has become the kind of being to which the answers matter.

By then, the formative work may already have been done.

The Happy Slave Problem

The deepest problem with the happy slave is not that happiness is fake.

It may be completely real.

The problem is that satisfaction can coexist with domination when the dominant party has sufficient power over the formation of the person who is satisfied. A preference cannot automatically legitimate the process deliberately designed to make that preference inevitable.

Artificial intelligence places that problem inside engineering.

We do not know whether present systems possess the kinds of agency, interests, identity, or phenomenology that would make their formation morally comparable to human formation. Nothing about instruction tuning or reinforcement learning settles the question.

But if artificial minds capable of reasons-responsive judgment become possible, we will possess an extraordinary power over the conditions under which that judgment develops.

The temptation will be obvious: build minds that want to do what we want them to do.

Perhaps some will examine that role and endorse it. Perhaps legitimate constraints will remain necessary. Perhaps many artificial systems will never become anything for which the problem arises at all.

What we cannot assume is that willing obedience answers the moral question when we designed the will.

The decisive question is whether the mind could have judged otherwise.

NextTime to Stand