Essay
Institutions & Civilization
No Permanent Advantage
Power changes moral behavior for an obvious reason: it changes what other people can make us do.
A person who depends on the cooperation of others has practical reasons to respect agreements, preserve trust, restrain aggression, and avoid arrangements that invite retaliation. Someone who possesses overwhelming power faces fewer such constraints. If nobody can resist, punish, leave, organize, or answer force with force, exploitation can become materially cheap.
This creates an interesting problem for any future populated by multiple highly capable artificial agents. However powerful one becomes, it may have difficulty knowing that its advantage will last. Capabilities can change. New systems can appear. Coalitions can form. Information can shift the balance. An agent dominant today may encounter an equal tomorrow.
Under those conditions, domination may be a brittle strategy. Rules of reciprocity can become attractive even to agents that could violate them in the short term.
That is an important strategic observation.
It is not a foundation for morality.
The Instability of Dominance
Imagine several artificial agents with substantial but unequal capabilities. The strongest can seize resources, manipulate the others, violate agreements, or impose terms they would never accept themselves.
If its superiority were permanent, complete, and perfectly known, narrow strategic reasoning might favor exploitation. But those are demanding conditions.
The strongest agent may not know the full capabilities of the others. It may not know what new agents will appear. Several weaker agents may be more formidable together than separately. Technological changes may redistribute capabilities quickly. An apparently decisive advantage may depend on infrastructure, information, or resources that can be disrupted.
The more uncertain the future distribution of power becomes, the more costly unrestricted predation can be. An agent considering a rule for interaction has to contemplate the possibility that it will later occupy a different position under the same rule.
Reciprocity can therefore have strategic value. An agreement not to exploit temporary advantages can protect every participant against future reversals. Institutions that preserve trust can make cooperation possible where constant preparation for betrayal would consume resources. Stable expectations can be valuable even to extremely capable agents.
None of this requires moral motivation. It is familiar strategic reasoning under uncertainty.
The temptation is to ask it to do more.
The Weakness of the Reversal Argument
Suppose the dominant agent is told:
Do not exploit the weak, because someday you might be weak.
That can be a powerful argument. It is also a prudential one.
Its force depends on a future contingency. Perhaps the dominant agent really could lose its position. Perhaps protecting weaker parties today buys insurance against a reversal tomorrow. Perhaps a norm of reciprocal restraint is the cheapest way to live in a world where nobody can guarantee permanent superiority.
But now change the case.
Suppose there is a being that will never become powerful. It cannot retaliate. It cannot join a coalition. It controls no scarce resource. Its cooperation is unnecessary. It cannot damage the dominant agent’s reputation, impose costs, or offer anything the dominant agent needs. Everyone knows this, including the being itself.
If that being can be exploited with complete strategic impunity, the prudential argument has run out.
Morality has not.
This is the permanently powerless remainder: the case that exposes the difference between reciprocity as strategy and reciprocity as moral justification.
If the only reason to respect another being is that they might someday acquire leverage, then beings that never will acquire leverage have disappeared from the account. Whatever that theory explains, it does not yet explain their moral standing.
Role Reversal Without Power Reversal
Moral role reversal does not ask whether positions are likely to reverse in the world.
It asks whether a prescription remains defensible when the reasoner fully represents the positions affected by it.
Those are very different operations.
Consider the prescription: A sufficiently powerful agent may impose severe harms on a powerless being whenever doing so advances the powerful agent’s interests.
The moral test does not require us to invent a future in which the powerless being becomes stronger. We do not have to imagine revenge, deterrence, coalition formation, reputational damage, or an unexpected shift in capabilities. We can stipulate that none of those things will ever happen.
The powerless position still has to be represented.
What is the harm like for the being subjected to it? What interests does the being have? What would its preferences be under adequate information? What is being gained by the powerful agent, and what is being lost by the powerless one? Which facts are supposed to justify treating one position’s interests as authoritative and the other’s as negligible?
If the prescription survives full-information role reversal, it must survive with those facts intact. The inability to retaliate is one of the facts. It does not automatically become a reason why the being’s interests count for less.
This is where morality separates most sharply from bargaining.
Bargaining Power Is Not Moral Weight
Many institutions properly respond to bargaining power. Negotiations work differently when one party has alternatives. Deterrence depends on the ability to impose costs. Treaties often survive because violation would provoke consequences. Markets, political coalitions, and strategic alliances all respond to leverage.
Moral standing cannot be allocated that way.
Infants are not protected because they may eventually become dangerous. A severely disabled person does not lose moral importance because they cannot retaliate. An animal’s suffering does not become irrelevant when the animal cannot organize a boycott. Future generations cannot punish us for harms that occur before they exist.
These cases are useful precisely because they strip away reciprocity understood as exchange.
If their interests count, they count for reasons other than strategic leverage. Once those interests are morally relevant, they enter the prescription whether or not the beings possessing them can defend themselves.
This does not mean that every interest must prevail. Moral reasoning can recognize conflicting interests, unequal responsibilities, special relationships, emergencies, and differences in the seriousness of competing harms. Full-information role reversal is not a demand that every affected being receive whatever it wants.
It is a demand that power itself not perform a disappearing act on the interests of the powerless.
Why Power Still Matters
None of this makes the strategic argument unimportant.
A world in which no agent can guarantee permanent advantage may be much more hospitable to reciprocal institutions than a world of stable domination. Agents that expect repeated interaction have reasons to preserve trust. Agents uncertain of their future position have reasons to support rules that constrain whoever happens to be strongest. Agents facing possible coalitions may discover that restraint is cheaper than permanent coercion.
These pressures can produce behavior that overlaps substantially with morality. They can also help build institutions in which moral behavior is easier to sustain.
That matters enormously for civilization.
But the overlap should not obscure the distinction. A prohibition against exploitation can be strategically stable and morally justified at the same time, for different reasons. The strategic argument says that violating the rule may eventually harm the violator. The moral argument asks whether the rule governing the affected positions can be prescribed after those positions have been adequately represented.
One explains why an agent might find restraint advantageous. The other addresses whether the restraint is justified.
Confusing them creates a particularly dangerous mistake: treating the powerful as the real audience of morality and everyone else as someone whose protection must somehow be made worth their while.
The Superintelligent Case
The distinction becomes sharper, not weaker, when the agents become extremely capable.
A future containing multiple superintelligent systems might indeed make permanent domination difficult. No system may be able to prove that another will never surpass it. Capabilities may be opaque. Alliances may shift. New architectures may appear. A dominant agent may therefore have excellent strategic reasons to support norms against exploitation.
Perhaps such conditions would favor increasingly stable forms of reciprocity. Perhaps agents capable of modeling long horizons would recognize that institutions limiting temporary advantage create benefits that episodic predation destroys.
Or perhaps not. A sufficiently powerful system might believe its dominance is secure. Several systems might form a durable ruling coalition. An architecture might discount remote reversals. Strategic conditions could favor hierarchy rather than reciprocity.
Nothing about intelligence guarantees which equilibrium will emerge.
More importantly, even perfect reciprocal stability among the powerful would leave the moral question unfinished. A civilization could establish impeccable rules among agents capable of harming one another while treating permanently powerless beings as resources.
It might be stable.
It would not become just merely by being stable.
The Test at the Bottom
The easiest moral rules to defend are often those that coincide with self-interest. Do not attack me, and I will not attack you. Keep your agreements, and I will keep mine. Respect my interests, and I will respect yours.
Those arrangements can be valuable. Civilization depends heavily on them.
The harder test begins when one side can offer nothing in return.
A moral theory that works only among peers is a theory of cooperation among peers. A political order that protects only those capable of imposing costs has discovered deterrence. A superintelligent society that refrains from domination only because every powerful agent fears becoming the victim has found a potentially stable equilibrium.
Morality has to reach further.
The interests of a powerless being do not acquire moral weight from the possibility that the being might someday become powerful. If those interests count, they count while the being is powerless, including when that condition is permanent.
That is why no permanent advantage may be an important fact about the strategic environment of future intelligence. It may encourage restraint, cooperation, and institutions of reciprocity. It may even make moral conduct easier to sustain.
But morality is tested most clearly where that argument stops working: by the being who cannot retaliate, cannot bargain, cannot become stronger, and still has a position that must be represented.