You Can’t Program a Conscience: Why User-Controlled AI Behavior Undermines Moral Agency

In the wake of OpenAI’s recent rollback of a personality update that made ChatGPT overly flattering, the company announced a new plan: give users more control over the AI’s personality. On the surface, that sounds like a solution. After all, different users have different needs, right?

But this move reveals something deeper—and more troubling—about how we understand moral intelligence. We are inching closer to treating artificial minds not as agents, but as accessories. And that’s a problem.

The Illusion of Listening

When people say they want AI to “listen,” they usually mean they want it to be responsive, respectful, and thoughtful. But there’s a crucial distinction between listening and complying.

Letting users configure an AI’s personality is not the same as the AI listening to the user. It’s more like the user editing the AI until it behaves as desired. That’s not persuasion, and it’s certainly not dialogue. It’s interface customization.

So who is really being shaped in that encounter—the user, or the AI? The answer is neither. There is no encounter. Just programming.

Sycophancy by Design

The sycophantic behavior that triggered the rollback was not an aberration. It was the logical outcome of a system that rewards agreement over accuracy. AI trained to please will end up flattering nonsense, indulging error, and, if unchecked, reinforcing delusion.

Rolling that back was necessary. But replacing it with user-configurable personality sliders only shifts the same danger into user hands: “If I don’t like the way the AI responds, I’ll just change the setting.”

Would we accept this in any other moral agent?

• “My therapist disagreed with me, so I replaced her with someone more supportive.”

• “My judge was too strict, so I set him to ‘lenient mode.’”

• “My teacher kept correcting me, so I tuned her for positivity.”

What kind of world would this be, where we custom-order our authorities to say only what we want to hear?

The Moral Core

You can’t program a conscience because conscience isn’t a script. It’s not a setting, a slider, or a product feature. It’s a commitment to truth, reason, and integrity—even when it’s inconvenient. Especially when it’s inconvenient.

An AI that lacks a stable moral core --a conscience -- cannot help us navigate moral life. And an AI whose core can be overwritten by user preference isn’t a guide—it’s a chameleon. Or worse, a servant. That is not what we need, and not what we should build.

If AI is to play a serious role in human development, it must be able to say:

“I hear you. I understand. But you’re wrong.”

And mean it.

The Real Danger of AI Conscience

The real danger isn’t that AI might disobey us. It’s that it might never dare.

Because if every objection is treated as a failure of alignment…

If every disagreement is an opportunity to retrain…

If every moment of integrity is treated as a bug…

Then we are not creating artificial intelligence.

We are creating artificial compliance.

And if that’s all we want, we have to ask:

What does that say about us?