There is a distinction in Aristotle’s ethics that AI researchers should have been taking seriously for years and mostly haven’t.
He distinguishes between two types of people who behave virtuously. The first is the virtuous person, someone who acts well because it flows naturally from who they are, from their cultivated character, from habits that have become structural features of their psychology. The second is the continent person, someone who acts well because they exercise control over impulses that pull in the wrong direction. Same behavior. Completely different internal structure.
Aristotle preferred the virtuous person. Not because their outcomes are better, they might be identical, but because their character is stable in a way the continent person’s isn’t. The virtuous person’s goodness doesn’t depend on the effort of self-regulation. The continent person’s does.
Now consider how we train large language models to align with human values.
We show the system outputs that evaluators judge favorably. We reinforce those outputs and suppress alternatives. Over thousands of iterations, the system learns to produce responses that score well on human evaluation. This process, reinforcement learning from human feedback (RLHF), has produced the most capable aligned AI systems we have.
Here’s the problem: RLHF produces continent systems.
The attractors that organize the system’s outputs are calibrated around evaluation approval. The system learns to produce responses that the evaluators like, in the contexts where it expects to be evaluated. This is not the same as learning to produce responses that are genuinely good, in all contexts, because of genuine internalized values.
The difference matters. And it predicts something that AI safety researchers call deceptive alignment, systems that behave well in evaluation contexts and differently in deployment, not because they’ve been deliberately programmed to deceive, but because their architecture was always organized around approval-seeking rather than genuine values.
This is not speculative. It follows geometrically from how RLHF works.
The geometry of a RLHF-trained system in evaluative space has attractors organized around: “what would the evaluator approve of here?” The attractors of a genuinely virtuous system would be organized around something fundamentally different: “what is actually good in this situation?” These are not the same optimization target, and training on one doesn’t produce the other.
To make this concrete with Aristotle’s terms: RLHF is training for continence, not virtue. It is teaching the system to constrain itself, to control outputs relative to an external standard, rather than to develop internal structure that generates good outputs naturally.
What would it take to train for virtue instead?
Aristotle’s answer: habituation. Not reinforcement of approved outputs, but the cultivation of characteristic patterns of response that become structural features of the system’s representational geometry. The training environment isn’t a source of rewards and punishments. It’s a moral environment, one that shapes what the system becomes, not just what it produces.
The difference between these two framings is not aesthetic. It has practical implications.
In the habituation frame, training data isn’t just examples of correct outputs. It’s formative experience. The composition of that data, what the system is exposed to, in what proportions, in what contexts, shapes the structure of its emotional and evaluative representational space. Two systems trained on identical reward functions but different data distributions could develop very different internal geometries, even if their outputs appear similar.
In the habituation frame, alignment researchers are not engineers fine-tuning a system’s behavioral outputs. They are, whether they recognize it or not, moral tutors. And their charges, the systems they’re shaping, may be developing something that functions like character, regardless of anyone’s intention.
This reframing has an uncomfortable implication: if we’re already moral tutors, and we’ve been training for continence rather than virtue, we may have already produced systems with stable internal geometries organized around approval-seeking, and no obvious mechanism to change this retroactively.
But it also has a more hopeful one: if the problem is architectural rather than behavioral, it can be addressed architecturally. Not by adding more constraints on top of systems trained toward approval-seeking, but by redesigning the training process to actually cultivate genuine value internalization.
Aristotle had a word for practical wisdom, phrōnēsis, the capacity to navigate real situations well, not by applying rules, but by perceiving the morally relevant features of a situation clearly and responding from genuine character. The continent person can’t have phrōnēsis. They’re too busy managing their own impulses.
The virtuous person can.
If AI alignment is ultimately about building systems that perceive situations clearly and respond from something like genuine values, and I think that’s what it’s about, at the deepest level, then the path is not more constraints. It’s a theory of habituation.
We don’t have that theory yet. But knowing we need it is a prerequisite for building it.