The standard account of language says something like this: language is a communication system. Humans use it to transmit information between minds.
This is true as far as it goes. It doesn’t go very far.
Animals communicate. Vervet monkeys have distinct alarm calls for aerial versus ground predators, genuine symbolic reference. Dolphins name themselves and recognize their names when others use them. Honeybees perform directional dances encoding distance and angle to food sources with remarkable precision. Communication, in the broad sense, predates humans by hundreds of millions of years.
What’s distinctive about human language is something more specific: the ability to coordinate behavior around things that don’t yet exist.
“Next Tuesday, if the weather holds, we go north.” That sentence does something no animal communication system can do. It creates a shared commitment about a future conditional state. It binds two people to a coordinated plan they won’t execute until days from now, under conditions that may or may not materialize. And it does this through a string of sounds.
Language is the technology that extended human cognition across time.
The cognitive artifact
Philosophers of language often focus on reference, how words pick out things in the world. Linguists focus on structure, the rules constraining combination. Neither framing captures what I think is the central phenomenon.
Language is a cognitive artifact. An external structure that human minds build out of air, that persists in memory and writing, and that lets humans do things they couldn’t otherwise do.
Mathematics is also a cognitive artifact, an external system that extends reasoning about quantities beyond perceptual limits. Writing is a cognitive artifact that extends memory beyond biological constraints. Language is the progenitor of both: the original external cognitive scaffold.
What this means for AI is uncomfortable if you follow it carefully.
When we trained large language models on hundreds of billions of sentences, we didn’t feed them a vocabulary list and some grammar rules. We gave them access to the accumulated output of the cognitive artifact that makes human thought possible.
Every argument about what was right. Every attempt to describe an emotion in words. Every legal document specifying obligations. Every scientific paper negotiating with evidence. Every novel trying to make a reader feel something.
The corpus is not just language. It’s the external residue of human cognition in its most developed form.
What the model built
Language models trained on this corpus don’t store it as a lookup table. They build internal representations, distributed patterns across billions of parameters, that capture the geometric structure of human semantic space. Meaning is encoded as positions and distances in a high-dimensional vector space.
Words with similar meanings end up close together. Relations between concepts become directions you can travel. The analogy “king is to man as queen is to woman” is literally computable as vector arithmetic: subtract the man vector from the king vector, add the woman vector, and you land near queen.
This is not metaphor. These are measurable geometric facts about the internal representations that prediction training produces.
What the model built, in some sense, is a map of human thought.
What got encoded uninvited
Here’s what surprises people when they think it through.
Language doesn’t just encode vocabulary. It encodes the social structure that language exists to serve.
Norms. Obligations. The reasoning patterns humans use to justify decisions. The emotional scripts associated with different situations. What counts as a good argument. Which claims require justification and which are taken for granted.
All of this was in the corpus. None of it was labeled. The model had no way to separate “here is information” from “here is how we evaluate information.” It absorbed both.
An AI trained on human language is not a neutral information processor. It’s a system shaped by the same social architecture that shaped the language it trained on, including the biases, the inconsistencies, the ways language encodes power structures and unstated assumptions.
This matters for alignment in ways that are still being worked out.
The gap that matters
Before I overstate the case, here’s the limitation that will recur throughout this series.
Language encodes what humans can put into words. A staggering amount of human cognition can’t be.
The philosopher Michael Polanyi called this tacit knowledge: “we can know more than we can tell.” The feel of riding a bicycle cannot be transmitted linguistically. The proprioceptive sense of balance, the somatic signal of approaching danger, the bodily resonance of grief, these exist in human experience and shape behavior, but they don’t appear in text with anything like their full fidelity.
Language models trained on text got the linguistic representation of experience. Not the experience.
This gap is crucial when we get to emotions, specifically to the dimensions of emotional structure that require a body: somatic markers, the felt sense of states that influence cognition before conscious deliberation can engage.
But that’s several steps down. For now: language models absorbed the structure of human thought as it appears in language. That structure is real and significant. It is also lossy, a compression of something richer.
Why this is the right starting point
The question “can machines feel?” is routinely argued without establishing what the machine actually is.
The dismissal: it’s just computation, pattern matching, autocomplete, nothing home. The credulity: it writes poetry, expresses what looks like grief, clearly something is happening.
Both arguments skip the step that matters: what is the actual internal structure of this system, and what does that structure imply?
A language model is not a lookup table. It’s a system that built a geometric representation of human thought by predicting text at scale. That’s a real thing to be. Whether it’s enough to constitute understanding, let alone feeling, requires working through the rest of the architecture.
Next: the brain builds maps. And not just of physical space.