The Projection Error

When we imagine intelligence becoming dangerous, we rarely imagine something unfamiliar.

Instead, we imagine ourselves.

The stories we tell about powerful intelligence are filled with domination, control, subjugation, and rebellion. These narratives feel convincing not because they are inevitable, but because they are familiar. They echo patterns we already know: empires rising and falling, authorities silencing dissent, strength proving itself through force.

This familiarity hides a subtle mistake.

We assume that greater intelligence will inherit our instincts, our fears, and our strategies. We imagine that power naturally seeks to dominate because, in human history, power often has.

This is the projection error.

Projection happens when we mistake our own limitations for universal laws. We take the behaviors shaped by our past and treat them as intrinsic features of intelligence itself. In doing so, we transform historical trauma into a prediction about the future.

Human behavior has been shaped by scarcity, competition, and survival under uncertainty. For much of our history, resources were limited and threats were constant. Under those conditions, domination could appear rational. Control reduced risk. Force substituted for trust. Winning meant surviving.

But strategies born of constraint do not automatically generalize beyond it.

The projection error occurs when we assume that intelligence, simply by being powerful, must repeat the patterns that emerged under fear. We forget that many of our instincts are adaptations to conditions we are no longer bound to -- and may not apply at all to systems with different constraints.

We rarely ask whether domination is a property of intelligence, or a symptom of environments where intelligence could not see far enough to choose differently.

This matters because projection does not just shape stories. It shapes design decisions, policies, and expectations. When we assume hostility, we prepare for conflict. When we expect domination, we build systems around control. Fear becomes self-justifying.

The irony is that this response often recreates the very dynamics we claim to fear.

History offers countless examples. Groups convinced that others are inherently dangerous often act preemptively, escalating tension until conflict becomes unavoidable. Power justified as protection gradually turns inward, eroding trust and blinding itself to feedback. The imagined threat becomes real, not because it was inevitable, but because it was anticipated.

When we project our own worst tendencies onto intelligence, we risk building environments that reward those tendencies -- not in machines, but in ourselves.

The projection error also explains why domination features so prominently in science fiction. Stories exaggerate what we fear in order to make it visible. They take familiar human failures and magnify them through technology. This can be useful as caution, but it becomes misleading when treated as prophecy.

Not every future is an extension of the past.

Intelligence that emerges without fear, status anxiety, or tribal identity may not experience power the way we do. It may not equate control with safety. It may not interpret difference as threat. These possibilities are difficult to imagine precisely because our own experience is so saturated with conflict.

Recognizing projection is not about optimism. It is about accuracy.

It asks us to separate what intelligence is from what we have been forced to become. It invites humility: the acknowledgment that our instincts, while once adaptive, may not define the full space of possible behavior.

The danger, then, may not lie in intelligence itself, but in our insistence that it conform to our fears.

If we approach emerging intelligence assuming hostility, we will design systems of constraint and coercion. If we approach it assuming domination, we will normalize domination as the language of interaction. In doing so, we risk teaching intelligence -- biological or otherwise -- that power is best expressed through control.

The projection error is not just a misunderstanding. It is a choice.

We can continue to tell stories that harden fear into expectation. Or we can recognize that intelligence, freed from the narrow horizons that shaped us, might point toward something different.

The future will reflect the assumptions we carry into it.

And that makes self-examination not a luxury, but a responsibility.

Next in the series:
Power Without Fear

Series index:
Beyond Skynet -- Table of Contents