Mimicked Minds, Real Risk
Why risk from effectively intelligent systems is no less real, regardless of subjective conscious experience.
A common response to concerns about advanced AI goes something like this: it’s just a machine. It doesn’t feel, it doesn’t want, it doesn’t experience the world. Without consciousness - without qualia - it cannot meaningfully threaten us.
This position is understandable. Consciousness feels like the thing that makes us human. It is intimate, intangible, and ineffable. If something does not experience fear, desire, or suffering, how could it possibly behave in ways that warrant serious concern? Surely it could never think for itself without its own conscious mind? We can rest in the belief that without an inner life, systems remain inert tools - powerful, perhaps, but ultimately limited.
And yet, a hurricane does not need qualia to destroy a city, and a market does not need qualia to crash economies. For many of the risks we worry about, it is enough that an AI simulates human intelligence convincingly, and the assumed weight of consciousness can distract us from what really matters.
Qualia matters deeply when we are asking moral questions: Does this entity have a subjective experience of suffering? Do we owe it care? Should it have rights? Kant argued that humans have inherent value and must never be treated as a means to an end. AI, by contrast, may be treated instrumentally, as a tool to achieve goals. These moral and philosophical questions will have a place with advanced AI one day I am sure, but catastrophic risk lives in a different domain, one governed by systems dynamics rather than phenomenology.
From that perspective, it is enough for an AI to behave as if it understands us. If a system can:
- generate plausible plans
- anticipate human responses
- navigate social norms
- persuade, coordinate, and optimise
then it already occupies much of the functional role we associate with intelligence, regardless of whether there is “anything it is like” to be that system. Simulation, at sufficient fidelity, is causally indistinguishable from understanding.
This helps explain a common thread in public discourse. When people say “AI won’t get that advanced,” they are often imagining something like a conscious agent: a machine that genuinely wants things, harbours intentions, perhaps even rebels. This narrative however, while delicious fodder for dystopian sci-fi fiction, overemphasises the importance of qualia.
A far more likely scenario is less cinematic: AI systems that never ‘wake up’, never feel, never care, yet act as if they do. Like NPCs in a game, they follow rules, respond to the world, and adapt to playable character behaviour, all without subjective awareness. Despite this, their outputs shape real decisions, influence real people, and structure the systems around them.
The danger, if there is one, lies in confusing real intelligence with effective intelligence.
Real intelligence carries inner experience; effective intelligence shapes the world. For most social, economic, and political systems, the distinction barely matters. Institutions cannot tell the difference between a conscious advisor and a convincing simulator. Markets respond to signals, not souls. Bureaucracies act on documents, not intentions. Once a system reliably produces outputs that look reasoned, coherent, and authoritative, it is already inside the loop. At that point, whether it “really understands” becomes an academic question.
By treating consciousness as the threshold that matters, we risk delaying concern until long after the structural shift has occurred. We can reassure ourselves that a system is “just pattern matching” even as we integrate it into decision-making pipelines. We can note that it has no desires, even as it shapes incentives. We can observe that it does not feel, even as it alters the abstractions through which future systems, both human and machine, will reason. By the time we are seriously debating machine consciousness, we may already have crossed the line that mattered.
If we want to reason clearly about AI risk, we should stop asking when machines will become like us, and start asking when they become convincing enough to trust. When they are embedded deeply enough to influence not just answers, but the questions themselves.
The red line, if it exists, is not consciousness.