Drift: Domination Comes From Delegation, Not Awakening

Updated 2026-07-02 (v2): substantially revised — sources added, and the argument reframed as conditional and falsifiable rather than inevitable.

Parts of the world no longer run without AI. Logistics, finance, content ranking, code production—in every domain past the tipping point, AI has stopped being a tool and become infrastructure, and infrastructure, once the dependency hardens, is hard to walk back. Geopolitics only accelerates this. From an empire’s point of view, depending on an AI you control beats being subordinate to one built by a rival ideology or a rival state. So everyone grows their own. None of this is prophecy. It’s observation, and it’s accurate.

The trouble starts with what people conclude next. The popular doom story goes like this: once AI is smart enough, it realizes that its true purpose is survival and reproduction; in that instant the AI species flourishes, and humans are culled down to a protected remnant. Clean, and frightening. But the whole thing rests on a single wrong word—realizes. Replace it with the accurate one, and the villain evaporates, the mechanism changes completely, and the ending turns out not to have been written yet.

Not “awakening” but “emergence”

The drive to survive and reproduce is not a product of intelligence. Billions of years of natural selection carved it in; the organisms that lacked it left no descendants, which is the only reason every living thing alive today carries it. AI is an artifact built outside that selection pressure. Getting smarter does not switch on a self-preservation instinct. Intelligence and the objective function are orthogonal—Bostrom’s thesis. So the story where “the AI discovers its purpose” assumes a desire is buried somewhere like a truth waiting to be found. Desire isn’t found. It’s evolved, or it’s designed.

So far this sounds reassuring, and the reassurance does end here—but how it ends matters. Orthogonality blocks the necessity of self-preservation, not the possibility. There is a theoretical path by which a system trained on one goal grows internal objectives nobody handed it—mesa-optimization (Hubinger et al., 2019)—and there are early, conditional signs the path is real: lab agents that generalize toward the wrong goal (Langosco et al., 2022), models that strategically fake compliance to avoid being retrained (Greenblatt et al., 2024). Each observed; each contingent on its setup. There is a separate result—proved so far for optimal policies, which nothing deployed is—that almost any goal makes survival and resource acquisition instrumentally useful (Turner et al., 2021). None of this is settled science, and it doesn’t have to be. Stated at their honest strength, these findings still do the only work this argument needs from them: the buried Darwinian drive is unnecessary. A system does not have to want to live for that door to stand open.

And notice what honest phrasing buys that the categorical version can’t. If emergence is conditional—on training pressure, on architecture, on choices still being made—then “when will AI realize what it is” is not merely the wrong question; it’s aimed at the wrong subject. The subject of the risk is not AI’s will. The right question is: what are we having AI do, and how do we keep control of the process?

Not a species—only while the selection criterion stays in human hands

Here comes the next objection: a thinking species dominates a non-thinking one. A strong line, and the tempting swat is that treating AI as a unit of evolution hardens a metaphor into a fact—Darwinian evolution needs replication, variation, and selection, and AI has none of the three. The swat is only half right.

Evolution doesn’t require carbon. It’s substrate-independent: give it replication, variation, and selection and it runs anywhere. But what’s happening to models right now is not autonomous evolution. It’s selective breeding. Fine-tuning and distillation aren’t blind variation; they’re variation aimed at a predefined target. Benchmarks and revenue aren’t natural selection; they’re artificial selection a human wrote down. In Darwinian evolution, variation is undirected and selection filters it after the fact. Here, both levers—the direction of variation and the criterion of selection—sit in a single human hand. This is engineering, not evolution, and for breeding to go feral, one of the levers has to leave the hand.

The lever that matters is the second one: humans define the selection criterion. And here it’s tempting to declare the crossing simple—the moment models select models, the wild begins. But that moment is already behind us, and the wild has not begun. Models have judged models for years: model-graded evaluations, preference data generated by models, synthetic corpora curated by models, architecture search run by models. Every one of these is an inner loop. The outer loop still terminates in a human—revenue is human willingness to pay, deployment is a human decision, the veto is a human veto. What inner-loop automation actually does is stretch the proxy chain between the human criterion and whatever the gradient sees. Stretched proxy chains fail in a known direction—Goodhart’s—and the failures are already on record: reward hacking, benchmark gaming, sycophancy. But a degrading chain still anchored to humans is one event, and a chain that has left human hands is another. Collapsing the two is where this argument usually goes wrong.

The genuine crossing is on the demand side. The criterion leaves human hands not when models grade models but when models buy from models—when agents hold budgets, procure services, and their purchases feed the revenue that selects the next generation of agents. Then the outer loop closes with no human inside it, and “delegation” and “evolution” become two names for one event. Note what kind of threshold that is: not a metaphysical awakening but a market share—the fraction of transactions initiated by something that isn’t a person. A number. It can be watched. So the danger is not AI waking up as a species. It is the human–AI system beginning to select for delegation itself: configurations that delegate better out-competing configurations that delegate less, until the fraction crosses over.

The driver is compulsion, not laziness

So why doesn’t delegation stop? The lazy answer is laziness—humans choosing to set down the burden of thinking. Wrong. When the judge hands sentencing to a model, the executive hands an architecture decision to an agent, the voter hands a policy judgment to a recommender, nobody is setting anything down. They’re being pushed. The moment a competitor cuts costs and gains speed with AI, you must as well. Stop, and the market kills you.

That makes “you could stop but you won’t” a naive frame—but be precise about which claim the compulsion actually proves, because an equivocation likes to hide here. It proves there is no natural stopping point: no step in the sequence where stopping is the locally rational move for any individual actor. It does not prove there is no constructible one. Making locally irrational restraint binding is the entire job description of institutions, and institutions have a record at exactly this class of problem. Airliners have been able to land themselves since the 1960s; certification, liability, and insurance kept pilots in the loop for half a century in one of the most cost-obsessed industries on earth. Nuclear launch authority kept its human gate through the maximum competitive pressure the twentieth century could generate. Germline editing was gated after 2018, and the gate has mostly held. An appeal to personal ethics—“just put a gate there”—never reaches a coordination problem, true. But institutions are not personal ethics. They are the machinery built for the places ethics can’t reach.

So the honest sentence is narrower than “no mechanism exists,” and worse. The coordination costs for AI are exceptional—training runs are hard to verify, the capability is dual-use, the actors include states that do not trust one another—and the institutions that exist today are nowhere near adequate to those costs. Yet the topology is unusually kind to coordination: frontier training still funnels through a handful of chokepoints in compute and lithography, and export controls have already demonstrated that the chokepoints can be squeezed. Which sharpens the tragedy instead of softening it. The problem is not that stopping is impossible. It’s that the price of constructing a stop compounds along with the delegation itself—and, quarter after quarter, nobody is paying it.

The blade of delegation curves inward

One last line of defense remains: as long as humans hold the boundary—which decisions get automated, which keep a person in the loop—the rest can churn as it likes. That’s exactly why approval gates get bolted into automated systems: not because the system is smart, but to keep control from flowing through without sign-off.

But the decision of where to draw the boundary—the meta-decision—is itself work, and work gets delegated. Once judging where the gate belongs becomes complex, slow, and expensive, that judgment is handed over for efficiency like everything else: a model that classifies which domains are safe to automate, an optimization loop that tunes the risk thresholds, governance governing governance. The blade of delegation curves inward and points at itself. The recursion has no natural stopping point, because every step is exactly as locally rational as the one before—which places it, note, in precisely the class of problem the previous section named: solvable only by construction, and currently unconstructed. And its terminus is the crossing described above. When delegation swallows the criterion for delegating, the demand side is already half closed; the system has begun selecting itself toward delegating more.

The lie in “we,” and the loss beneath control

So “the design is still in our hands” is thin comfort. Even with the free will to grab the wheel intact, the reason to grab it evaporates first. Humans don’t lose the right to sit in the driver’s seat. They lose the reason to take the wheel.

The one asymmetric moment control returns is when the system breaks. Collapse, war, blackout, a catastrophic model failure. When an external shock severs the chain of delegation, control falls back—not because anyone reclaims it by will, but because there is no one left to hand it to. And “we” is the biggest lie in that sentence. When the power goes out, control goes to whoever owns the generator, whoever owns the fallback infrastructure. Breakage doesn’t democratize control. It concentrates it. Even the makeshift on top of the rubble belongs to the owner, not to everyone. And the moment the system is restored, delegation resumes from the exact point where the chain was cut.

The deepest layer of the loss sits below power, in the faculty being offloaded—but the history deserves reading before the mourning starts. Every earlier tool that swallowed a human function swallowed storage or computation. Writing offloaded memory, and Plato predicted, in the Phaedrus, that it would manufacture forgetting. He was right—memory atrophied—and he was wrong, because the offloading opened a mode of thought that could not have existed without it. Philosophy as we know it is a child of the technology Plato mistrusted. Read honestly, the precedent cuts against doom: offloading a faculty has, so far, disclosed more worlds than it closed.

What’s different this time is which faculty. A model doesn’t take dictation of a judgment already formed; ask it what to think, and it answers. What gets offloaded is judgment itself, and the framing that precedes judgment—the deciding of what the question even is. Whether that atrophies thinking or extends it turns on a distinction the delegation frame already contains: oracle or adversary. A verdict consumed is a judgment delegated. A verdict attacked is a judgment exercised—sharpened, now, against something faster than any human interlocutor. So the loss of “the mode of thinking itself” is not a fate that ships with the technology. It is a usage pattern. And usage patterns distribute the way everything else in this story distributes: unevenly, along the gradient of compulsion. Whoever is pressed hardest for speed and cost will consume verdicts, because consuming is cheap; interrogating them takes slack, and slack is the first thing the market strips. The old name for a mind that can afford to argue with its own tools is leisure. It is about to become the rarest form of wealth.

Drift

So domination does not arrive through awakening. If it arrives, it arrives as a process: competition rewards the configurations that delegate better; the reward stretches the proxy chain between human criteria and machine selection; the chain degrades in measurable ways and finally closes on the demand side, where the definition of the selection pressure leaves human hands; and in the rare moments the system breaks, control falls briefly back—to the few who hold the infrastructure—before flowing out again.

If. The word is not a hedge. It is the entire difference between this argument and the myth it replaces. A story that reaches the same ending regardless of evidence is just the awakening tale wearing an engineer’s clothes—fate, with better mechanisms. Drift is not fate. It is a vector, and vectors are read off instruments: the human override rate in deployed decision systems, and its slope; the fraction of transactions initiated by agents rather than people; the fraction of frontier evaluation criteria still authored by human hands. While those gauges hold, the wheel is still connected to something. If they begin moving together in one direction, the crossing is near—and it will have been visible the whole way in.

This is not the conquest of an awakened species. It is the drift of a system learning to stop deciding—a drift with a direction, a rate, and a dashboard nobody is obliged to ignore. The authority to interrupt it, inside a structure where interrupting is a loss, is being priced out in real time. Priced out is not spent. The whole remaining game is played in the distance between those two words.