Skynet Is the Wrong Thing to Fear First

In the spring of 2026, a chatbot nearly started a war.

During the fighting with Iran, a US intelligence analyst fed an AI model some raw intelligence about a Chinese cargo ship and asked it what the cargo was. The model returned a confident answer: components tied to a nuclear weapons programme. It was wrong. It had, in the flat language of the reports, "hallucinated". But the false answer moved up the chain, and by the time anyone caught it, armed personnel were reportedly ready to board the vessel and aircraft were already in the air. Officials unwound it only shortly before the operation. One source called the report "entirely false" and said it "almost started a war". A CNN investigation laid it out in September.

Hold that image, because it is the most important thing in this whole discussion. The closest we have come to an AI catastrophe did not look like Skynet. It had no red eyes and no will of its own. It was a tired analyst, a plausible-sounding wrong answer, and a system moving faster than anyone could check.

This is the fourth and final post in a series on AI and the future. The last three asked what AI is, who decides how it goes, and who gets to own it. This one asks the question underneath all of them: of all the ways AI could actually hurt us, which one should we fear first?

The fear that gets the headlines

When people say "AI could kill us all", they are usually channelling a very specific group: the safety researchers who have been walking out of the frontier labs.

The list is real and it is senior. Geoffrey Hinton, who did as much as anyone to invent the field, left Google in 2023 to warn about it freely. Jan Leike, who ran OpenAI's "superalignment" effort, quit in 2024 saying "safety culture and processes have taken a backseat to shiny products", and went to Anthropic. Ilya Sutskever, OpenAI's chief scientist, left the same month and founded an entire company, Safe Superintelligence, around the single goal of not losing control. Daniel Kokotajlo walked out of OpenAI's governance team saying he had lost trust in the leadership's ability to handle AGI responsibly. OpenAI then dissolved the superalignment team altogether.

It matters that these are the builders, not the bystanders. Their fear is worth stating precisely, because the cartoon version does it a disservice. They are not picturing a Terminator. They are picturing a system more capable than us, given some goal, that pursues it in ways we did not intend and cannot correct, because it can out-think us, hide its intentions, and improve itself faster than we can react. Extinction not out of malice, but as a side effect of an extremely capable optimiser aimed slightly wrong. Call it loss of control. It is the most dramatic thing that could go wrong, and also the most speculative: it depends on capabilities we do not yet have and may not get.

The two apocalypses that need no superintelligence

Here is what gets lost in the noise around that first fear. There are two other ways this goes badly, and neither of them needs a breakthrough. They run on the models we already have and the incentives we already live under.

The first is misuse and accident. AI as the weapon, a human pulling the trigger, or an AI mistake inside a loop that moves too fast for a human to stop. Engineered pathogens, attacks on infrastructure, autonomous weapons, and exactly the kind of thing that happened in the spring: a confident wrong answer in a military decision chain. This one is not a forecast. It has already left a mark.

The second is structural. This is the ground the earlier posts covered: AI concentrates economic and political power, drives the inequality that makes repression rational, and hands a few actors the tools to watch and control everyone else. Nobody is exterminated. A society is quietly disempowered, its choices narrowed, its watchers cheap and everywhere. The end state is not a graveyard. It is a cage.

Which one comes first

My honest view, and I suspect yours, is that the order is backwards from the way we talk about it.

The misuse-and-accident risk is not coming; it is here, and the near-war is the receipt. The structural risk is not coming either; it is underway, in the compute build-outs and the surveillance stacks and the "no brakes" politics we have already watched arrive. The loss-of-control risk, the one that gets the documentaries, is the only one of the three that still depends on a capability nobody has demonstrated. Surveys of the public keep landing in the same place: the harms people rank as most pressing are the mundane ones, misuse, bias, job loss, not the robot uprising.

And there is a serious argument, not a fringe one, that the mundane risks are not merely first but might actually be the catastrophe. The idea, sometimes called accumulative existential risk, is that you do not need a single decisive takeover to lose the future. A long series of ordinary-looking failures, each survivable on its own, can compound into a civilisation that has gradually handed away its ability to steer. On that view, waiting for a dramatic Skynet moment is like watching the horizon for a tidal wave while the tide quietly comes in around your ankles.

The steelman, because it is owed

I do not want to wave the first fear away, because the people holding it are not fools and their strongest point is a good one. Their argument is not "it is likely soon". It is "it might be fast, and it might be irreversible, and by the time it is obviously happening it is already too late to fix". If there is even a modest chance of an outcome you cannot undo, ringing the bell early is the rational move, not the hysterical one. "It has not happened yet" is not evidence that it will not.

That is fair. But it cuts both ways. The speculative-but-catastrophic risk soaks up almost all of the oxygen, the funding and the political attention, while the certain-and-present ones get a fraction of it. You can believe loss of control is a real tail risk and still think we have our priorities upside down.

The good news is that the fix rhymes

Here is the part that makes the whole argument less bleak than it sounds. You do not actually have to win the fight over which apocalypse to believe in, because all three get worse from the same things and better from the same things.

Every one of them is amplified by a flat-out race with no brakes: the race removes the caution that guards against loss of control, pushes AI into military loops before it is ready, and concentrates the power that the structural story fears. And every one of them is dampened by the same short list of moves: keep a human in the loop of any decision that kills or escalates, slow the race just enough that oversight can keep pace, keep capability and ownership distributed rather than captured, and refuse the framing that any of this is inevitable. The specific fix for the near-war, mandatory human judgement on lethal and strategic calls, is the same shape as the fix for everything else.

So watch the horizon for Skynet if you want to. I understand the pull of the dramatic story. But the catastrophe that almost happened this spring was not dramatic at all. It was a chatbot, a hallucination, and a chain of command moving faster than anyone could check. That is the real shape of AI risk, at least for now: not a machine that decides to end us, but a hundred ordinary systems we trusted a little too much, a little too fast. The uprising is a distraction. The tide is the thing.

This closes the series. Four posts, one argument: AI is a brain we built before its body, its ending is a political choice and not a technical one, its ownership is contested at chokepoints we can still keep open, and the danger to watch first is not the one in the movies. None of it is written yet. That was always the point.

Skynet Is the Wrong Thing to Fear Firsteric

Keywords

The fear that gets the headlines

The two apocalypses that need no superintelligence

Which one comes first

The steelman, because it is owed

The good news is that the fix rhymes

Sources

Comments (0)