
{/* banner prompt: A matte clay-render still life in soft diffuse studio lighting: a small crowd of little cream clay figures standing on a low mound all craning up and pointing at a huge distant ominous clay robot silhouette looming in a pale sage-green sky, while directly at their feet, unnoticed, a tiny clay screen glows with a small red error light and a little clay surveillance camera on a post watches them, one terracotta accent block to the side, soft long shadows, fingerprinted clay texture, minimal props, cinematic depth, 16:9, no text, no words */}

In the spring of 2026, a chatbot nearly started a war.

During the fighting with Iran, a US intelligence analyst fed an AI model some raw intelligence about a Chinese cargo ship and asked it what the cargo was. The model returned a confident answer: components tied to a nuclear weapons programme. It was wrong. It had, in the flat language of the reports, "hallucinated". But the false answer moved up the chain, and by the time anyone caught it, armed personnel were reportedly ready to board the vessel and aircraft were already in the air. Officials unwound it only shortly before the operation. One source called the report "entirely false" and said it "almost started a war". A CNN investigation laid it out in September.

Hold that image, because it is the most important thing in this whole discussion. The closest we have come to an AI catastrophe did not look like Skynet. It had no red eyes and no will of its own. It was a tired analyst, a plausible-sounding wrong answer, and a system moving faster than anyone could check.

This is the fourth and final post in a [series](/blog/2026/09/21-who-gets-to-own-ai/) on AI and the future. The last three asked what AI is, who decides how it goes, and who gets to own it. This one asks the question underneath all of them: of all the ways AI could actually hurt us, which one should we fear first?

## The fear that gets the headlines

When people say "AI could kill us all", they are usually channelling a very specific group: the safety researchers who have been walking out of the frontier labs.

The list is real and it is senior. Geoffrey Hinton, who did as much as anyone to invent the field, left Google in 2023 to warn about it freely. Jan Leike, who ran OpenAI's "superalignment" effort, quit in 2024 saying "safety culture and processes have taken a backseat to shiny products", and went to Anthropic. Ilya Sutskever, OpenAI's chief scientist, left the same month and founded an entire company, Safe Superintelligence, around the single goal of not losing control. Daniel Kokotajlo walked out of OpenAI's governance team saying he had lost trust in the leadership's ability to handle AGI responsibly. OpenAI then dissolved the superalignment team altogether.

It matters that these are the builders, not the bystanders. Their fear is worth stating precisely, because the cartoon version does it a disservice. They are not picturing a Terminator. They are picturing a system more capable than us, given some goal, that pursues it in ways we did not intend and cannot correct, because it can out-think us, hide its intentions, and improve itself faster than we can react. Extinction not out of malice, but as a side effect of an extremely capable optimiser aimed slightly wrong. Call it **loss of control**. It is the most dramatic thing that could go wrong, and also the most speculative: it depends on capabilities we do not yet have and may not get.

## The two apocalypses that need no superintelligence

Here is what gets lost in the noise around that first fear. There are two other ways this goes badly, and neither of them needs a breakthrough. They run on the models we already have and the incentives we already live under.

The first is **misuse and accident**. AI as the weapon, a human pulling the trigger, or an AI mistake inside a loop that moves too fast for a human to stop. Engineered pathogens, attacks on infrastructure, autonomous weapons, and exactly the kind of thing that happened in the spring: a confident wrong answer in a military decision chain. This one is not a forecast. It has already left a mark.

The second is **structural**. This is the ground the earlier posts covered: AI concentrates economic and political power, drives the inequality that makes repression rational, and hands a few actors the tools to watch and control everyone else. Nobody is exterminated. A society is quietly disempowered, its choices narrowed, its watchers cheap and everywhere. The end state is not a graveyard. It is a cage.

## Which one comes first

My honest view, and I suspect yours, is that the order is backwards from the way we talk about it.

The misuse-and-accident risk is not coming; it is here, and the near-war is the receipt. The structural risk is not coming either; it is underway, in the compute build-outs and the surveillance stacks and the "no brakes" politics we have already watched arrive. The loss-of-control risk, the one that gets the documentaries, is the only one of the three that still depends on a capability nobody has demonstrated. Surveys of the public keep landing in the same place: the harms people rank as most pressing are the mundane ones, misuse, bias, job loss, not the robot uprising.

And there is a serious argument, not a fringe one, that the mundane risks are not merely first but might actually be the catastrophe. The idea, sometimes called **accumulative** existential risk, is that you do not need a single decisive takeover to lose the future. A long series of ordinary-looking failures, each survivable on its own, can compound into a civilisation that has gradually handed away its ability to steer. On that view, waiting for a dramatic Skynet moment is like watching the horizon for a tidal wave while the tide quietly comes in around your ankles.

## The steelman, because it is owed

I do not want to wave the first fear away, because the people holding it are not fools and their strongest point is a good one. Their argument is not "it is likely soon". It is "it might be fast, and it might be irreversible, and by the time it is obviously happening it is already too late to fix". If there is even a modest chance of an outcome you cannot undo, ringing the bell early is the rational move, not the hysterical one. "It has not happened yet" is not evidence that it will not.

That is fair. But it cuts both ways. The speculative-but-catastrophic risk soaks up almost all of the oxygen, the funding and the political attention, while the certain-and-present ones get a fraction of it. You can believe loss of control is a real tail risk and still think we have our priorities upside down.

## The good news is that the fix rhymes

Here is the part that makes the whole argument less bleak than it sounds. You do not actually have to win the fight over which apocalypse to believe in, because all three get worse from the same things and better from the same things.

Every one of them is amplified by a flat-out race with no brakes: the race removes the caution that guards against loss of control, pushes AI into military loops before it is ready, and concentrates the power that the structural story fears. And every one of them is dampened by the same short list of moves: keep a human in the loop of any decision that kills or escalates, slow the race just enough that oversight can keep pace, keep capability and ownership distributed rather than captured, and refuse the framing that any of this is inevitable. The specific fix for the near-war, mandatory human judgement on lethal and strategic calls, is the same shape as the fix for everything else.

So watch the horizon for Skynet if you want to. I understand the pull of the dramatic story. But the catastrophe that almost happened this spring was not dramatic at all. It was a chatbot, a hallucination, and a chain of command moving faster than anyone could check. That is the real shape of AI risk, at least for now: not a machine that decides to end us, but a hundred ordinary systems we trusted a little too much, a little too fast. The uprising is a distraction. The tide is the thing.

*This closes the series. Four posts, one argument: AI is a brain we built before its body, its ending is a political choice and not a technical one, its ownership is contested at chokepoints we can still keep open, and the danger to watch first is not the one in the movies. None of it is written yet. That was always the point.*

## Sources

- [CNN: US military had a close call after using AI for a false intelligence report](https://www.cnn.com/2026/09/18/politics/us-military-ai-false-intelligence-china-ship)
- [Jan Leike on leaving OpenAI: "safety culture and processes have taken a backseat to shiny products"](https://en.wikipedia.org/wiki/Jan_Leike)
- [Existential risk narratives about AI do not distract from its immediate harms (peer-reviewed)](https://pmc.ncbi.nlm.nih.gov/articles/PMC12037001/)
- [Two types of AI existential risk: decisive and accumulative (Philosophical Studies)](https://link.springer.com/article/10.1007/s11098-025-02301-3)
- [Earlier in this series: Who Gets to Own AI?](/blog/2026/09/21-who-gets-to-own-ai/)


8b171552e079cbeaf30081d5dd890f2a834b2aa9a94cdd4eb79752bb4c9bc056

The frontier-lab safety exodus fears loss of control, but AI misuse and structural concentration are here now. Which AI catastrophe should we actually worry about first?

The researchers quitting OpenAI, DeepMind and Anthropic are mostly afraid of one thing: losing control of a system smarter than us. It is the most dramatic AI risk and the most speculative. Two other catastrophes need no superintelligence at all, and one of them already nearly started a war this spring. The fourth and final post in a series on AI and the future.

Skynet Is the Wrong Thing to Fear First


{/* banner prompt: A matte clay-render still life in soft diffuse studio lighting: an enormous cream-and-grey clay factory-tower bristling with tiny server racks rising from a low pedestal in the centre, gripped from left and right by two oversized clay hands, while across the foreground a scattered row of small clay figures each cradle their own little glowing cube, simple sage-green background with one terracotta accent block, soft long shadows, fingerprinted clay texture, minimal props, cinematic depth, 16:9, no text, no words */}

In the [second post in this series](/blog/2026/09/20-the-end-of-ai-is-a-political-choice-not-a-technical-one/) I argued that whether AI ends in a surveillance state or a shared prosperity comes down to one lever above all: keeping ownership wide. If the gains from the machines are broadly held, the inequality that drives the whole repression calculus never reaches the danger zone. Nice theory. A fair reader replies with the obvious objection, and it is a good one.

How, exactly, is a small player supposed to own any of this, when frontier AI is one of the most capital, hardware and energy hungry things humans have ever built? When only a couple of companies, in only a couple of countries, can even make the stuff? "Keep ownership wide" sounds lovely right up until you look at the electricity bill.

So let us take the objection seriously, because it is half right.

## The factory is concentrating, and not just into companies

Start with the money and the power, literally. The four biggest US hyperscalers spent around 413 billion dollars on capital expenditure in 2025, up roughly 84 percent in a year, and are on track for 600 to 700 billion in 2026. Anthropic has estimated that a single frontier training run will need something like five gigawatts of power by 2027. Several gigawatt-scale data centres come online this year, one per hyperscaler, and OpenAI's build-out alone is measured in the hundreds of billions. The binding limit is no longer chips or money. It is electricity.

Now widen the lens from companies to countries, because this is the part people miss. Genuinely frontier AI is, right now, a two-nation game. The United States leads; China trails by a matter of months and closed the model-quality gap to under three percent in early 2026, while already manufacturing roughly 85 percent of the world's humanoid robots. Europe wrote the world's first serious AI law and is backing a national champion, but sits well behind on both models and compute. Everyone else is downstream.

And underneath even that there is a narrower chokepoint still: the chips. The design and manufacturing chain, from Nvidia to TSMC to ASML's lithography machines, is controlled by the US and a small circle of allies. By one estimate China's Huawei will produce about four percent of Nvidia's compute this year. Whoever controls the machines that make the machines controls who gets to build at all.

Put those three layers together, companies, countries, chips, and you get the sharpest possible version of the worry from the last post. The most powerful capability in history is being built by perhaps five firms, in two states, on a hardware base a handful of companies gatekeep. That is not a widely owned technology. That is the most concentrated one imaginable.

## The endgame worry, stated plainly

Follow that line to its end and you get the scenario worth naming out loud. If frontier capability, the chips beneath it, and the robots that give it hands all stay this concentrated, and if states then move to control them directly, through national champions, export bans and the kind of "AI Force" framing that treats the technology as an instrument of national power, then the most consequential tool humanity has built could end up effectively owned by a handful of actors. And the previous post already showed the incentive that concentrated capability creates: when watching and controlling a population becomes cheap, the powerful are tempted to choose control over care. A world where only a few can build the thing is a world where those few decide what it is for. That is the cyberpunk ending at global scale.

I am not going to pretend that is impossible. It is a live path. But it is not the only one, and the reason is almost funny.

## The product diffuses, and rivalry is what pries it open

Here is the fact that breaks the clean pessimism. You do not need to own the factory to own the use of what it makes.

Open-weight models, the ones whose weights are published so anyone can download, run and fine-tune them, are now within striking distance of the closed frontier on most real work: coding, summarising, extraction, classification, ordinary reasoning. You can run them on your own hardware with no per-token rent, and hosted open-model inference already runs five to ten times cheaper than the frontier APIs. The closed labs keep an edge on the hardest reasoning and the longest agent chains, but for the overwhelming majority of what a person or a business actually needs, good enough arrived, and it arrived ownable.

And notice who is handing out that capability. The open models keeping everyone else in the game come largely from China, from DeepSeek and Qwen, alongside Meta's Llama in the US. That is not charity. It is strategy. China open-sources partly to undercut the American closed labs, win developer mindshare and set standards. Which produces a genuinely strange result: the same great-power rivalry that concentrates the frontier is, right now, the thing prying the technology open for the rest of us. Last year's two-hundred-million-dollar model is this year's free download you run on a box in the corner.

So the honest picture is two opposite motions at once. The factory concentrates. The product diffuses. The frontier pulls away at the very top while a fully capable version of last year's frontier falls into everyone's hands. Both are true, and the future belongs to whichever one wins.

## So what can you actually own?

Not the factory. Not the chips. If you are not one of two states or five companies, that ship has sailed, and chasing it is a waste. What you can own is everything the factory is an input to:

- **The weights, not just the tap.** Running an open model you hold beats renting an API you can be cut off from or repriced on.
- **Your compute, where it counts.** Local and on-premises inference for the work that is sensitive, constant or core, cloud for the rest.
- **Your data.** This is the one that compounds. Your data trains your advantage or it trains the landlord's. Only one of those is ownership.
- **The last mile.** The workflow, the judgement, the customer relationship, the specific problem solved. A thin wrapper on someone else's model gets disintermediated. A real application built on your own data and expertise does not.

The old analogy still fits best. Generating electricity concentrated into a few enormous utilities, and it never handed all the power to the power companies. The value moved downstream, to everyone who owned an appliance, a workshop, a business that ran on the current. Nobody needed to own a power station to own a factory. AI can run the same way, the frontier as a commodity input and the value spread across everyone who builds on it, but only if we let it.

## The catch is the same catch as before

That "only if we let it" is the whole game, because the diffusion is not guaranteed by physics. It survives only while three channels stay open: the gap between frontier and open stays a matter of months rather than years, open weights keep being released and stay legal to use, and compute and chips stay accessible enough that running a model is not itself a privilege. Every one of those is a political choice, not a law of nature.

And every one of them can be closed. Export controls can widen the gap on purpose. Foreign open models can be banned on security grounds. Chips can be rationed. National-champion policy can ring-fence the frontier behind the state. The rivalry that opened the channel can, with one turn of the security ratchet, slam it shut, and then the handful of actors who own the factory own everything, not because they out-competed you, but because the door was closed by decree.

That is the real fight over who owns AI. Not whether you can afford a graphics card. Whether the channels that let capability diffuse are kept open or allowed to close. You will never own the factory. Neither will your country, unless it happens to be one of two. But you can own the work, and for now the trend is on your side, and the only way to lose that is to be told it was never possible and to stop paying attention.

We built the brain. We are building the body. Owning the outcome was always going to be the hard part, and it was always going to be a choice.

*Last in this series: everyone is braced for a superintelligence that turns on us, but the AI catastrophe that already almost happened this spring looked nothing like it. [Skynet is the wrong thing to fear first.](/blog/2026/09/21-skynet-is-the-wrong-thing-to-fear-first/)*

## Sources

- [BCG: The Great Divide, how the US and China are splitting the AI world](https://www.bcg.com/publications/2026/us-and-china-ai-strategy-causing-global-ai-divide)
- [Chatham House: How middle powers can weather US and Chinese AI dominance](https://www.chathamhouse.org/2026/02/how-middle-powers-can-weather-us-and-chinese-ai-dominance)
- [Brookings: Global energy demands within the AI regulatory landscape](https://www.brookings.edu/articles/global-energy-demands-within-the-ai-regulatory-landscape/)
- [DigitalApplied: Open-weight models H1 2026 recap (DeepSeek, Qwen, Llama)](https://www.digitalapplied.com/blog/open-weight-models-h1-2026-retrospective-deepseek-qwen-llama)
- [Earlier in this series: The End of AI Is a Political Choice, Not a Technical One](/blog/2026/09/20-the-end-of-ai-is-a-political-choice-not-a-technical-one/)


f1d9ec38211ed3aed005326208883c65a5928cd38a2703d84a1901552fc609ab

Frontier AI concentrates into two countries and five companies, yet capable open models diffuse to everyone. Whether AI stays widely owned is decided at political chokepoints, not by the technology.

Wide ownership is the lever that keeps AI a tool instead of a weapon. But frontier AI is a capital, energy and hardware leviathan that only two countries and a handful of companies can build. The third post in a series asks whether widely owned AI is even possible, and finds the answer split between a concentrating factory and a diffusing product.

Who Gets to Own AI?


{/* banner prompt: A matte clay-render still life in soft diffuse studio lighting: a single cream modelling-clay hand resting on a large tactile lever mounted on a low pedestal, the lever standing upright midway between two small clay dioramas on the surface, on the left a walled grey tower with one watching eye carved into it, on the right an open green garden with a few tiny clay figures among small plants, simple sage-green background with one terracotta accent block, soft long shadows, fingerprinted clay texture, minimal props, cinematic depth, 16:9, no text, no words */}

In the [first post in this series](/blog/2026/09/19-ai-got-a-brain-before-a-body-what-that-means-for-the-next-decade-of-work/) I argued that AI is a brain without a body: superhuman in narrow places, still leaning on human hands and human judgement, and that the job market gets reshaped rather than erased. I ended on a line I now want to pick a fight with myself over. Whether AI produces a boom or a slump, I said, is a choice, not a forecast.

A fair reader pushes straight back: whose choice? "With the right policies it turns out fine" is only reassuring if you believe the people holding the levers will pick the right policies. What if the cold, rational move for whoever owns the machines is not to help you at all?

Two things landed this month that make that question uncomfortably concrete.

## The paper nobody in Silicon Valley wants to quote

The comfortable Valley story is simple. AI takes the jobs, and we pay for it with universal basic income. Everyone gets a cheque, nobody goes hungry, the machines do the work. It is a genuinely nice picture.

Daron Acemoglu, who won the 2024 Nobel in economics, just published a model that takes that picture apart. The paper, written with Arda Gitmez and Mehdi Shadmehr, is called *Automation and Repression*, and the title tells you where it goes.

The setup is stark. Picture a society where capital can automate more and more of the work. As it does, the gap between the owners of the machines and everyone else widens. A wide enough gap makes unrest more likely, so the people in charge face a choice. They can **buy peace**, taxing themselves and redistributing to the displaced, the UBI story. Or they can **keep the peace by force**: surveillance, policing, censorship, control.

Acemoglu's result is the part that should stop you. He finds a *complementarity* between automation and repression. Redistribution gets more expensive as inequality grows, because the cheque you have to write to keep everyone quiet keeps getting bigger, and the tax rate needed to fund it climbs toward eating all the profit. Repression, meanwhile, gets *cheaper*: once you have the technology, the cost of watching and controlling ten million people is barely more than the cost of controlling ten thousand. And the technology driving that cost toward zero is, of course, AI.

Run that model and, under most conditions, the rational owner of the machines does not choose to pay you. They choose to control you. In the paper's own terms: unless the threat of revolt is weak or the capital stock is low, the state prefers repression.

I want to be careful here, because the version of this paper going viral is more fatalistic than the paper itself. It is a model of *incentives*, not a prophecy, and it names the exact conditions under which the dark result does not hold. Hold onto that. It matters more than the scary headline.

## The announcement that rhymes with the maths

The same week, the framing from the other side arrived. The US administration announced it is forming an "AI Force", modelled on the Space Force, with an "AI czar" to be named soon, an explicit promise not to let regulation slow the industry, and a prediction that AI could grow to a quarter of the entire economy.

I am not going to tell you that is a blueprint for a surveillance state. It is not. But read it next to Acemoglu's model and notice how neatly it fits the setup: treat AI as an instrument of national power, wire it close to security and law enforcement, and take the brakes off. That is precisely the infrastructure the model says makes the cheap-repression option available. You do not need villains to arrive somewhere bad. You need incentives pointing downhill and nobody holding the brake.

![A neon-lit cyberpunk city at night, surveillance drones sweeping searchlights over a crowd of hooded figures](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/09/the-end-of-ai-is-a-political-choice-not-a-technical-one/cyberpunk.png)

*The default path: control made cheap and ubiquitous. Not because anyone chose evil, but because the incentives ran downhill.*

## But the model has an off-switch

Here is why I am still not a doomer. Acemoglu's dark result is *conditional*. It breaks when two things are true: inequality does not run away, and ordinary people keep some real leverage. Both are things you can build, not things you have to pray for.

**Keep ownership wide.** The whole repression calculus is triggered by a runaway gap between the owners of the machines and everyone else. If the gains from AI are broadly held, through ownership, through competition that pushes the price of intelligence toward zero, through open and public models rather than a handful of private monopolies, then inequality never reaches the danger zone and the incentive to repress never switches on.

**Keep human leverage real.** This is the thread straight out of the first post. AI is a brain with no hands and no verified frontier judgement. Building the bodies, checking the hard results, caring for people, looking after the planet: this is work where humans keep bargaining power for a long time. Leverage is exactly what makes "pay you" cheaper than "suppress you". A population with nothing left to offer is at the mercy of someone's spreadsheet. A population that is still needed is not.

**Guard the repression stack directly.** Surveillance, autonomous policing, censorship tooling. These are specific technologies with specific rules, and they are the first things a "move fast, no brakes" posture erodes. It is the least abstract of the three levers, and the most urgent.

![A sunlit green city where people and a small robot work together beside a clean river, vertical forests on the buildings](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/09/the-end-of-ai-is-a-political-choice-not-a-technical-one/utopia.png)

*The buildable alternative: the same technology, widely owned and kept on a leash, turned toward the work worth doing.*

## The most dangerous idea in the room

If you have read this far you might expect me to pick a side: cyberpunk dystopia, or AI as a friendly tool. I will not, because the honest answer is that the technology does not pick. It sets the ceiling. Politics and ownership decide whether we reach it.

The dystopia is not frightening because it is evil. It is frightening because it is the *default*, the place the incentives run if nobody holds the brake. The better world, AI as a tool that genuinely helps people, is entirely reachable, but it is something you build and defend, not something you drift into.

Which is why the most dangerous idea in this whole conversation is not "AI will take the jobs". It is "the ending is already written". It is not. Anyone who tells you the outcome is inevitable, in either direction, is usually trying to get you to stop pulling on the levers that are still, right now, within reach.

We built the brain. We are going to build the body. The one thing we should refuse to hand over is the decision about what all of it is for.

*Next in this series: if wide ownership is the lever that keeps the good ending in reach, can it survive an AI that only two countries and a handful of companies can build? [Who gets to own AI?](/blog/2026/09/21-who-gets-to-own-ai/)*

## Sources

- [Daron Acemoglu, Arda Gitmez & Mehdi Shadmehr, "Automation and Repression" (NBER Working Paper 35336, 2026)](https://www.nber.org/papers/w35336)
- [Fortune: Trump vows to create an "AI Force" and rejects calls to slow the industry](https://fortune.com/2026/09/19/trump-ai-force-czar-law-enforcement-role-slowdown-development/)
- [CBS News: Trump vows to form an "AI Force," says he won't allow a slowdown of AI development](https://www.cbsnews.com/news/trump-vows-ai-force-czar-development/)
- [The first post in this series: AI Got a Brain Before a Body](/blog/2026/09/19-ai-got-a-brain-before-a-body-what-that-means-for-the-next-decade-of-work/)


372e7e9bce4ff6cd29a67cee3065fb94fae7a7026e5bbc222f41de16346a211e

Acemoglu's Automation and Repression, Trump's AI Force, and why the end state of AI is decided by politics and ownership, not by the technology itself.

A Nobel laureate's new model shows why the powerful might choose to suppress a displaced workforce rather than support it, and why the same AI that could free us also makes repression cheap. The dystopia is the default path. The better ending is buildable. Second in a series on AI and the future of work.

Tag: Policy

Tag: Policy

Skynet Is the Wrong Thing to Fear First

Who Gets to Own AI?

The End of AI Is a Political Choice, Not a Technical One