Permitted Intelligence

When a danger is announced, the announcement tells you less than the remedy that follows it. People argue about whether the danger exists, and that argument deserves to happen. But a remedy has a shape, and the shape decides who pays and who collects. Every industry that has lived through a safety scare learns this: whether the danger existed gets settled in a few years, and who holds the keys gets settled for a generation.

This week the danger on offer is a small AI model running on cheap hardware. On October 12, 2026, two researchers on Anthropic’s Frontier Red Team, William Zhou and Daniel Freeman, published a post asking whether AI models may soon be capable of self-replicating without access to a datacenter. The post measures what it claims, and the recommendations at the end are the ordinary recommendations of network defense. It does not ask for a ban on anything, and the argument I am about to make is not a rebuttal of a proposal nobody wrote.

My argument is about what happens next. A finding like this enters a policy conversation in which “local”, “open weights”, and “small” keep sliding together into a category called risk, while “hosted”, “closed”, and “monitored” keep sliding together into a category called safety. If that sorting hardens into rules, the rules will decide where intelligence is allowed to live. So here is my thesis, stated once without hedging: AI safety must weigh the risk of concentrating capability, access, and decision-making in a few organizations as carefully as it weighs the danger used to justify that concentration.

One disclosure first. I build PeerLLM, a network where people serve language models from hardware they own. I have a stake in independent inference remaining legal and practical, and I will not pretend otherwise. Hold my argument to the standard I am asking of everyone else: follow the evidence, and ask who gains.

0/ Fifteen Televisions

The researchers bought fifteen off-brand Android TV boxes at about $55 each, wired them together, and split a 26-billion-parameter open model across them. It worked, at roughly 0.39 tokens per second. In separate tests, they asked models of 30 billion parameters and under to perform the individual steps of replication: find a device, set up inference on it, launch a copy. The models could do many of the steps. Composing them end to end was another matter, and only one open-weight model completed the full loop in a simplified evaluation.

Read the simplifications as carefully as the results, because the authors did. The break-in step was made trivial through unauthenticated debug ports, and the authors call the outcome an upper bound. The targets were emulators in a cloud, not appliances in a living room. The model doing the thinking ran on separate infrastructure. The forecast, that a small model could reach the capability of the one that succeeded by the end of 2026, is an extrapolation from a trend line, and the authors call it rough. The post ends with defense: watch ISP traffic, run honeypots, quarantine infected subscribers, ship secure defaults, harden systems.

Not everyone was impressed. Reuters reported on October 13 that Ciaran Martin, who formerly ran Britain’s National Cyber Security Centre, responded with a line worth keeping: “‘We managed to run a small model on cheap TVs,’ does not a security crisis make.” He read the post as a technical paper with vendor exaggeration in it. Other experts in the same report shared the concern without endorsing the timeline. That is the honest range: a measured experiment, a modest result, an uncertain forecast. My question starts on the far side of the forecast. Suppose it comes true. What follows?

1/ Follow the Money

Start with what does not follow. It does not follow that the researchers want independent AI gone, and nobody at Anthropic has said so. What I will do is trace the consequences if a finding like this becomes a restriction on running models outside approved infrastructure, because that chain runs in one direction regardless of intent. If restrictions make self-hosting unavailable or expensive, the people who still need AI do not stop needing it. They move to whatever remains permitted, and what remains permitted is hosted access from a handful of providers. Those providers gain customers, pricing power, and something harder to measure: the authority to decide what counts as an acceptable use. A free tier creates the same dependency as a subscription, since access continues exactly as long as the provider’s permission does.

I call the thing on the far end of that chain Permitted Intelligence: intelligence you may use only for as long as someone else continues to allow it. Permitted Intelligence is not a slur on hosted models, which I use. It is a description of a relationship. The relationship is fine when it is one option among several. It becomes something else when it is the only option left standing, and a safety rule is the most respectable way to clear the field.

Three things have to be kept apart here. An incentive is a fact about who would benefit if a policy passed. A policy outcome is what happens when it does. Regulatory capture is a claim about intent. The first is easy to establish, the second can be measured, and the third needs evidence I do not have. A sincere safety proposal can still produce an outcome favorable to its advocates, and financial benefit does not disprove a security finding. The correct response to an incentive is not an accusation. It is scrutiny.

2/ The Marshal’s Restaurant

Imagine a town where a kitchen fire burns down a house. The fire marshal proposes that nobody cook at home anymore. Everyone should eat at licensed restaurants, where the stoves are inspected and a fire can be contained. The marshal also happens to own the town’s largest restaurant chain. That ownership makes neither the fire fake nor inspected stoves a bad idea. But the town would be foolish not to ask two questions before voting. How many fires would the ban prevent, given that people also burn candles and run space heaters? And what does the town lose when every meal passes through a kitchen it does not own?

Those two questions already have a formal shape. In 2024, Sayash Kapoor, Rishi Bommasani, Arvind Narayanan and their colleagues published a framework for assessing the marginal risk of open foundation models: not whether an open model can participate in a harm, but how much risk it adds beyond what closed models and existing tools already allow. The framework asks you to identify the threat and the attacker, establish the risk that exists without open models, assess the defenses already in place, show evidence of the added risk, and state your assumptions. It refuses to assume that restricted releases are safer, treating them instead as a comparison point whose defenses must be measured too.

The self-replication post passes the first steps of that framework and leaves the rest open, which is fine for research and inadequate for policy. Nothing in the experiment establishes that restricting local inference would remove the capability from an attacker who can rent cloud compute, buy a used GPU, or download weights that have been public for a year. The United States government reached a similar position in July 2024, when the NTIA recommended against restricting open model weights for the systems that then existed, in favor of building the evidence and acting on it. That was one agency’s position at one time, not a guarantee, but it is a precedent for the shape of the decision: evidence first, restriction only when the evidence demands it.

3/ Comparing Machines

Here is the comparison I keep seeing skipped. On one side, a model running on an RTX 3080 at home. On the other, a frontier model in a data center, running as an agent with tools, credentials, and a budget of millions of GPU hours. If the question is which system has the greater capacity to cause damage, the answer is not the one with the TV boxes. Capability scales with compute, and so does the ability to plan, to persist through failures, and to act across many targets at once. The concentrated system deserves at least the same scrutiny as the distributed one.

But the careless version of that argument is wrong too. Danger is not a property of location. It is the product of at least five variables: what the model can do, how much compute it has, what tools and credentials it can reach, how much autonomy it is given, and how well it is contained. A small local assistant with no network access and read-only files is close to harmless. A small local agent holding a production database password and a shell can drop the database, and the parameter count will not save you. Compute creates potential capacity. Exposure, controls, and the task decide what harm follows.

The same discipline applies to the words. Local, open weights, small, and decentralized are four different things that happen to travel together in this debate. An open model can run in the largest data center on earth, and a closed model can be small. When a policy restricts “local models”, it should be made to say which of these properties it is targeting and why. If it cannot, it is restricting a label.

4/ Hard to Kill

The strongest technical point in the self-replication post is about persistence, not power. A replicator spread across thousands of consumer devices has no central switch, so it is hard to eradicate even if each copy is weak. But persistence and severity are different dimensions, and a safety argument that measures one and reports the other is making an error.

Think of the difference between a weed and a flood. The weed is almost impossible to eliminate and does little harm on any given day. The flood is over in a week and takes the town with it. A centralized system with an off switch can still do enormous damage before anyone reaches the switch, and the switch only helps if someone notices in time and has the authority to act. A distributed system without a switch can still be contained device by device, as the post’s own recommendations describe. Score likelihood, severity, spread, persistence, and recoverability separately, then ask which of them the restriction addresses.

I will grant the irreversibility point in full, because it is true. Once weights are released, they cannot be recalled, and safeguards trained into them can be removed by anyone with modest compute. What that does not do is justify the leap from “this release cannot be undone” to “all independent inference should be restricted”. The first is a statement about model weights. The second is a statement about people and their hardware.

5/ One Switch

The concentrated alternative has failure modes of its own, and they belong in the same ledger. Shared dependencies mean correlated failures: when one provider goes down, every product built on it goes down with it. Access can be revoked for a person, a company, or a country. Prices, terms, and the definition of acceptable use can all change, decided by a small number of people the user has never met. None of these is a cyberattack, and I will not pretend they are the same category of harm. They are a different category, and a policy that moves everyone onto fewer switches has to count them.

The economics point the same way. Anton Korinek and Jai Vipra, in a 2024 working paper for the National Bureau of Economic Research, find economies of scale and scope in foundation models that push toward concentration, a risk of the market tipping toward a few players, and vertical integration that strengthens the pull. They also note that concentrated control over increasingly capable AI systems could create a kind of power without precedent. The Kapoor and Bommasani work adds the monoculture problem. When everyone runs the same model, everyone shares the same failures and the same blind spots. This is the part of the ledger that a safety argument built only on misuse never reaches. The marshal’s restaurant may have better stoves, but every meal in town now depends on one supplier’s opening hours.

6/ The Strongest Case

Now the other side, given its due. The best argument for provider-controlled deployment is that hosted providers can see abuse and stop it: monitor patterns across millions of requests, revoke a bad actor’s key, patch a jailbreak in an afternoon. A released model offers none of that, and the gap favors the hosted provider. The honest questions are how effective those controls are in practice, what they miss, and what new risks come from depending on them. Monitoring that sees every abuse also sees every conversation, and that is a cost to privacy I would not pay lightly.

The second argument is aggregate scale: many weak agents on many cheap devices could collectively matter even if no single one does. I agree, and the answer is to compare realistic resources, coordination, access, and defenses, rather than treating a hypothetical fleet as a certainty. The third argument is aimed at me: a company can benefit from a policy that protects people, and local AI advocates have financial interests too. The benefit does not invalidate the policy, and my stake does not invalidate my argument. What either one earns is a demand for independent evidence of net benefit, applied in both directions, and a disclosure, which I have made.

7/ The Future

Here is the test to carry into the next safety announcement, because there will be more. First, name the harm precisely: who is hurt, how badly, how often, and how recoverably. Second, demand the comparison: what would an attacker retain through other means, and what would the restriction remove. Third, inspect the remedy: does it target unauthorized behavior or lawful possession of a general-purpose tool, and have the less restrictive measures (scoped credentials, execution boundaries, independent testing) been shown inadequate. Fourth, ask who acquires power if it passes. A proposal that survives all four is worth supporting, whoever wrote it. A proposal that fails the fourth and refuses to discuss it is telling you something.

I think about this as someone who believes technology is supposed to elevate people. The ability to operate useful intelligence independently, on hardware you own, for purposes you chose, is not a line item on a subscription. It is the difference between a tool and a permission. A hammer that someone else can take back is not a hammer, it is a loan, and a civilization that can only borrow its intelligence has agreed to a dependency it may not be able to undo.

Safety is not optional. But safety that quietly relocates power into fewer hands is not safer, it is only tidier. The work ahead is to build the evidence, defend the devices, and keep the question of who owns intelligence open long enough for ordinary people to answer it for themselves.

Hassan