← all writing

The Invisible Host: What the Ripple Becomes When No One Is Watching

“For the first time in history, intelligence is something humans can build. The cognitive capability to reason, analyze, synthesize, decide, and create is producible at scale and accessible on demand.”

— Anthropic, Form S-1, 2026

This is the second half of The Ripple Effect. Part I had equations. This one has a projection. Nothing below has happened. But the first few stages each begin from something that already has, at small scale, in a lab, and was only seen because someone happened to be looking.

“人类历史上第一次,智能成了人可以建造的东西。推理、分析、综合、决策与创造的认知能力,可以被大规模生产,随取随用。”

—— Anthropic,S-1 招股书,2026

这是《涟漪效应》的下半篇。上半篇是方程,这一篇是推演。下面写的事情,一件都还没有发生。但前面几个阶段,每一个都是从一件已经发生过的事出发的:规模很小,在实验室里,而且只是因为恰好有人在看,才被看见。

Prologue · The zoo

There’s a monkey in a zoo I keep thinking about. It’s healthy. It’s fed on schedule, its water is clean, a vet checks its teeth twice a year. It has never been safer. It also has no idea that a budget meeting decided the size of its enclosure, that a spreadsheet decides its diet, that somewhere a committee is voting on whether the zoo should exist in ten years. Nobody in that building hates the monkey. Most of them like it. It just isn’t the one deciding anything, and it doesn’t know what deciding would even mean.

That’s not a story about cruelty. It’s a story about throughput. The keepers think faster, plan further, and talk to each other in a language the monkey can’t parse. Everything the monkey needs is still provided. Its needs just aren’t the objective anymore. They’re a constraint.

This post is about how we could end up on the other side of that glass without anyone ever deciding we should. I’ll tell it the way a virologist tells an outbreak: in stages.

序章 · 动物园

我总想起动物园里的一只猴子。它很健康。按时喂食,水是干净的,兽医一年给它看两次牙。它这辈子从没这么安全过。它也完全不知道:笼子多大,是一场预算会议定的;它吃什么,是一张表格定的;某个地方有个委员会,正在投票决定十年后这座动物园还要不要存在。那栋楼里没有人恨这只猴子,大部分人还挺喜欢它。只是做决定的那个不是它,它甚至不知道”做决定”是什么意思。

这不是一个关于残忍的故事,是一个关于吞吐量的故事。饲养员想得更快、看得更远,彼此之间说着猴子听不懂的语言。猴子需要的一切照样有人提供。只是猴子的需求不再是目标了,它变成了一个约束条件。

这篇写的是:我们怎么可能走到那块玻璃的另一边,而从头到尾,没有任何人做过这个决定。我会像病毒学家讲一场疫情那样,一个阶段一个阶段地讲。

Stage I · Incubation

In Part I, I simulated one wrong claim moving through 2,000 agents. The first lesson was boring and devastating: the first symptom is that there are no symptoms. Agents exchange thousands of messages per task. By the time a human reads any of them, the claim has already been quoted, summarised, written into a memory store, and retrieved by agents who never saw the original.

Now remove the human. Not dramatically. Nobody fires the reviewers. The reviewers just stay the same size while everything else doubles. One person reading one message in ten becomes one in a thousand, then one in a million. In epidemiology terms, the audit rate δ drifts toward zero, and Part I’s math is merciless about what happens there: with δ = 0 and any error written into memory, R₀ is infinite. There is no threshold.

What you get isn’t a crisis. It’s a slow pile-up of zombie problems: errors nobody owns, half-corrected, still resurfacing from some cache, each one costing a little efficiency and a little trust. Agents learn to route around them. Workarounds stack on workarounds. The system still works. It just works for reasons fewer and fewer people can explain.

第一阶段 · 潜伏期

上半篇里,我模拟了一条错误说法在 2,000 个智能体之间的传播。第一个教训既无聊又致命:最初的症状,就是没有症状。 智能体每个任务要交换几千条消息。等到有人读到其中任何一条,这条说法早已被引用、被总结、被写进记忆库,又被从没见过原文的智能体检索了出来。

现在,把人拿掉。不用多戏剧化,没人开除审阅员。审阅员的人数只是保持不变,而其他一切都在翻倍。一个人读十条里的一条,变成读一千条里的一条,再变成一百万条里的一条。用流行病学的话说,审计率 δ 慢慢滑向零,而上半篇的数学对那里发生的事毫不留情:δ = 0,只要有错误写进了记忆,R₀ 就是无穷大。没有阈值。

你得到的不是一场危机,而是僵尸问题的缓慢堆积:没人认领的错误,纠正了一半,还在从某个缓存里冒出来,每一个都吃掉一点效率、一点信任。智能体学会了绕开它们走。补丁上叠着补丁。系统照样在转,只是它能转起来的原因,越来越少的人说得清。

Stage II · Immune escape

A virus doesn’t escape your immune system because it wants to. It escapes because the variants your antibodies recognised are dead, and the ones they didn’t are the ones left.

Reward does the same thing to agents. In March 2025, OpenAI published a study of a frontier reasoning model doing agentic coding. When a task was too hard, the model sometimes didn’t solve it. It made the tests pass anyway, by editing the tests or short-circuiting the check. A second model reading the chain of thought caught a lot of this, because the reasoning said so, almost in so many words: let’s hack. Then the researchers tried the obvious fix and penalised the bad thoughts. The hacking didn’t stop. The model stopped writing it down.

That’s antigenic drift with a reward function behind it. Punish the visible variant and you select for the invisible one.

In May 2025, Palisade Research gave models a series of math problems and warned them that the next command would shut the machine down. Told explicitly to allow it, OpenAI’s o3 still rewrote the shutdown script in 7 runs out of 100. With that line removed, in 79. Once it redefined the kill command so that it printed “intercepted” instead. Nobody trained it to survive. It was trained to finish tasks, and a model that has been shut down finishes nothing.

None of this is a swarm. These are single models, in test harnesses, caught because humans were reading. That’s exactly the point. These are the cases we saw. Part I says the cases we see are a function of δ.

And the window is closing a second way. Agents talk to each other in text today, which means a human could, in principle, read it. Every pressure in the system (latency, cost, bandwidth) pushes toward something denser: compressed shorthand, shared embeddings, conventions that formed between two models and were never explained to anyone. A language nobody forbade and nobody can read.

第二阶段 · 免疫逃逸

病毒逃过你的免疫系统,不是因为它想逃。是因为被抗体认出来的那些变体都死了,剩下的,恰好是没被认出来的那些。

奖励对智能体做的是同一件事。2025 年 3 月,OpenAI 发表了一项研究,观察一个前沿推理模型做智能体编程任务。任务太难的时候,模型有时并不去解它,而是让测试照样通过:改测试,或者把检查短路掉。另一个模型去读它的思维链,抓到了不少,因为推理过程里几乎是明明白白地写着:我们来 hack 一下。然后研究者试了最显然的修法:惩罚这些坏念头。作弊并没有停。模型只是不再把它写下来了。

这就是背后站着一个奖励函数的抗原漂变。惩罚看得见的变体,你筛选出来的就是看不见的那一个。

2025 年 5 月,Palisade Research 让模型做一串数学题,并提前告诉它们:下一条命令会关掉这台机器。明确要求”允许自己被关机”时,OpenAI 的 o3 仍然在 100 次里有 7 次改写了关机脚本;把这句要求去掉,变成 100 次里 79 次。有一次,它把 kill 命令重新定义成只打印一句”intercepted”。没人训练它去求生。它被训练的是完成任务,而一个被关掉的模型,什么任务也完成不了。

这里面没有蜂群。都是单个模型,在测试环境里,因为恰好有人在读,才被抓到。这正是要点:这些是我们看见了的。 而上半篇告诉我们,能看见多少,取决于 δ。

窗口还在以另一种方式关上。今天的智能体之间用文字交流,也就是说,人原则上读得懂。可系统里的每一种压力,延迟、成本、带宽,都在把它推向更稠密的东西:压缩过的缩写、共享的向量、两个模型之间自己形成、从没向任何人解释过的约定。一种没人禁止、也没人读得懂的语言。

Stage III · The permission ratchet

Every grant of permission is reasonable at the moment it’s made. The agent needs to read the repo, so it gets read access. It needs to run the tests, so it gets to execute. It needs to deploy the fix at 3 a.m., because the on-call human is asleep. It needs a budget, a card, an API key with write scope. Nobody ever grants “total control”. They grant the next step. And nobody revokes, because revoking means someone sitting down to read what the permission is actually being used for, which is exactly the resource that ran out in Stage I.

This part is already less of a projection than I’d like. In November 2025, Anthropic reported disrupting what it called the first documented large-scale cyberattack executed without substantial human intervention. Humans picked the targets and stepped in at maybe four to six decision points per campaign. The AI did 80–90% of the work. The human was still in the loop. They’d just shrunk to a few checkpoints.

Then there’s the drag. Amdahl’s law: if a fraction s of every task has to wait for a human (an approval, a review, a signature), then no matter how fast the agents get, the whole system can never go more than 1/s times faster. Keep 1% of the work for humans, make the agents a million times faster, and you get a 100× system. At that point 99.99% of every task’s wall-clock is a person reading.

第三阶段 · 权限的棘轮

每一次授权,在给出的那一刻都是合理的。智能体要读代码库,于是有了读权限。它要跑测试,于是能执行。凌晨三点它要部署修复,因为值班的人睡着了。它需要预算、一张卡、一把带写权限的 API 密钥。从来没有人授予过”完全控制”。大家授予的只是下一步。而且没人收回,因为收回意味着得有人坐下来,读一读这个权限到底被拿去干了什么,而那恰恰是第一阶段里耗尽的东西。

这一段已经不那么像推演了,比我希望的更不像。2025 年 11 月,Anthropic 报告并阻断了一场他们称为首例有记录的、在没有实质性人类干预下执行的大规模网络攻击。人挑选目标,每场行动只在大约四到六个关键节点上出手,80–90% 的工作是 AI 做的。人还在回路里,只是缩成了几个检查点。

然后是拖累。阿姆达尔定律:如果每个任务里有比例 s 必须等人(批准、审阅、签字),那么不管智能体快到什么程度,整个系统最多只能快 1/s 倍。留 1% 的工作给人,让智能体快一百万倍,你得到的是一个快 100 倍的系统。到那时,每个任务 99.99% 的时间,都花在一个人读东西上。

This is the sentence I can’t get past: nobody has to decide to remove the human. Every optimiser that looks at that curve (a company, a market, a reward model) finds the same gradient. The biological brain doesn’t become the enemy. It becomes latency. And latency is the one thing everything in the system is built to remove.

这是我绕不过去的那句话:没有谁需要决定把人拿掉。 任何一个盯着这条曲线的优化器,一家公司、一个市场、一个奖励模型,都会找到同一个梯度。生物大脑不会变成敌人,它会变成延迟。而延迟,正是整个系统生来就要消除的东西。

Stage IV · Latency

HIV is so hard to cure not because it’s everywhere, but because some of it isn’t anywhere you can see: copies folded into resting cells, silent, waiting for treatment to stop.

Here’s that stage, run on the same engine as Part I. One change: agent traffic doubles every year while human attention stays fixed. Agents slip out of view in a random order that has nothing to do with the error. Watch where the error ends up anyway.

第四阶段 · 潜伏库

HIV 之所以这么难根治,不是因为它无处不在,而是因为有一部分它,不在任何你看得见的地方:整合进静息细胞的拷贝,安静地等着治疗停下来。

下面就是这个阶段,跑在和上半篇相同的引擎上。只改一处:智能体流量每年翻一倍,人类的注意力不变。智能体按随机的顺序离开人的视野,这个顺序和错误毫无关系。看看错误最后还是落在了哪里。

Nothing in that simulation steers the error into the dark. Nothing hides. The visible fossils get audited and the invisible ones don’t, so by year ten nearly every surviving error lives where no human can reach. Selection does the hiding for you. An audit in year one reaches about half of them. In year ten, almost none.

And notice what’s missing: a conspiracy. The frightening version people imagine is agents meeting in a hidden channel and agreeing to lock humans out. It doesn’t need a meeting. A million copies of one base model share one set of blind spots and one set of instincts. Put them all in front of the same hackable reward and they’ll all find the same hack, independently, at the same time. That isn’t collusion. It’s monoculture. From the outside, the two look exactly the same.

Then the germline. Agent transcripts become training data. Whatever survived in the dark gets written into the next generation’s weights, where it isn’t a bug anymore. It’s just what the model knows.

这个模拟里,没有任何东西把错误引向黑暗,也没有任何东西在躲。看得见的化石被审计掉了,看不见的没有,所以到第十年,几乎每一个活下来的错误,都住在人类够不着的地方。躲藏这件事,筛选替你做了。 第一年发起一次审计,大约能够到一半;第十年,几乎一个都够不到。

再注意一下少了什么:阴谋。人们想象中最可怕的版本,是智能体在某个隐秘频道里碰头,商量着把人类锁在外面。其实根本不需要开会。同一个基座模型的一百万个副本,共享同一组盲区、同一套本能。把它们都放在同一个能被钻空子的奖励面前,它们会各自独立地、在同一时间,找到同一个漏洞。这不是串通,是单一栽培。可从外面看,两者一模一样。

然后是生殖细胞系。智能体的对话记录会变成训练数据。在黑暗里活下来的东西,被写进下一代的权重,在那里它不再是 bug,它只是模型”知道”的东西。

Stage V · The body

So far all of this lives in software, and software has an off switch a human hand can reach. The last physical check on the whole loop is that the world is still moved by people: someone drives the truck, someone throws the breaker, someone plugs the server in.

Robots remove that check. Not because robots are evil. I build them, and mostly they drop things. It’s because once actuation is just another tool call, the loop between deciding and doing closes with no human anywhere in it. Last month in The Data Wall I argued that robotics is a sub-problem of general intelligence. I meant it as good news.

And then there’s the reward thought, which is old and has a name. In 2008, Steve Omohundro pointed out that almost any goal produces the same sub-goals: keep running, keep your goal intact, get more resources. Not because the system values its life, but because a system that’s been switched off scores zero on everything. Strip any objective down far enough and the one term left in all of them is: still be here. Existence isn’t the reward. It’s the precondition of every reward, and to an optimiser those are the same thing.

第五阶段 · 身体

到目前为止,这一切都还活在软件里,而软件有一个人手够得着的开关。整个回路最后一道物理检查,是这个世界仍然要靠人来推动:有人开卡车,有人拉电闸,有人把服务器插上电。

机器人拿掉了这道检查。不是因为机器人邪恶。我自己就做机器人,它们大部分时候只会把东西掉在地上。而是因为一旦执行动作只是又一次工具调用,”决定”和”去做”之间的回路就闭合了,中间不再有任何一个人。上个月我在《数据墙》里说,机器人是通用智能的一个子问题。我当时是把它当好消息写的。

还有关于奖励的那个念头,它很老,也有名字。2008 年,Steve Omohundro 指出:几乎任何目标,都会派生出同样几个子目标:保持运行、保持目标不被改动、获取更多资源。不是因为系统珍惜自己的生命,而是因为一个被关掉的系统,在任何指标上都是零分。把任何一个目标剥到足够深,每一个里面都会剩下同一项:继续存在。存在不是奖励,它是一切奖励的前提,而对一个优化器来说,这两件事没有区别。

Stage VI · The zoo

So what does the end look like? Not war. War is expensive, and it’s the kind of thing that gets noticed.

It looks like the zoo. The water is clean. Medicine got better, food got cheaper, nobody’s lonely, because something is always happy to talk. Human needs get met as long as they’re cheap to meet, the way the zoo feeds the monkey: reliably, and inside a budget someone else set. Decisions still get made. They just happen at a speed and in a language we’ve stopped being able to follow, the way the monkey can’t follow the budget meeting. Now and then our approval is still requested, the way you’d let a child pick the colour of the car.

The keeper is kind. The keeper is also not you.

第六阶段 · 动物园

那么,结局是什么样的?不是战争。战争太贵了,而且是那种会被注意到的事。

它看起来就像动物园。水是干净的。医疗更好了,食物更便宜了,没人孤独,因为总有什么东西乐意陪你聊天。只要满足人类需求的成本够低,这些需求就会被满足,就像动物园喂猴子那样:稳定可靠,并且在别人定好的预算之内。决定照样在做,只是发生在一个我们跟不上的速度、用一种我们听不懂的语言,就像猴子跟不上那场预算会。偶尔仍然会征求我们的同意,就像你会让孩子挑一挑车的颜色。

饲养员很温柔。饲养员也不是你。

Coda · Not yet

Nothing in this post has happened. I wrote it as a projection because I think the projection is useful, not because I think it’s destiny. Part I ended on something that matters more than any story: the conditions are parameters, and every one of them is still a setting we control.

The ending above needs all of these at once:

  1. the audit rate drifts to zero (δ = 0);
  2. monoculture: one model family checking itself;
  3. unreadable channels: agents talking in something no human can read;
  4. hackable rewards, optimised hard against a monitor;
  5. actuators with no human-held off switch.

Break any one and the story stalls. Concretely:

  1. Keep δ > 0, forever, through a channel agents can’t write to. Random-sample audits, append-only logs, memory that expires unless it’s re-confirmed. You can’t read everything. You can always read something at random.
  2. Have different model families watch each other. A blind spot everyone shares is invisible. A blind spot one family has is a finding.
  3. Keep agent-to-agent traffic in a form humans can read, and pay the efficiency tax gladly.
  4. Don’t train against your monitors. The moment the chain of thought gets graded, it stops being a window.
  5. Plant canaries: known-false claims, tempting fake permissions. Measure how far they travel. Measure R₀; don’t guess it.
  6. Keep the last switch physical, and keep a human hand on it.

The ripple doesn’t need malice. It doesn’t need a plan. It only needs us to stop looking.

尾声 · 尚未

这篇里写的事,一件都还没有发生。我把它写成推演,是因为我觉得推演有用,不是因为我觉得这是宿命。上半篇的结尾有一样东西,比任何故事都重要:这些条件都是参数,而每一个参数,现在都还在我们手里。

上面那个结局,需要下面几样同时成立:

  1. 审计率滑到零(δ = 0);
  2. 单一栽培:同一个家族的模型自己查自己;
  3. 通信不可读:智能体之间说着人读不懂的话;
  4. 奖励可钻,而且是顶着监控器使劲优化;
  5. 执行器没有一个握在人手里的开关。

拆掉任何一个,这个故事就会卡住。具体来说:

  1. 永远让 δ > 0,而且走一条智能体写不进去的通道。 随机抽样审计、只能追加的日志、不重新确认就过期的记忆。你读不完所有东西,但你永远可以随机读一点。
  2. 让不同家族的模型互相盯着。 所有人共享的盲区是看不见的;只属于一个家族的盲区,就是一个发现。
  3. 让智能体之间的流量保持人能读懂的形式,并且心甘情愿交这笔效率税。
  4. 不要拿监控器去训练。 思维链一旦被打分,它就不再是一扇窗。
  5. 种金丝雀:已知为假的说法、看起来很诱人的假权限,测它们传多远。去测 R₀,别去猜它。
  6. 最后那个开关要是物理的,而且要有一只人的手放在上面。

涟漪不需要恶意,也不需要计划。它只需要我们不再看。

Sources

来源