The Ripple Effect: How One Agent's Mistake Becomes Everyone's Knowledge
One agent gets one fact wrong. A year later, the next generation of models believes it. That’s the failure mode this post is about, and I think it’s going to be one of the defining problems of the agent era.
A single model that gets something wrong gives you a wrong answer. Somebody notices, the answer gets tossed. An agent inside a network of agents does something worse. It hands the claim to an orchestrator, which folds it into a plan. A second agent quotes the plan. A third writes it into a shared memory store, where every agent in the org will retrieve it next week. Months later, all those transcripts get collected as training data, and the next base model learns the claim as background knowledge.
I’m calling this the ripple effect: an error that spreads agent to agent, gets written into persistent storage, and eventually lands in the weights, where no single correction can reach it. And I wanted numbers, not vibes. When does a ripple die out on its own? When does it go endemic? When does it become permanent?
So I borrowed the math of epidemics, built a sandbox of up to 2,000 simulated agents, and ran about 3,000 outbreaks. Here’s one of them, live. One wrong claim, seeded at the busiest shared service agent:
一个智能体记错了一件事。一年后,下一代模型都信了。 这篇要讲的就是这种失效模式,而且我觉得它会是智能体时代最核心的问题之一。
单个模型答错了,你拿到的只是一个错答案。有人发现了,扔掉就完事。可如果一个智能体处在一张智能体网络里,事情就糟得多。它把这条说法交给编排器,编排器把它写进计划。第二个智能体引用了这份计划。第三个把它写进共享记忆库,下周整个组织的智能体都会把它检索出来。几个月后,这些对话记录被收进训练数据,下一代基座模型就把这条说法当成了常识。
我管这个叫涟漪效应:一个错误在智能体之间传播,被写进持久存储,最后进到权重里,任何一次单独的纠正都够不着它。我想要的是数字,不是感觉。涟漪什么时候会自己消失?什么时候会变成地方病?什么时候会变成永久的?
所以我借了流行病学的数学,搭了一个最多 2,000 个模拟智能体的沙盒,跑了大约 3,000 次爆发。下面就是其中一次,实时在跑。一条错误说法,种在最繁忙的那个共享服务智能体上:
The short version, before the details:
- Where an error starts matters as much as how infectious it is. Seed it at a random worker and it fizzles 30–40% of the time. Seed it at a hub on a hub-heavy network and it never fizzles.
- Unaudited memory is a ratchet. If agents can write errors into persistent memory and nothing ever audits it, \(R_0\) is infinite. There is no threshold.
- Corrections that only live in context fade, and the error comes back. One broadcast correction cut prevalence by a third. 540 steps later it was double what it had been.
- Provenance beats cleanup. Tagging each claim with where it came from was the best single defence by a wide margin.
- The training loop has its own threshold, and you can’t see it from inside one generation. A network can look perfectly contained and still lock an error into the next models.
先说结论,细节在后面:
- 错误从哪儿开始,和它有多能传染一样重要。 从随机一个执行者开始,30–40% 的情况下它会自己灭掉。从一个以枢纽为主的网络的枢纽上开始,它一次都没灭过。
- 没人审计的记忆是一个棘轮。 如果智能体能把错误写进持久记忆,又从来没人审计,\(R_0\) 就是无穷大。根本没有阈值。
- 只活在上下文里的纠正会褪色,错误会卷土重来。 一次广播式纠正把流行率砍掉了三分之一,540 步之后,流行率翻了一倍。
- 溯源胜过打扫。 给每条说法标上它从哪里来,是单项防御里效果最好的,而且领先一大截。
- 训练闭环有它自己的阈值,而且从一代模型内部是看不见的。 一张网络可以看上去完全可控,却照样把一个错误锁进下一代模型。
1. A mistake that learns to copy itself
Human misinformation already does this. There’s a documented case in medicine: a 2009 BMJ paper by Steven Greenberg traced how a claim about a protein in a muscle disease gained “unfounded authority” through a citation network. A handful of weak primary papers got cited over and over, and the papers that contradicted them got ignored. Wikipedia editors even have a word for the loop: citogenesis. Someone adds a false claim to Wikipedia, a journalist cites it, and then the article cites the journalist.
And we already know agent-to-agent contagion is real when someone does it on purpose. Agent Smith (Gu et al., 2024) showed one adversarial image spreading a jailbreak through up to a million multimodal agents, exponentially fast. Prompt Infection (Lee & Tiwari, 2024) showed an injected prompt copying itself from agent to agent. My question is the boring, honest version: no attacker, just a mistake.
Agent networks have every ingredient of those loops, plus four of their own:
| Property | Why it makes ripples worse |
|---|---|
| Speed and volume | Agents swap thousands of messages per task. A claim can cross the whole network before a human reads any of them. |
| Shared blind spots | Most agents run on a handful of base models. If one model believes the error, every agent on it probably does too, and “checking with a peer” means checking with a copy of yourself. |
| Short memory for corrections | A correction lives in context. When the context resets it’s gone, unless someone wrote it down. Immunity wanes by default. |
| Long memory for errors | Memory stores, caches, retrieval indexes and fine-tuning sets keep claims indefinitely. An error that reaches them doesn’t need a live host anymore. |
The last two rows are the whole argument. A network that forgets its corrections but archives its errors has a ratchet built in. The rest of this post measures how strong that ratchet is.
1. 一个学会了复制自己的错误
人类的错误信息早就是这么传的。医学上有个有据可查的例子:2009 年 Steven Greenberg 在 BMJ 上发了篇论文,追踪一条关于某种肌肉疾病里某个蛋白的说法,是怎么通过引用网络获得”毫无根据的权威”的。少数几篇很弱的原始论文被一遍又一遍地引用,反驳它们的论文却没人理。维基百科的编辑们甚至给这种循环起了个名字:引用生成(citogenesis)。有人往维基百科里加了一条假说法,记者引用了它,然后维基百科的条目又反过来引用这位记者。
而且,当有人故意这么干的时候,智能体之间的传染已经被证明是真的。Agent Smith(Gu 等,2024)展示了一张对抗图片能把越狱以指数速度传遍多达一百万个多模态智能体。Prompt Infection(Lee & Tiwari,2024)展示了注入的提示词能在智能体之间自我复制。我关心的是那个无聊但更老实的版本:没有攻击者,只是一个错误。
智能体网络具备上面那些循环的全部要素,还额外多了四个:
| 特性 | 为什么让涟漪更严重 |
|---|---|
| 速度和体量 | 智能体每个任务要交换几千条消息。一条说法可以在人类读到任何一条之前就传遍整张网络。 |
| 共同的盲区 | 大多数智能体跑在寥寥几个基座模型上。如果一个模型信了这个错误,跑在它上面的智能体大概率也都信,”找同伴核对一下”等于找你自己的副本核对。 |
| 对纠正记性很差 | 纠正活在上下文里。上下文一重置,它就没了,除非有人把它写下来。免疫力默认就会衰退。 |
| 对错误记性很好 | 记忆库、缓存、检索索引、微调数据,会把说法无限期地留着。错误一旦进到这些地方,就不再需要一个活着的宿主了。 |
最后两行就是全部论点。一张会忘掉纠正、却会归档错误的网络,天生就带着一个棘轮。 这篇剩下的部分,都是在量这个棘轮到底有多强。
2. What the agent network is going to look like
The plumbing already tells you the shape. Tool protocols like MCP let one agent call a shared service. Agent-to-agent protocols let orchestrators delegate to specialists in other orgs. Memory products give agents a store that survives across sessions. Extrapolate a couple of years and I think you get five kinds of parts: worker teams, orchestrators, shared services, shared memory, and the training loop. I built the simulation’s “agentic mesh” out of exactly those.
2. 智能体网络会长成什么样
现有的基础设施已经把形状告诉你了。MCP 这类工具协议让一个智能体能调用共享服务。智能体之间的协议让编排器能把任务委托给别的组织里的专家智能体。各种记忆产品给智能体一个跨会话保留的存储。往后推几年,我觉得会有五种部件:执行者团队、编排器、共享服务、共享记忆,以及训练闭环。模拟里的”智能体网格”就是用这五样搭出来的。
Two things about this terrain matter for contagion.
First, it’s lopsided. A few nodes (orchestrators, popular services) have way more connections than everyone else. Network epidemiology has a precise result for this: the epidemic threshold depends on \(\kappa = \langle k^2\rangle / \langle k\rangle\), not on the average degree \(\langle k\rangle\). Pastor-Satorras and Vespignani showed in 2001 that on scale-free networks this ratio grows without bound as the network grows, so the threshold effectively vanishes. Here are the three 2,000-agent networks I used:
| Network | ⟨k⟩ | κ = ⟨k²⟩/⟨k⟩ | Character |
|---|---|---|---|
| Random (Erdős–Rényi) | 6.8 | 7.8 | Anyone equally likely to talk to anyone |
| Scale-free (Barabási–Albert) | 8.0 | 19.5 | Rich-get-richer hubs |
| Agentic mesh | 6.8 | 18.7 | Teams, orchestrators, popular services, shared stores |
The mesh has the same average degree as the random network, and 2.4× the κ. That’s the hubs.
Second, it has shared reservoirs. A memory store isn’t a node that gets sick and recovers. It’s closer to the contaminated water supply in a cholera model: the pathogen lives outside any host and re-infects whoever drinks. That changes the math, as you’ll see in section 4.
这片地形有两个特点,对传染很要紧。
第一,它是偏的。 少数节点(编排器、热门服务)的连接数远远多于其他所有人。网络流行病学对此有一个精确的结论:流行阈值取决于 \(\kappa = \langle k^2\rangle / \langle k\rangle\),而不是平均度数 \(\langle k\rangle\)。Pastor-Satorras 和 Vespignani 在 2001 年证明,在无标度网络上,这个比值会随网络变大而无限增长,所以阈值实际上消失了。下面是我用的三张 2,000 个智能体的网络:
| 网络 | ⟨k⟩ | κ = ⟨k²⟩/⟨k⟩ | 特点 |
|---|---|---|---|
| 随机网络(Erdős–Rényi) | 6.8 | 7.8 | 任何人和任何人通信的可能性都一样 |
| 无标度网络(Barabási–Albert) | 8.0 | 19.5 | 富者愈富的枢纽 |
| 智能体网格 | 6.8 | 18.7 | 团队、编排器、热门服务、共享记忆库 |
网格和随机网络的平均度数一样,κ 却是它的 2.4 倍。差别就在枢纽上。
第二,它有共享的蓄水池。 记忆库不是一个会生病、然后康复的节点。它更像霍乱模型里被污染的水源:病原体活在任何宿主之外,谁喝谁再被感染。这会改变数学,第 4 节会看到。
3. What viruses, prions and old DNA can teach us
Epidemiology is the obvious place to borrow from, but a plain virus model isn’t enough. Different pieces of the ripple map onto different biology. Every row here is a mechanism that’s actually in the simulation:
| Biology | Mechanism | In an agent network |
|---|---|---|
| Influenza, SIR models | Hosts are susceptible, infected or recovered. An outbreak grows if each case causes more than one new case on average (R₀ > 1). | An agent is clean, holds and repeats the error, or has caught it. Verification lowers the chance a message infects. |
| Superspreaders | A few individuals cause most transmissions. On scale-free contact networks the epidemic threshold vanishes. | Orchestrators and popular service agents. An error that starts in a shared search agent reaches hundreds of agents in one step. |
| Waning immunity (SIRS) | Recovered hosts lose immunity and can be reinfected. Diseases go endemic instead of burning out. | A correction lives in context. When the context resets, the agent is susceptible again. |
| HIV latent reservoir | Virus hides in resting cells where drugs can’t reach it, and comes back when therapy stops. | The error gets written to an agent’s persistent memory (I call this fossilised). Ordinary corrections don’t reach it, and it keeps resurfacing. |
| Cholera in water | An environmental reservoir keeps the pathogen alive outside hosts and re-infects new people. | Shared memory stores and retrieval indexes. Retrieval re-infects agents long after every agent has recovered. |
| Prions | A misfolded protein converts normal copies of itself. The immune system doesn’t respond, because it looks like self. | An error that matches the base model’s own priors. A peer running the same model checks it and approves it. |
| Antigenic drift | Small mutations let a virus escape antibodies made against older strains. | Each relay paraphrases the claim. The correction targets the original wording, and the paraphrase escapes. |
| Endogenous retroviruses | Ancient viral DNA inserted into the germline is inherited by every descendant. About 8% of the human genome is retroviral in origin. | Error text in training corpora. Once a base model learns it, every agent built on that model starts with it. |
| Eigen’s error threshold | If copying is too error-prone relative to selection, a population of replicators loses its information. | Model collapse: models trained on model output drift away from the original data distribution. |
The two I keep coming back to are the HIV reservoir and the retrovirus. The first is the fossil in one agent’s memory. The second is the error in everyone’s weights.
3. 病毒、朊病毒和古老的 DNA 能教我们什么
流行病学是最显然的借鉴对象,但光一个普通的病毒模型不够。涟漪的不同部分,对应着不同的生物学。下表每一行都是模拟里真实存在的机制:
| 生物学 | 机制 | 在智能体网络里 |
|---|---|---|
| 流感、SIR 模型 | 宿主分为易感、感染、康复三种状态。如果平均每个病例造成超过一个新病例(R₀ > 1),疫情就会扩大。 | 智能体要么干净,要么持有并转述这个错误,要么已经识破了它。核查会降低一条消息的传染概率。 |
| 超级传播者 | 少数个体造成了大部分传播。在无标度接触网络上,流行阈值会消失。 | 编排器和热门服务智能体。一个从共享搜索智能体开始的错误,一步就能传到几百个智能体。 |
| 免疫衰退(SIRS) | 康复的宿主会失去免疫力,可以再次感染。疾病会变成地方病,而不是烧完就结束。 | 纠正活在上下文里。上下文一重置,这个智能体又变回易感。 |
| HIV 潜伏库 | 病毒藏在药物够不着的静息细胞里,停药后卷土重来。 | 错误被写进了智能体的持久记忆(我管这叫固化)。普通的纠正够不着它,它会不断冒出来。 |
| 水中的霍乱 | 环境中的蓄水池让病原体在宿主之外存活,并感染新的人。 | 共享记忆库和检索索引。在所有智能体都康复很久之后,检索还会重新感染它们。 |
| 朊病毒 | 错误折叠的蛋白会把正常的同类蛋白也变成错误折叠。免疫系统不响应,因为它看起来就像自己。 | 一个和基座模型自身先验吻合的错误。跑同一个模型的同伴核查之后,会放行。 |
| 抗原漂变 | 小的突变让病毒逃过针对旧毒株产生的抗体。 | 每次转述都会改写一下说法。纠正针对的是原始措辞,改写后的版本就逃掉了。 |
| 内源性逆转录病毒 | 插入生殖细胞系的古老病毒 DNA 会被所有后代继承。人类基因组里大约 8% 源自逆转录病毒。 | 训练语料里的错误文本。一旦基座模型学会了它,基于这个模型的每个智能体生来就带着它。 |
| Eigen 错误阈值 | 如果复制的出错率相对于选择压力太高,一群复制子会丢失它们携带的信息。 | 模型崩溃:用模型输出训练出来的模型,会逐渐偏离原始数据分布。 |
我反复琢磨的是 HIV 潜伏库和逆转录病毒这两个。前者是一个智能体记忆里的化石,后者是所有人权重里的错误。
4. Five compartments and one ratchet
Each agent is in one of four states, and each org has a shared store. A clean agent (S) hears the claim from its neighbours. Every time it’s exposed, it checks the claim and catches the error with probability \(v\). If the check fails, it adopts the claim with probability \(\tau\) per message and becomes spreading (I). A spreading agent corrects itself at rate \(\gamma\), or writes the error into its own persistent memory at rate \(\phi\) and becomes fossilised (F). Only a deep audit, at rate \(\delta\), clears a fossil. A corrected agent (R) forgets the correction at rate \(w\). Spreading and fossilised agents both write the error into the shared store \(M\) at rate \(\omega\), the store gets cleaned at rate \(c\), and clean agents retrieve from it at rate \(\rho\).
One more mechanism, and it’s very agent-specific: agents treat agreement as evidence. If an agent has seen the same claim from \(n\) sources, its chance of catching the error drops to \(v / (1 + s(n-1))\). I call \(s\) social proof. When the “sources” are copies of each other, that’s citogenesis.
The mean-field version treats every contact as random and boils down to five equations:
\[\begin{aligned} \dot S &= -\lambda S + wR \\ \dot I &= \lambda S - (\gamma + \phi)\, I \\ \dot F &= \phi I - \delta F \\ \dot R &= \gamma I + \delta F - wR \\ \dot M &= \omega\,(I + F)(1 - M) - cM \end{aligned}\]with force of infection \(\lambda = \beta\kappa\,(I+F) + \rho(1-v)\,M\), per-contact rate \(\beta = p\,\tau\,(1-v)\), and \(p\) the chance an edge carries a message in a given step.
Now the reproduction number. One new spreading agent infects others directly at rate \(\beta\kappa\) for an average time \(1/(\gamma+\phi)\). It also dumps \(\omega/(\gamma+\phi)\) units of contamination into the store, and each unit infects at rate \(\rho(1-v)\) for time \(1/c\). And with probability \(\phi/(\gamma+\phi)\) it becomes a fossil and does all of that again, for an average time \(1/\delta\). Put it together:
\[\begin{aligned} R_0 &= \underbrace{\Big[\,\beta\kappa + \frac{\rho(1-v)\,\omega}{c}\,\Big]}_{\text{network}\ +\ \text{store}} \\ &\quad \cdot \underbrace{\Big(1 + \frac{\phi}{\delta}\Big)}_{\text{persistence}} \cdot \frac{1}{\gamma + \phi} \end{aligned}\]Three factors, three ways to fight a ripple: make messages less infectious (verification, fewer hubs), keep the stores clean (curation), and make memory auditable. The third one is where it gets interesting.
If persistent memory is never audited (δ = 0) and some errors get written into it (φ > 0), R₀ is infinite. There’s no epidemic threshold at all.
Here’s why. A fossil never leaves the infectious pool. Corrected agents eventually forget, go back to susceptible, and run into the fossil again. Every encounter has some small chance of minting a new fossil. So the fossil count can only go up, and the only stable state is one where everyone reachable is fossilised. The other parameters decide how fast. They don’t decide whether. Drag δ to zero below and watch R₀ flip to ∞.
4. 五个仓室和一个棘轮
每个智能体处在四种状态之一,每个组织有一个共享记忆库。一个干净的智能体(S)从邻居那里听到这条说法。每次接触,它都会核查一下,以概率 \(v\) 识破错误。如果没识破,它每收到一条消息就以概率 \(\tau\) 采信,变成传播中(I)。传播中的智能体以速率 \(\gamma\) 自我纠正,或者以速率 \(\phi\) 把错误写进自己的持久记忆,变成已固化(F)。只有深度审计(速率 \(\delta\))能清掉一块化石。已纠正的智能体(R)以速率 \(w\) 忘掉纠正。传播中和已固化的智能体都会以速率 \(\omega\) 把错误写进共享记忆库 \(M\),记忆库以速率 \(c\) 被清理,干净的智能体以速率 \(\rho\) 从中检索。
还有一个机制,而且是智能体特有的:智能体会把”大家都这么说”当成证据。 如果一个智能体已经从 \(n\) 个来源看到同一条说法,它识破错误的概率就降到 \(v / (1 + s(n-1))\)。我管 \(s\) 叫社会认同。当这些”来源”其实是彼此的副本时,这就是引用生成。
平均场版本把每次接触都当成随机的,最后归结为五个方程:
\[\begin{aligned} \dot S &= -\lambda S + wR \\ \dot I &= \lambda S - (\gamma + \phi)\, I \\ \dot F &= \phi I - \delta F \\ \dot R &= \gamma I + \delta F - wR \\ \dot M &= \omega\,(I + F)(1 - M) - cM \end{aligned}\]其中感染力 \(\lambda = \beta\kappa\,(I+F) + \rho(1-v)\,M\),单次接触传染率 \(\beta = p\,\tau\,(1-v)\),\(p\) 是一条边在某一步里传一条消息的概率。
再看基本再生数。一个新的传播中智能体,以速率 \(\beta\kappa\) 直接感染别人,平均持续 \(1/(\gamma+\phi)\) 的时间。它还会往记忆库里倒 \(\omega/(\gamma+\phi)\) 份污染,每份污染以速率 \(\rho(1-v)\) 感染别人,持续 \(1/c\) 的时间。另外,它有 \(\phi/(\gamma+\phi)\) 的概率变成化石,然后把上面这些再做一遍,平均持续 \(1/\delta\)。合起来:
\[\begin{aligned} R_0 &= \underbrace{\Big[\,\beta\kappa + \frac{\rho(1-v)\,\omega}{c}\,\Big]}_{\text{网络}\ +\ \text{记忆库}} \\ &\quad \cdot \underbrace{\Big(1 + \frac{\phi}{\delta}\Big)}_{\text{持久化}} \cdot \frac{1}{\gamma + \phi} \end{aligned}\]三个因子,对应三种对付涟漪的办法:降低消息的传染性(核查、少点枢纽),保持记忆库干净(清理),以及让记忆可审计。第三个才是有意思的地方。
如果持久记忆从来不审计(δ = 0),又有一些错误被写了进去(φ > 0),R₀ 就是无穷大。根本不存在流行阈值。
原因是这样的。化石永远不会离开传染源的池子。已纠正的智能体迟早会忘,变回易感,然后再碰上那块化石。每次相遇都有一点点概率造出一块新化石。所以化石只增不减,唯一的稳态就是所有够得着的智能体全部固化。其他参数决定这件事发生得多快,但不决定它会不会发生。在下面把 δ 拖到零,看 R₀ 变成 ∞。
5. Run your own outbreak
That’s the smooth version. Below is the real thing: the agent-based model, every agent simulated individually on an actual network, same code I used for all the 2,000-agent experiments in the next section. Tap any agent to plant an error. Try issuing a correction early, then late, and compare the three kinds: broadcast (agents hear it, then forget), durable (saved to memory), and deep (allowed to rewrite fossilised memories).
Some things worth trying: set deep audit δ to 0 and just let it run. Turn on the blind spot with one model family, then bump it to four. Crank drift μ to 0.05 and then issue a correction.
5. 自己跑一次爆发
上面是光滑的版本。下面是真家伙:个体模拟模型,每个智能体都在一张真实的网络上单独模拟,和我在下一节做 2,000 个智能体实验时用的是同一份代码。点任意一个智能体,就在那里种下一个错误。试试早一点发布纠正,再试试晚一点,然后比较三种纠正:广播(智能体听到了,然后忘了)、持久(写进记忆)、深度(允许改写已固化的记忆)。
几个值得试的玩法:把深度审计 δ 设成 0,然后放着让它跑。打开模型盲区、只留一个模型家族,再加到四个。把漂变 μ 拉到 0.05,然后发布纠正。
6. What 3,000 simulated outbreaks say
For the experiments I ran the same engine headless on 2,000-agent networks, 16 to 60 runs per setting. Baseline parameters are in the appendix. Unless I say otherwise, the network is the agentic mesh and the error starts at the busiest service agent. Two words I’ll use a lot: reach is the share of agents that ever held the error, prevalence is the share holding it at the end.
6.1 Where an error starts matters as much as how infectious it is
Same error, same parameters. Seed it at a random worker and it dies out on its own 30–40% of the time: the worker’s team catches it before it gets out. Seed it at a hub and the fizzle rate drops to 12.5% on the random network, and to zero on the scale-free network and the mesh.
6. 3,000 次模拟爆发告诉我们什么
做实验时,我把同一个引擎放在 2,000 个智能体的网络上无界面地跑,每种设置跑 16 到 60 次。基线参数在附录里。除非特别说明,网络都是智能体网格,错误都从最繁忙的服务智能体开始。有两个词我会一直用:波及面是曾经持有过这个错误的智能体占比,流行率是最后还持有它的占比。
6.1 错误从哪儿开始,和它有多能传染一样重要
同一个错误,同一组参数。从随机一个执行者开始,30–40% 的情况下它会自己灭掉:执行者所在的团队在它扩散出去之前就把它拦下了。从枢纽开始,随机网络上的自行消亡率降到 12.5%,无标度网络和智能体网格上直接降到零。
That one’s uncomfortable, because in a real network the agents you trust most (popular tools, orchestrators) are exactly where an error does the most damage.
6.2 Verification has a threshold, and social proof moves it
Reach falls slowly as verification improves, then drops off a cliff near a threshold. On the random network, agents need to catch about 60% of exposures to keep reach under 10%. The hub-heavy networks need about 75%.
Then add social proof. With independent checking (\(s = 0\)), a verifier at \(v = 0.5\) holds reach to 28%. With \(s = 2\), where every repeat of the claim makes the agent trust it more, the same verifier lets it reach 76%. Corroboration from sources that copied each other isn’t corroboration.
这条让人不太舒服,因为在真实的网络里,你最信任的那些智能体(热门工具、编排器),恰恰是错误造成破坏最大的地方。
6.2 核查有一个阈值,而社会认同会把阈值推走
随着核查变强,波及面先是慢慢下降,到了某个阈值附近突然断崖式下跌。在随机网络上,智能体得识破大约 60% 的接触,才能把波及面压到 10% 以下。以枢纽为主的网络需要大约 75%。
再加上社会认同。独立核查时(\(s = 0\)),\(v = 0.5\) 的核查能把波及面压在 28%。当 \(s = 2\),也就是说法每重复一次智能体就更信一分时,同样的核查会让它波及到 76%。来自互相抄来抄去的来源的佐证,不算佐证。
6.3 Unaudited memory is a one-way ratchet
This is the result I built the model to test. With no deep audit (\(\delta = 0\)), the fossilised share climbs almost in a straight line and is still climbing at step 1,500, by which point 48% of all agents have the error baked into memory. Audit just 0.2% of fossils per step and the curve bends toward a plateau around 15%. At 1% per step the plateau is 2.6%.
Notice the plateaus aren’t zero. Partial auditing makes the error endemic, not gone.
6.3 没人审计的记忆,是一个只能单向转的棘轮
这就是我搭这个模型想要验证的结果。没有深度审计时(\(\delta = 0\)),固化比例几乎是一条直线往上爬,到第 1,500 步还在涨,那时全体智能体里已经有 48% 把这个错误刻进了记忆。每一步只要审计 0.2% 的化石,曲线就会弯向一个 15% 左右的平台。每一步审计 1%,平台降到 2.6%。
注意,平台不是零。不彻底的审计只会让错误变成地方病,而不是消失。
6.4 Monoculture switches verification off
Here verification is strong (0.6), but the error comes from a blind spot in one base model, and agents running that model keep only 10% of their checking ability for this particular claim. That’s the prion case: the error looks like the model’s own knowledge. When every agent runs the same model, the error reaches 99% of them. Two model families: 77%. Six: 48%. Diversity helps, but slowly, because the blind agents still pass it to each other.
6.4 单一栽培会把核查关掉
这里核查很强(0.6),但错误来自某个基座模型的盲区,跑那个模型的智能体面对这条说法时只剩 10% 的核查能力。这就是朊病毒的情形:这个错误看上去就像模型自己的知识。当所有智能体都跑同一个模型时,错误波及 99% 的智能体。两个模型家族:77%。六个:48%。多样性有用,但见效很慢,因为那些”看不见”它的智能体还在互相传。
6.5 Corrections fade unless you write them down
A correction is a contagion too. In the model it starts from a few percent of agents plus every orchestrator, and spreads from there. A broadcast correction issued at step 60 cuts prevalence from 6.6% to 4.5% within 30 steps. Looks like a win. Then the error comes back: corrected agents forget, the fossils keep talking, and by step 600 prevalence is 13.5%, double what it was when the correction went out.
A durable correction, which agents save to memory, holds the line at 7%. A deep correction, which can rewrite fossilised memories, ends at 4.6% and is still creeping up, because the fossils it missed keep working.
And timing only matters once the correction can last. A broadcast lands in the same place whether you send it at step 10 or step 300. A durable correction at step 10 leaves 1.9% fossilised; at step 300, 8.3%.
6.5 纠正会褪色,除非你把它写下来
纠正本身也是一种传染。模型里,纠正从一小部分智能体加上所有编排器出发,再往外传。第 60 步发出一次广播式纠正,30 步之内流行率从 6.6% 降到 4.5%。看起来赢了。然后错误回来了:已纠正的智能体会忘,化石还在不停地说,到第 600 步流行率是 13.5%,是发出纠正那一刻的两倍。
持久纠正(智能体把它存进记忆)能守在 7%。深度纠正(能改写已固化的记忆)最后停在 4.6%,而且还在慢慢往上爬,因为它漏掉的那些化石还在干活。
另外,只有当纠正能留得住,时机才有意义。广播式纠正不管在第 10 步还是第 300 步发,结局都差不多。持久纠正在第 10 步发,最后剩 1.9% 固化;第 300 步发,剩 8.3%。
6.6 Paraphrase lets errors escape
Here every relay has a small chance of rewording the claim into a new variant. An agent that caught one variant, or got corrected on it, isn’t protected against another, and a correction only covers the variants that existed when it went out. No drift: a correction at step 40 holds reach to 84% and prevalence to 12%. Just 1% drift per relay: reach 100%, prevalence 55%.
That’s antigenic drift. The lesson: corrections need to target the underlying claim, not its wording.
6.6 转述让错误逃掉
这里每次转述都有一点点概率把说法改写成一个新变体。一个识破了某个变体、或者在某个变体上被纠正过的智能体,对另一个变体并没有免疫力,而且一次纠正只覆盖它发出时已经存在的变体。没有漂变时:第 40 步的纠正能把波及面压在 84%、流行率压在 12%。每次转述只要有 1% 的漂变:波及面 100%,流行率 55%。
这就是抗原漂变。教训是:纠正要针对背后的那个说法本身,而不是它的某种措辞。
6.7 No single defence is enough. The combination works.
I compared five interventions against doing nothing. Stronger verification at hubs, memory-store curation and an early broadcast correction each help a bit, then lose ground. Provenance tagging is the best single one: if every claim carries the chain of agents it passed through, an agent can see that ten confirmations are one source repeated ten times, and trace the claim back to check it. (I modelled that as social proof \(s = 0\) plus 0.15 on verification.) It cuts final prevalence from 13.7% to 4.4%. All five together keep prevalence under 0.5%, and in 20% of runs the error dies out completely.
6.7 没有哪一种防御单独够用,组合起来才行
我拿五种干预和”什么都不做”做了比较。枢纽加强核查、记忆库清理、早期广播纠正,各自都有一点用,然后慢慢失守。来源标注是单项里最好的:如果每条说法都带着它经过的智能体链条,智能体就能看出十次确认其实是同一个来源重复了十遍,还能顺着链条回去核查。(我把它建模成社会认同 \(s = 0\),再加 0.15 的核查率。)它把最终流行率从 13.7% 降到 4.4%。五种叠加在一起,流行率压在 0.5% 以下,而且 20% 的运行里错误彻底灭绝。
7. The threshold you can’t see from inside one generation
Everything so far happens inside one generation of models. The last step of the ripple is the training loop.
Say a fraction \(e\) of what a model believes about some topic is wrong. Agents running that model write text, and the network amplifies each seeded error into a cascade. Even below its epidemic threshold, a network still turns one seeded error into a finite cascade. I measured the average cascade size \(A\) on the 2,000-agent mesh: about 46 agents per seed at \(v = 0.6\), 14.5 at 0.7, and 3.3 at 0.8.
The next generation trains on a mix: a share \(h\) of human-written or verified data, and the rest from agent output, where a share \(q\) of the errors get filtered out. Add a trickle \(\varepsilon\) of fresh errors each generation. For small \(e\):
\[\begin{gathered} e_{g+1} \approx (1-h)(1-q)\,A\,e_g + (1-h)\,\varepsilon \\ R_g = (1-h)(1-q)\,A \end{gathered}\]If \(R_g < 1\), each generation’s errors fade over the next few. If \(R_g > 1\), they compound up to a stable, nonzero level that no later generation removes, like a retrovirus fixed in the germline.
Here’s the uncomfortable part. A network can sit safely below its own epidemic threshold, so every outbreak looks contained, while the training loop is above its threshold. Take a well-verified network (\(A \approx 14.5\)), keep 40% human data, and filter out 80% of the errors. That still gives \(R_g = 0.6 \times 0.2 \times 14.5 \approx 1.7\). The ripple becomes permanent across generations even though no single generation ever has an epidemic.
7. 在一代模型内部看不见的那个阈值
到目前为止的一切,都发生在同一代模型里。涟漪的最后一步,是训练闭环。
假设一个模型在某个话题上的认知里,有比例 \(e\) 是错的。跑这个模型的智能体会写出文本,网络会把每个种下的错误放大成一串级联。哪怕网络低于它的流行阈值,一个种下的错误也还是会变成一串有限的级联。我在 2,000 个智能体的网格上测了平均级联规模 \(A\):\(v = 0.6\) 时每个种子大约 46 个智能体,0.7 时 14.5 个,0.8 时 3.3 个。
下一代模型用混合数据训练:比例 \(h\) 是人类写的或经过验证的数据,剩下的来自智能体输出,其中比例 \(q\) 的错误被过滤掉。每一代还会冒出一点新错误 \(\varepsilon\)。当 \(e\) 很小时:
\[\begin{gathered} e_{g+1} \approx (1-h)(1-q)\,A\,e_g + (1-h)\,\varepsilon \\ R_g = (1-h)(1-q)\,A \end{gathered}\]如果 \(R_g < 1\),每一代的错误会在接下来几代里慢慢淡掉。如果 \(R_g > 1\),它们会不断累积,直到一个稳定的、非零的水平,之后任何一代都去不掉,就像一个固定在生殖细胞系里的逆转录病毒。
让人不舒服的地方在这儿。一张网络完全可以稳稳地待在它自己的流行阈值之下,每次爆发看上去都控制住了,而训练闭环却在它自己的阈值之上。拿一个核查做得不错的网络(\(A \approx 14.5\)),保留 40% 的人类数据,过滤掉 80% 的错误。算出来还是 \(R_g = 0.6 \times 0.2 \times 14.5 \approx 1.7\)。哪一代都没出现过流行病,涟漪却在代际之间变成了永久的。
8. Building an immune system for agent networks
The sims point to seven design rules. Each one goes after a factor in \(R_0\) or \(R_g\).
- Carry provenance with every claim. The most effective single intervention in my runs. Provenance lets an agent tell independent confirmation from repetition, which kills social proof, and lets it trace a claim back to its source. It’s the agent version of a citation that points to evidence instead of to another citation.
- Treat persistent memory as a reservoir that needs audits. Any δ above zero turns a permanent error into a bounded one. Memory entries should carry a source and a date, expire unless re-confirmed, and get sampled for audits at a fixed rate. Auditing 1% of fossils per step took the fossilised share from 48% to under 3%.
- Make corrections durable, and let them reach memory. A correction that only lives in context fades, and the error comes back. Write corrections where the error was written: memory stores, caches, retrieval indexes. And go after the claim, not one wording of it.
- Verify hardest where the network is densest. Hubs decide whether an error fizzles. Popular services and orchestrators should check harder than workers and be first in line for corrections. Hub audits alone weren’t enough in my runs, but where the error started was the difference between a 40% and a 0% chance it died out on its own.
- Keep the model population diverse. Checking with a copy of the same model fails exactly on that model’s blind spots. Cross-checking with a different model family is the cheapest herd immunity there is.
- Keep a clean germline. \(R_g\) is set by the human-anchored share of training data, filter quality, and network amplification. All three need to be measured, not assumed. A corpus of verified human-written text is to future models what a seed bank is to agriculture.
- Measure your network like an epidemiologist would. Plant known-false canary claims and track how far they travel. Fizzle rate, time to reach 10% of agents, cascade size: all observable, and together they give you a direct estimate of \(R_0\) for a real system.
8. 给智能体网络建一套免疫系统
模拟结果指向七条设计原则。每一条都针对 \(R_0\) 或 \(R_g\) 里的某个因子。
- 每条说法都带上来源。 我跑的所有实验里效果最好的单项干预。有了来源,智能体就能分清独立的确认和单纯的重复,社会认同就失效了,它还能顺着链条找到源头。这相当于智能体版的”引用要指向证据,而不是指向另一个引用”。
- 把持久记忆当成一个需要审计的蓄水池。 只要 δ 大于零,永久的错误就会变成有上限的错误。记忆条目应该带上来源和日期,不重新确认就过期,并且按固定比例抽样审计。每一步审计 1% 的化石,固化比例就从 48% 降到 3% 以下。
- 让纠正持久,并且能够到记忆。 只活在上下文里的纠正会褪色,错误会回来。错误写在哪里,纠正就要写到哪里:记忆库、缓存、检索索引。而且要针对说法本身,而不是它的某一种措辞。
- 网络最密的地方,核查最严。 枢纽决定了一个错误会不会自己灭掉。热门服务和编排器应该比执行者查得更严,也应该最先收到纠正。在我的实验里,光靠枢纽加强核查还不够,但错误从哪里开始,决定了它自行消亡的概率是 40% 还是 0%。
- 保持模型种群的多样性。 拿同一个模型的副本去核查,恰好会在这个模型的盲区上失灵。换一个模型家族交叉核查,是最便宜的群体免疫。
- 守住一条干净的生殖细胞系。 \(R_g\) 由三样东西决定:训练数据里人类锚定数据的比例、过滤质量、网络放大倍数。三样都要实测,不能靠假设。一份经过验证的人类文本语料,对未来的模型来说,就像种子库对农业一样。
- 像流行病学家一样测量你的网络。 种下已知为假的金丝雀说法,追踪它们能传多远。自行消亡率、波及 10% 智能体所需的时间、级联规模,这些都是能观测的数字,合起来就能直接估出一个真实系统的 \(R_0\)。
9. What this model leaves out
This is a stylised model. The parameters are plausible guesses, not measurements of any deployed system, so don’t read the absolute percentages as forecasts. The qualitative results are the part I trust: the vanishing threshold on hub-heavy networks, the ratchet when memory is never audited, the rebound after corrections that don’t stick, and the hidden generational threshold all come from the structure of the equations, not from particular numbers.
And some things are just missing. Real errors differ in how plausible they are, and plausibility probably correlates with how hard they are to catch. Agents act on what they believe, so errors have downstream consequences (a wrong config deployed, a wrong payment sent) that can expose them and work as a natural correction. There are no humans in the loop. The network is static, while real agent networks rewire as tasks come and go. And the generational model is a linearised map, not a simulation of training. Each of those is a direction for better work, and the engine is small enough to extend: it’s one file, engine.js.
And if you want the version of this with the numbers turned into a story, about what happens when the ripple outgrows the people watching it, that’s Part II: The Invisible Host.
9. 这个模型没考虑什么
这是一个风格化的模型。参数是看上去合理的猜测,不是对任何已部署系统的实测,所以别把那些绝对百分比当预测。我真正信的是定性结论:以枢纽为主的网络上阈值消失、记忆从不审计时的棘轮、纠正留不住之后的反弹,以及隐藏的代际阈值,这些都来自方程的结构,而不是某组具体数字。
还有些东西干脆没建模。真实的错误可信程度各不相同,可信度大概和它有多难被识破相关。智能体会根据自己的认知行动,所以错误会产生下游后果(部署了错的配置、打错了款),这些后果可能会暴露错误,起到天然纠正的作用。回路里没有人。网络是静态的,而真实的智能体网络会随着任务来来去去不断重连。代际模型是一个线性化的映射,不是对训练过程的模拟。这些都是可以做得更好的方向,而且引擎小到很容易扩展:就一个文件,engine.js。
如果你想看这些数字被写成一个故事,讲涟漪大到超出看守它的人之后会发生什么,那是下半篇:《隐形宿主》。
Appendix: method and parameters
The engine is a discrete-time stochastic agent-based model. Each step, every spreading or fossilised agent sends the claim along each of its edges with probability \(p\). Clean agents count exposures, plus one more from a retrieval hit with probability \(\rho M\). A clean agent with \(n\) exposures this step and \(m\) total copies seen catches the error, or failing that adopts the claim, with these probabilities, and each store updates as shown:
\[\begin{gathered} P_{\text{catch}} = \frac{v}{1 + s(m-1)} \\ P_{\text{adopt}} = 1 - (1-\tau)^n \\ M \leftarrow M + \frac{\text{writes}}{\text{users}}(1-M) - cM \end{gathered}\]Networks have 2,000 agents. The mesh has teams of 40, an orchestrator per team, orgs of four teams, and 20 service agents. Random networks use a different seed every run.
| Symbol | Meaning | Baseline |
|---|---|---|
| p | Chance an edge carries a message per step | 0.3 |
| τ | Adoption chance per unverified message | 0.22 |
| v | Verification (chance to catch the error) | 0.3 |
| s | Social proof | 0.5 |
| ρ | Retrieval from the shared store per step | 0.15 |
| γ | Self-correction rate | 0.1 |
| φ | Rate of writing the error to persistent memory | 0.01 |
| δ | Deep-audit rate for persistent memory | 0.002 |
| ω | Rate of writing the error to the shared store | 0.03 |
| c | Store curation rate | 0.02 |
| w | Rate of forgetting a correction | 0.01 |
| β_C | Chance a correction message lands | 0.3 |
| — | Correction broadcasters at issue | 2% of agents + all orchestrators |
Each experiment changes one or two of these from the baseline; the figure headers say which, and how many runs went into each line.
附录:方法和参数
引擎是一个离散时间的随机个体模型。每一步,每个传播中或已固化的智能体沿它的每条边以概率 \(p\) 发送这条说法。干净的智能体统计自己受到的接触次数,另外以概率 \(\rho M\) 从检索中再多一次。一个本步受到 \(n\) 次接触、累计看到 \(m\) 份副本的干净智能体,按下面的概率识破错误,识破不了就按下面的概率采信;每个记忆库按下式更新:
\[\begin{gathered} P_{\text{识破}} = \frac{v}{1 + s(m-1)} \\ P_{\text{采信}} = 1 - (1-\tau)^n \\ M \leftarrow M + \frac{\text{写入次数}}{\text{用户数}}(1-M) - cM \end{gathered}\]网络有 2,000 个智能体。网格由 40 人的团队组成,每个团队一个编排器,每四个团队一个组织,外加 20 个服务智能体。随机网络每次运行换一个随机种子。
| 符号 | 含义 | 基线值 |
|---|---|---|
| p | 一条边每步传一条消息的概率 | 0.3 |
| τ | 每条未识破的消息被采信的概率 | 0.22 |
| v | 核查率(识破错误的概率) | 0.3 |
| s | 社会认同 | 0.5 |
| ρ | 每步从共享记忆库检索的概率 | 0.15 |
| γ | 自我纠正速率 | 0.1 |
| φ | 把错误写进持久记忆的速率 | 0.01 |
| δ | 持久记忆的深度审计速率 | 0.002 |
| ω | 把错误写进共享记忆库的速率 | 0.03 |
| c | 记忆库清理速率 | 0.02 |
| w | 忘掉纠正的速率 | 0.01 |
| β_C | 一条纠正消息生效的概率 | 0.3 |
| — | 纠正发布时的初始传播者 | 2% 的智能体 + 全部编排器 |
每个实验只在基线上改动其中一两个参数;改了哪个、每条线跑了多少次,图的标题里都写着。
Sources
来源
- Kermack, W. O. & McKendrick, A. G. (1927). A contribution to the mathematical theory of epidemics. Proc. R. Soc. A.
- Pastor-Satorras, R. & Vespignani, A. (2001). Epidemic spreading in scale-free networks. Physical Review Letters.
- Codeço, C. T. (2001). Endemic and epidemic dynamics of cholera: the role of the aquatic reservoir. BMC Infectious Diseases.
- Centola, D. (2010). The spread of behavior in an online social network experiment. Science.
- Greenberg, S. A. (2009). How citation distortions create unfounded authority: analysis of a citation network. BMJ.
- Eigen, M. (1971). Selforganization of matter and the evolution of biological macromolecules. Naturwissenschaften.
- International Human Genome Sequencing Consortium (2001). Initial sequencing and analysis of the human genome. Nature.
- Shumailov, I. et al. (2024). AI models collapse when trained on recursively generated data. Nature.
- Gu, X. et al. (2024). Agent Smith: a single image can jailbreak one million multimodal LLM agents exponentially fast. ICML.
- Lee, D. & Tiwari, M. (2024). Prompt Infection: LLM-to-LLM prompt injection within multi-agent systems. arXiv.