Under 100 ms: Breadth, Depth, and the Steerable Wheel
Three days ago OpenAI folded its robotics team into the rest of the company. Per Fortune’s report on November 13, the reasoning is that a general model can now drive a robot directly (The Decoder). OpenAI says robotics work continues, just closer to research and product. There’s a number behind the move. On NVIDIA’s RoboLab-120, a GPT-6 Astra agent with no robot training finished 74.8% of the tasks. The best dedicated robot model on that leaderboard got 51.0%. π0.5 got 28.0% (NVIDIA).
A month ago in The Data Wall I wrote that general intelligence eats robotics. I didn’t expect it to take this big a bite this fast.
Now read the setup section of that same study. One sentence: “The simulator is paused during model inference.” Each call takes 13 to 19 seconds. The world waits. Put the same model on two real arms, where the world doesn’t wait, and it fully completes 7 of 100 attempts (Robocurve).
Both numbers are true, and the gap between them is this whole post. Here’s my bet in one line: anything that can wait 100 milliseconds belongs to the driver now. Whatever can’t wait is the wheel, and the wheel is what’s left to build. Four claims:
- The 100 ms line is physics, not a benchmark gap. A smarter model doesn’t make a mug fall slower.
- What lives under the line is atomic behaviour, and its one job is to be steerable. The driver steers it with words, points, traces and pictures of the next state. That’s reading a manual.
- Steerability is bought with depth, not breadth. Depth has a clean definition: how many different things were done from the same state. Most robot data has a depth of about one.
- A deep wheel plus a smart driver beats buying nines. What compounds over a long task isn’t failure. It’s failure you can’t recover from.
Start with a mug.
三天前,OpenAI 把机器人团队拆进了公司的其他部门。 按《财富》11 月 13 日的报道,理由是通用模型现在可以直接开机器人了(The Decoder)。OpenAI 自己的说法是机器人还在做,只是离研究和产品更近了。这个动作背后有一个数字:在 NVIDIA 的 RoboLab-120 上,一个没受过任何机器人训练的 GPT-6 Astra 智能体做成了 74.8% 的任务;榜单上最好的专用机器人模型是 51.0%,π0.5 是 28.0%(NVIDIA)。
一个月前我在《数据墙》里写,通用智能会吃掉机器人。我没想到它这一口咬得这么大、这么快。
现在去读同一份研究的实验设置,有一句话:「模型推理期间,仿真器是暂停的。」 每次调用要 13 到 19 秒,世界就停在那儿等它。把同一个模型放到两条真实的机械臂上,世界不等了,它 100 次尝试里完整做成的是 7 次(Robocurve)。
两个数字都是真的,这篇文章讲的就是它们之间的那道缝。我的判断一句话:凡是等得起 100 毫秒的事,现在都归司机管。等不起的那部分是轮子,而轮子就是剩下要造的东西。 四个论点:
- 100 毫秒这条线是物理,不是榜单差距。 模型再聪明,杯子也不会掉得慢一点。
- 线下面的东西是原子行为,它唯一的任务就是听话。 司机用文字、点、轨迹和「下一个状态的图」来指挥它,这就是在读说明书。
- 可操控性是用深度买的,不是用广度。 深度有一个很干净的定义:从同一个状态出发,做过多少种不同的事。今天大部分机器人数据的深度大约是 1。
- 一只「深」轮子加一个聪明司机,比去买更多个 9 划算。 长任务里复利的不是失败,是救不回来的失败。
先从一只杯子说起。
1. The line at 100 ms
A mug that starts to slip falls $d = \tfrac{1}{2} g t^2$. That’s the worst case, real slips start slower, but it sets the scale. If the fingers overlap the mug by 5 cm, the grip is gone at
\[t = \sqrt{2 \times 0.05 / 9.81} = 101 \text{ ms}\]A human hand starts correcting grip force about 74 ms after a slip begins. That’s 2.7 cm, and you never notice it happened. Astra’s direct action call averages 18.56 s. In 18.56 seconds a dropped mug falls 1.7 kilometres. So the model isn’t correcting the slip. It’s reading about it afterwards.
Here’s the whole ladder, one decision per row:
1. 100 毫秒这条线
一只开始打滑的杯子,下落距离是 $d = \tfrac{1}{2} g t^2$。这是最坏情况,真实的打滑一开始会慢一些,但量级就是这个。如果手指和杯子有 5 cm 的重叠,那这一抓在
\[t = \sqrt{2 \times 0.05 / 9.81} = 101 \text{ 毫秒}\]之后就没了。人手在打滑开始后大约 74 毫秒就开始修正握力,杯子只掉了 2.7 cm,你自己都不会察觉。Astra 直接出动作的一次调用平均 18.56 秒。18.56 秒里,一只脱手的杯子能掉 1.7 公里。所以模型不是在纠正打滑,它是事后在读关于这次打滑的报道。
整张阶梯在这儿,每行是一次决策:
Two objections, both fair.
“The driver will get faster.” It will. A paper from last week trains a 27B open-weight reasoning model to control robots natively, and gets it decoding at 747 tokens a second on one GB300 (Tang et al.). One move still takes 1.1 s at low effort and 4.3 s at xhigh, because it thinks for 860 to 3,230 tokens first. Do the division the other way: at 747 tokens a second, 100 ms buys you 75 tokens. That’s not reasoning any more. The thing that makes a driver a driver is that it thinks before it acts, and thinking is exactly what doesn’t fit.
“The line isn’t sharp.” Right, and the wheels themselves show it. PI’s real-time chunking paper lists the budget for one π0.5 call on a mobile robot: 97 ms in the model, 21 on the network, 11 resizing images, 10 for the rest. That’s 139 ms, and they make it work by committing to a chunk of actions and training the model to tolerate the delay (RTC). So today’s best wheels sit right on the line. Read “100 ms” as an order of magnitude.
There’s a money version of the same line. The agentic setup that scored 74.8% burned 30.9M tokens, about \$36, per episode. It’s a superb way to decide what to do. It’s a terrible way to hold a mug.
A survey from the same week calls the two layers System 2 and System 1 and measures the gap: Astra used directly is 40 to 8,000 times slower per decision than the control skills beneath it (Grid Robotics). I’ll call them the driver and the wheels, because that’s the relationship. Wheels don’t know where the car is going, and they don’t need to. They need to do exactly what the steering column says, right now.
两个反驳,都有道理。
「司机会变快的。」 会。上周有一篇论文,把一个 27B 的开源推理模型训到能原生控制机器人,在一张 GB300 上解码速度做到每秒 747 个 token(Tang 等)。但它走一步,低推理强度下还是要 1.1 秒,超高强度下 4.3 秒,因为它动手之前要先想 860 到 3,230 个 token。把除法反过来做:每秒 747 个 token,100 毫秒只够 75 个 token。那已经不叫推理了。司机之所以是司机,就在于它先想再动,而「想」恰恰是塞不进去的那部分。
「这条线没那么清楚。」 对,轮子自己就说明了这一点。PI 的实时动作分块论文列过移动机器人上一次 π0.5 调用的时间账:模型 97 毫秒,网络 21 毫秒,缩放图像 11 毫秒,其他 10 毫秒,一共 139 毫秒。他们的办法是一次承诺一整段动作,并且在训练时就让模型学会容忍这段延迟(RTC)。所以今天最好的轮子也正好压在线上。「100 毫秒」请当成一个数量级来读。
这条线还有一个钱的版本。拿到 74.8% 的那套智能体方案,每局要烧 3,090 万个 token,大约 36 美元。用它来决定做什么非常好,用它来握住一只杯子非常糟。
同一周的一篇综述把这两层叫 System 2 和 System 1,并且量了一下差距:直接拿 Astra 来用,每次决策比它下面的控制技能慢 40 到 8,000 倍(Grid Robotics)。我管它们叫司机和轮子,因为它们就是这个关系。轮子不知道车要去哪,也不需要知道。它只需要方向盘说什么就做什么,而且是现在就做。
2. What the driver says: reading a manual
So what does a steering column for a robot look like? Think about how you assemble furniture you’ve never seen. You don’t get a policy. You get a manual, and a manual page has three kinds of ink:
- Words. “Attach the leg.” Names the step.
- Arrows and circles. Where to touch, which way to move. In robot terms: a point, or a trace of pixel coordinates for the gripper to follow.
- A picture of what it looks like after. The next state.
Your hands have never built this shelf, and they don’t have to have. They’re steerable: they can turn any page into motion. That’s the whole trick, and the driver models can now write all three inks. Astra is reported to be a step change in spatial reasoning, so it can put the point in the right place. World models can generate the next-state picture.
The robot side has started to line up with this, fast:
- Steerable Policies (Berkeley, Stanford and PI, February) trains one VLA to take commands at six levels, from task and subtask down to atomic motions (“move left”, “open gripper”), gripper traces, points, and mixes of them. The line from that paper I keep rereading: a human issuing unrestricted steering commands “can solve all tasks at nearly 100% success rate” (Chen et al.). The hands were never the bottleneck. The interface was.
- π0.7 (PI, April) builds its prompt out of subtask language, subgoal images from a small world model, and metadata about how the episode should go. Seen tasks often pass 90%; unseen tasks land at 60 to 80% (π0.7).
Here’s one manual page, live. Same scene, same starting pixels, four different commands. Toggle the inks.
2. 司机怎么说话:读说明书
那机器人的方向盘长什么样?想想你是怎么装一件从没见过的家具的。没人给你一个策略,给你的是一本说明书,而说明书的每一页有三种墨水:
- 文字。「装上桌腿。」告诉你这一步叫什么。
- 箭头和圆圈。 碰哪里,往哪边动。换成机器人的话:一个点,或者一串让夹爪跟着走的像素坐标轨迹。
- 一张做完之后的图。 也就是下一个状态。
你的手从没装过这个架子,也不需要装过。它们是可操控的:随便哪一页,都能变成动作。全部的诀窍就在这儿,而司机模型现在三种墨水都会写了。有报道说 Astra 在空间推理上是一次台阶式的跃升,所以它能把点落在对的位置;世界模型能生成「下一个状态」的那张图。
机器人这一侧也在很快地对上来:
- Steerable Policies(Berkeley、Stanford 和 PI,2 月)训练一个 VLA 去接受六个层级的指令:从任务、子任务,一直到原子动作(「向左移」「张开夹爪」)、夹爪轨迹、点,以及它们的混合。那篇论文里我反复读的一句是:一个人如果可以不受限制地下操控指令,「几乎能以 100% 的成功率解决所有任务」(Chen 等)。瓶颈从来不是手,是接口。
- π0.7(PI,4 月)的提示由三样东西拼成:子任务语言、一个小世界模型生成的子目标图像,以及描述这一段该怎么做的元数据。见过的任务常常超过 90%,没见过的落在 60% 到 80%(π0.7)。
下面是一页活的说明书。同一个场景,同样的起始像素,四条不同的指令。把几种墨水开关一下试试。
Notice what the wheel is being asked to do. Not “make breakfast”. One contact-rich move lasting a second or two, picked out of everything this scene allows, and done the way the page says. Call it an atomic behaviour. That’s the unit that lives under 100 ms, and the unit a manual is written in.
So the wheel has two jobs. It has to work in a scene it’s never seen. And it has to listen: do the thing that was asked instead of the thing this scene usually calls for. The first job is the one everybody has been funding. The second is where it falls apart.
留意一下轮子被要求做的是什么。不是「做早饭」,而是一个持续一两秒、充满接触的动作:从这个场景允许的所有事情里挑出来的那一件,并且按这一页说的方式去做。我把它叫作原子行为。活在 100 毫秒以下的就是这个单位,说明书也是用这个单位写的。
所以轮子有两份工作。一是在没见过的场景里做得成。二是听得进:做被要求的那件事,而不是这个场景里通常会做的那件事。第一份工作是所有人都在砸钱的。第二份,就是现在散架的地方。
3. Why today’s wheels don’t steer, and what depth means
Last week’s 27B paper ran π0.5 through LIBERO-Pro, which takes LIBERO and perturbs one thing at a time (Tang et al.):
| What changed | π0.5 success |
|---|---|
| The instruction is reworded | 95.8% |
| The objects are restyled | 96.0% |
| The objects swap positions | 20.8% |
| A different task, same scene | 0.75% |
Reword the instruction and it’s fine. Ask for a different task in a scene it knows, and it’s at 0.75%. That’s a 96-point drop. The policy isn’t reading the instruction. It’s reading the scene, and the scene has only ever meant one thing.
This has a name. The CAST paper calls it posterior collapse: “the policy learns to ignore language, instead inferring the task and actions purely from observations” (Glossop et al.). And it’s not a bug in the model. It’s the correct thing to learn from the data. If every time the camera sees this counter, the human puts the block in the tray, then the image already tells you the action. The instruction adds nothing, so the gradient never asks the model to look at it.
You can say exactly how much nothing. The most an instruction can tell the wheel about what to do, given what it already sees, is
\[I(a;\ \ell \mid o)\ \le\ H(\ell \mid o) = \log_2 K \ \text{bits}\]where $K$ is the number of different behaviours that were recorded, equally often, from that same observation. One task per scene means $K = 1$, and $\log_2 1 = 0$. Zero bits. No architecture gets steering out of data that contains none.
That gives two clean axes for a dataset:
- Breadth, $N$: how many different scenes, objects and embodiments. It widens where the wheel works.
- Depth, $K$: how many different labelled behaviours start from (nearly) the same state. It’s what makes the wheel listen.
The field counts breadth. Hours, scenes, homes, embodiments: every one of those is a breadth number. I haven’t seen a dataset card that reports K.
Here’s what the difference looks like at one table. Both modes produce the same number of episodes.
3. 今天的轮子为什么不听话,以及「深度」是什么
上周那篇 27B 的论文拿 π0.5 跑了一遍 LIBERO-Pro,它是在 LIBERO 的基础上一次只扰动一样东西(Tang 等):
| 改了什么 | π0.5 成功率 |
|---|---|
| 指令换一种说法 | 95.8% |
| 物体换个外观 | 96.0% |
| 物体互换位置 | 20.8% |
| 同一个场景,换一个任务 | 0.75% |
指令换个说法,没事。在它认识的场景里要它做另一件事,0.75%,掉了 96 个点。这个策略没在读指令。它读的是场景,而这个场景在它眼里从来只意味着一件事。
这个现象有名字。CAST 那篇论文叫它后验坍缩:「策略学会了忽略语言,转而只从观测里推断任务和动作」(Glossop 等)。而且这不是模型的 bug,这是从数据里该学到的正确结论。如果每次相机看到这张台面,人都是把方块放进托盘,那图像本身就已经告诉你动作了。指令没有带来任何新东西,所以梯度从来不会要求模型去看它。
「没有任何新东西」可以精确到数。在已经看到画面的前提下,一条指令最多能告诉轮子的信息量是
\[I(a;\ \ell \mid o)\ \le\ H(\ell \mid o) = \log_2 K \ \text{比特}\]其中 $K$ 是从同一个观测出发、被同样频繁地记录下来的不同行为数。一个场景一个任务,就是 $K = 1$,而 $\log_2 1 = 0$。零比特。数据里没有的可操控性,什么架构也变不出来。
这就给了数据集两根很干净的轴:
- 广度 $N$: 有多少不同的场景、物体和本体。它拓宽的是轮子在哪里做得成。
- 深度 $K$: 从(几乎)同一个状态出发,有多少种带标签的不同行为。它决定的是轮子听不听。
这个行业数的是广度。小时数、场景数、家庭数、本体数,每一个都是广度的数字。我还没见过哪张数据集卡片上写了 K。
下面是在一张桌子上,这个区别长什么样。两种模式产出的片段数是一样的。
That toy table has four objects and fifteen behaviours. A real kitchen counter is much deeper. With $n$ objects, $v$ things you can do to each on its own, and $r$ ways to relate one to another, a single state affords about
\[K \approx n \cdot v + n(n-1) \cdot r\]Eight objects, five verbs, three relations: $40 + 168 = 208$ behaviours, or 7.7 bits. A task-demo dataset records one or two of them and drives on to the next kitchen.
Counterfactuals are the part people skip. I mean two things by the word. One is the action nobody would put on a task list: the block on the mug, the mug pushed away from the tray. From the same starting state, a different instruction leads to a different motion. That minimal pair is the cleanest gradient language ever gets. The other is the action that fails: the tray that’s wider than the gripper. That’s a label too, and section 5 is about why it’s worth so much.
So here’s play done properly, on three objects nobody would write a paper about: a book, a bottle and a cup. Teal lines are what a task list would have. Blue ones are counterfactual. Let it run, or click any line.
那张玩具桌子上有四个物体、十五种行为。真实的厨房台面深得多。有 $n$ 个物体,每个物体单独能做 $v$ 种事,两两之间有 $r$ 种关系,那么一个状态大约允许
\[K \approx n \cdot v + n(n-1) \cdot r\]种行为。八个物体、五个动词、三种关系:$40 + 168 = 208$ 种,也就是 7.7 比特。任务示范式的数据集录下其中一两种,就开车去下一间厨房了。
反事实是大家跳过的那部分。 我用这个词指两件事。一是没人会写进任务清单的动作:把方块放到杯子上,把杯子往远离托盘的方向推。同一个起始状态,换一条指令,就对应一个不同的动作。这种最小对比对,是语言能拿到的最干净的梯度。二是会失败的动作:那个比夹爪还宽的托盘。这也是一条标签,第 5 节讲的就是它为什么这么值钱。
所以下面就把「玩」认真做一遍,用三件没人会为它们写论文的东西:一本书、一个瓶子、一个杯子。青绿色的行是任务清单上会有的,蓝色的是反事实。可以让它自己跑,也可以点任意一行。
To a wheel trained on task demos, a bottle is “the thing you lift”. That’s one line out of eight. A bottle is also a thing with a thread, a thing that gives when you squeeze it, a thing that falls over, rolls once it has, and flies if you let go at the right moment. None of that is in the pixels of a clean pick-and-place. All of it is what a driver will eventually ask for.
Four of the twenty-one need two objects at once: the bottle on the book, the cap in the cup. That’s the $n(n-1)$ term from the formula starting to show, and with three objects it has barely begun.
The evidence that depth works is already in, from people who mined it out of data they had:
- CAST took existing robot data and had a VLM write counterfactual instructions and matching actions. For manipulation that’s 500k new instruction-action pairs from 60k Bridge trajectories. The paper reports that this “roughly doubles the success rate” at following instructions, in both navigation and manipulation, with the biggest gains when distractors are around (Glossop et al.).
- Steerable Policies relabelled Bridge’s 38k task labels into 206k subtasks and close to 2M steering commands. Same videos, about 50x more language.
- The idea is old. In 2020, Lynch and Sermanet trained a language-conditioned policy on unstructured play, with under 1% of the data labelled with language (Lynch & Sermanet).
Here’s the trap, though. Relabelling can only describe what happened. You can caption one trajectory in six inks, and you should. But you can’t relabel the mug into the tray if nobody put it there. Relabelling mines depth out of pixels you already have. Play creates it.
对一只只见过任务示范的轮子来说,瓶子就是「用来拿起来的那个东西」。这只是八行里的一行。瓶子还是一个有螺纹的东西,一个捏下去会瘪的东西,一个会倒、倒了会滚、在恰当的时刻松手还会飞出去的东西。这些在一段干净的抓取放置的像素里一样都没有,而它们全都是司机迟早会开口要的。
二十一种里有四种要同时用到两个物体:瓶子立在书上,瓶盖丢进杯子。这就是公式里 $n(n-1)$ 那一项开始露头了,而三个物体才刚刚起步。
深度管用的证据已经有了,来自那些从手头数据里把它挖出来的人:
- CAST 拿现成的机器人数据,让 VLM 去写反事实的指令和对应的动作。操作任务上,是从 6 万条 Bridge 轨迹里生成了 50 万组新的「指令 - 动作」对。论文报告说,这让指令跟随的成功率「大致翻倍」,导航和操作都是,场景里有干扰物时提升最大(Glossop 等)。
- Steerable Policies 把 Bridge 的 3.8 万条任务标签,重新标注成 20.6 万条子任务和将近 200 万条操控指令。视频还是那些视频,语言多了大约 50 倍。
- 这个想法并不新。2020 年,Lynch 和 Sermanet 就在无结构的玩耍数据上训出了语言条件策略,其中带语言标签的数据不到 1%(Lynch & Sermanet)。
不过这里有个坑。重新标注只能描述发生过的事。同一条轨迹,你可以用六种墨水各写一遍,也应该这么做。但如果从来没人把杯子放进托盘,你标不出一条「杯子进托盘」。重新标注是从已有的像素里挖深度,玩才是创造深度。
4. Depth is the cheap axis
Now the part I find hard to un-see. Look at what each axis costs.
A new scene costs logistics: get there, set up, calibrate, reset, explain to somebody why you’re filming their kitchen. A new behaviour in the scene you’re already standing in costs seconds. The block is right there. Push it the other way.
Here’s a toy model of that trade. Every assumption is on a slider, so try to break it.
4. 深度是便宜的那根轴
接下来这一点,我看明白之后就再也回不去了。看看两根轴各自要花什么。
一个新场景花的是后勤:赶过去,架设备,标定,复位,还得跟人解释为什么要拍他家厨房。而在你已经站着的这个场景里多做一种行为,花的是几秒钟。方块就在那儿,往另一边推一下就行。
下面是这笔交易的一个玩具模型。每个假设都放在滑块上,你可以试着把它搞垮。
With the defaults (10,000 hours, half an hour of overhead per scene, 64 things a scene affords), going from one behaviour per scene to 64 means giving up 80% of your scenes. That costs about three points of breadth, 94.8% down to 91.6%. And steered success goes from 1.5% to 91.6%.
I don’t trust those numbers and you shouldn’t either. It’s three assumptions and a multiplication. What I do trust is the shape, because it survives every slider setting I’ve tried. Breadth enters through a power law, and at today’s scale that curve is already flat: halving your scenes barely moves it. Depth enters as a gate that starts at chance. When one factor is 95% and the other is 2%, you know which one to spend on.
And this isn’t an argument against breadth. A deep dataset from three kitchens gives you a wheel that listens beautifully in three kitchens. The claim is narrower: the field has spent five years on one axis, and the other one is nearly free.
用默认值(1 万小时,每个场景半小时开销,一个场景允许 64 种事),从每个场景 1 种行为做到 64 种,意味着放弃 80% 的场景。代价是广度掉大约三个点,从 94.8% 到 91.6%。而可操控的成功率从 1.5% 涨到 91.6%。
这些数字我自己不信,你也别信。它就是三个假设加一次乘法。我信的是形状,因为我怎么拖滑块它都不变。广度是通过一条幂律起作用的,在今天这个规模上,那条曲线已经是平的了:场景砍一半,它几乎不动。深度则是一道从瞎猜水平开始的闸门。一个因子是 95%,另一个是 2%,钱该花在哪边,不用想。
这也不是在反对广度。只从三间厨房采来的深数据,给你的是一只在三间厨房里特别听话的轮子。我的论点要窄得多:这个行业在一根轴上花了五年,而另一根几乎是白送的。
5. Wheels compose
Why does any of this get you to long-horizon tasks? Because a steerable wheel turns a long task into a sentence. The driver writes it one atom at a time, watches, and rewrites when something goes wrong.
In The Data Wall I did the grim arithmetic: an $n$-step task at per-step success $p$ finishes with probability $p^n$, and each extra nine of $p$ costs about 100x the data. Fifty steps at 90% is 0.5%.
There’s another way out, and it doesn’t touch $p$. Say a fraction $\rho$ of misses are recoverable: the robot ends up somewhere the wheel has seen before and can still be steered out of. The driver notices and tries again, up to $r$ times. Then each atom succeeds with probability
\(P = p \cdot \frac{1 - q^{\,r+1}}{1 - q}, \qquad q = (1 - p)\,\rho\)
5. 轮子可以组合
这一切凭什么能通向长程任务?因为一只可操控的轮子,把长任务变成了一句话。司机一个原子一个原子地往下写,盯着看,出了岔子就改写。
在《数据墙》里我算过那笔很惨的账:一个 $n$ 步的任务,每步成功率 $p$,整体做成的概率是 $p^n$,而 $p$ 每多一个 9,数据大约要多 100 倍。50 步、每步 90%,是 0.5%。
还有另一条出路,而且不用动 $p$。假设失手里有 $\rho$ 这么大比例是可恢复的:机器人落在了轮子见过、还能被指挥着走出来的状态。司机发现了,再试一次,最多试 $r$ 次。那么每个原子动作的成功概率是
\(P = p \cdot \frac{1 - q^{\,r+1}}{1 - q}, \qquad q = (1 - p)\,\rho\)
Same 90% wheel on all three lines. With nobody watching, fifty steps finish 0.5% of the time. With a driver and a wheel that can recover from half its misses: 6.7%. With a driver and a wheel that recovers from 97% of them: 84.3%.
What compounds isn’t the miss rate. It’s the unrecoverable miss rate, roughly $(1-p)(1-\rho)$. At $p = 0.9$ and $\rho = 0.97$ that’s 0.3% per step. You got the long-task behaviour of a 99.7% wheel without buying a single nine.
And $\rho$ is a depth number. A wheel recovers from a state only if it has been there, and the states after a mistake aren’t in a dataset of clean demos. They’re in play: the failed lift, the mug knocked sideways, the drawer left half open. I said in The Data Wall that the most valuable part of a demo is the judgment around the mistake. Here’s that same thing with a formula attached.
It also explains the 7 out of 100. Of Astra’s 100 runs on the real arms, 83 ended with the model giving up. It wasn’t confused. It knew it was failing and said so. It just had nothing underneath it that could take a correction. Steerable Policies makes the same point from the other side: with only subtask commands, a VLM “cannot act on this knowledge”.
三条线用的是同一只 90% 的轮子。没人盯着,50 步做完的概率是 0.5%。有司机,轮子能从一半的失手里救回来:6.7%。有司机,轮子能救回 97% 的失手:84.3%。
复利的不是失手率,是不可恢复的失手率,大约是 $(1-p)(1-\rho)$。$p = 0.9$、$\rho = 0.97$ 时,每步是 0.3%。你一个 9 都没多买,却拿到了一只 99.7% 的轮子在长任务上的表现。
而 $\rho$ 是一个深度的数字。轮子要能从某个状态里救回来,前提是它到过那里,而犯错之后的状态不在干净示范的数据集里。它们在玩耍里:没抬起来的那一下,被碰歪的杯子,拉开一半的抽屉。我在《数据墙》里说过,一条示范里最值钱的是错误周围的判断。这里是同一件事,配上了公式。
这也解释了那个 7/100。Astra 在真实机械臂上的 100 次运行里,有 83 次是模型自己放弃的。它不是糊涂了,它知道自己在失败,也说出来了。只是它下面没有一个能接住纠正的东西。Steerable Policies 从另一头讲的是同一件事:只有子任务指令的话,VLM「没法把这份认识付诸行动」。
6. How I’d collect it
The recipe is short, and none of it needs a new sensor.
- Stay in the scene. Budget minutes per scene, not seconds. You already paid for the setup.
- Let the driver write the grid. Show a frontier model the scene and ask for every object × verb × relation it affords, including the pointless ones and the ones that probably won’t work. That list is your shot list.
- Play the grid, and say what you’re doing. A few seconds per cell. Narration is nearly free at capture time and expensive to reconstruct later.
- Force the pairs. From the same reset, do at least two different things. That’s the counterfactual, and it’s the whole gradient for language.
- Keep the failures and what you did next. Don’t trim them. That’s where $\rho$ comes from.
- Relabel in every ink afterwards. Subtask, atomic motion, point, trace. The subgoal image is free: it’s the frame two seconds later in the same video.
- Report K. Print behaviours-per-state next to hours on the dataset card.
The last one is the one I care about most. Hours is a breadth number. As long as it’s the only number on the slide, depth will keep being what gets cut.
6. 换我来采,我会怎么采
配方很短,而且没有一条需要新传感器。
- 待在场景里别走。 每个场景按分钟算预算,不是按秒。架设的钱你已经付过了。
- 让司机来写这张表。 把场景给一个前沿模型看,让它列出这里所有的「物体 × 动词 × 关系」,包括没意义的和多半做不成的。这张单子就是你的拍摄清单。
- 把表玩一遍,边做边说。 每格几秒钟。采的时候顺口说一句几乎不要钱,事后再去还原就贵了。
- 硬凑出对比对。 从同一个复位状态出发,至少做两件不同的事。这就是反事实,语言的全部梯度都在这儿。
- 留下失败,以及你接下来做了什么。 别剪掉。$\rho$ 就是从这里来的。
- 事后用每一种墨水重新标一遍。 子任务、原子动作、点、轨迹。子目标图像是白送的:就是同一段视频里两秒之后的那一帧。
- 把 K 报出来。 在数据集卡片上,把「每个状态的行为数」印在小时数旁边。
最后一条是我最在意的。小时数是一个广度的数字。只要幻灯片上只有它一个数,被砍掉的就永远是深度。
7. Where I could be wrong
The driver might just grow hands. That 27B model learned robot control purely in simulation, with no demonstrations, and got 18 of 30 on a real dual-arm robot it had never seen. It also handles the “different task, same scene” test that sinks π0.5: 57.4%. If models like that learn to emit action chunks and get ten times faster, the line moves down. The authors are upfront that it’s not there: one pose per round, and it “is not yet able to plan action chunks reliably”. My guess is that it becomes a better driver, one that speaks in poses, long before it becomes a wheel. But that’s a guess.
The line is fuzzy. I said so in section 1. If on-robot chips get fast enough to run a few hundred reasoning tokens inside a control step, some of what I’m calling wheel becomes driver.
The models in sections 4 and 5 are toys. The bits bound is real. The success curves built on top of it are mine.
Depth has its own failure mode. Play without labels is noise, and machine labels are noisy too. CAST reports its own hindsight and counterfactual labels are only about 60 to 70% accurate. A deep dataset with wrong captions teaches the wheel to listen to the wrong thing.
Still, look at how the two halves age. The driver gets replaced every few months. Astra shipped in September, Fable 5.2 last week, and something else will be along by spring. Nobody building robots has to do anything to get that upgrade. The wheels are different. They’re on the robot, they’re trained on data only robotics people will ever collect, and they keep their value across every driver swap, as long as they speak manual: words, points, traces, next states.
So that’s where I’d put the work. Not on a robot that thinks. That part showed up on its own. On hands that listen, and on the deep, strange, play-everything data that makes them listen.
7. 我可能错在哪
司机也许会自己长出手来。 那个 27B 的模型完全在仿真里学会了控制机器人,没用任何示范,放到一台它从没见过的真实双臂机器人上,30 次里做成了 18 次。让 π0.5 翻车的「同场景换任务」测试,它也接得住:57.4%。如果这类模型学会了一次输出一段动作,再快上十倍,这条线就会往下移。作者自己说得很坦白,现在还没到:每轮只出一个位姿,而且「还不能可靠地规划动作块」。我的猜测是,它会先变成一个更好的司机,一个直接用位姿说话的司机,这比它变成轮子要早得多。但这只是猜测。
这条线是模糊的。 第 1 节我就说过。如果机器人上的芯片快到能在一个控制周期里跑几百个推理 token,我现在叫作轮子的东西,有一部分就会变成司机。
第 4、5 节的模型是玩具。 比特数的那个上界是真的,搭在它上面的成功率曲线是我自己画的。
深度有它自己的失败方式。 没有标签的玩耍只是噪声,而机器打的标签也有噪声。CAST 报告说,它自己的事后标签和反事实标签,准确率只有大约 60% 到 70%。一份深但标错了的数据,会教轮子去听错的东西。
不过,看看这两半各自是怎么变旧的。司机每隔几个月就换一个。Astra 是 9 月发的,Fable 5.2 是上周,到春天还会有下一个。做机器人的人什么都不用干,就能拿到这次升级。轮子不一样。它们待在机器人身上,训练它们的数据只有做机器人的人才会去采,而且不管司机换了多少任,它们都还值钱,前提是它们读得懂说明书:文字、点、轨迹、下一个状态。
所以我会把力气花在这儿。不是花在一台会思考的机器人上,那部分自己就来了。而是花在听话的手上,以及让它们听话的那种又深又怪、什么都玩一遍的数据上。
Appendix: where the numbers come from
| Number | Value | Source |
|---|---|---|
| Astra agentic policy, RoboLab-120 | 74.8% success; 13.37 s per call; 30.9M tokens, \$36.40 per episode | NVIDIA Astranomy; 120 simulated tasks × 5 episodes, 15 Hz, simulator paused during inference |
| Astra direct policy | 66.3% success; 18.56 s per call | same |
| Best dedicated model (Lakeside); π0.5 | 51.0%; 28.0% | same, leaderboard references |
| π0.5 and Cosmos3-Nano-Policy call time | 46.96 ms; 713.76 ms | same |
| Astra on real bimanual arms | 7 of 100 completions; mean progress 46 of 100; 83 give-ups | Robocurve StationeryBench; 5 tasks × 20 trials, 25% speed cap, thinking pauses cut from videos |
| MolmoAct2, same benchmark | 0 of 100; never left the start pose in 47 | same; it ran zero-shot on instructions longer than it was trained on, so not a fair fight for the VLA either |
| Human grip correction; trip reflex | ~74 ms; 60–140 ms | as cited in the Grid Robotics survey |
| Helix System 1; “System 2” | 200 Hz; 111–143 ms | same |
| Astra navigation decision | 7.7 s | same |
| 27B reasoning model | 747 tokens/s on one GB300; 1.1–4.3 s per move; 18/30 real trials | Tang et al. |
| π0.5 on LIBERO-Pro | 95.8 / 96.0 / 20.8 / 0.75% | same |
| π0.5 call budget on a mobile robot | 97 + 21 + 11 + 10 = 139 ms | RTC paper |
| Slip arithmetic | $d = \tfrac12 g t^2$, $g = 9.81$ | Free fall; the 5 cm finger overlap and 20 cm height are my choices |
| Kitchen-counter K | 8 objects, 5 verbs, 3 relations | My assumption, to show the scale |
| Budget model | 2 min per behaviour per scene; error ∝ N−0.3; obeyed ∝ log K ÷ log M | Toy; all on sliders |
| Composition model | independent retries; the driver always notices a miss | Toy; both assumptions are generous |
The arm, the table and everything on it are drawings. Nothing on this page is a recording of a real robot.
附录:这些数字从哪来
| 数字 | 取值 | 来源 |
|---|---|---|
| Astra 智能体策略,RoboLab-120 | 成功率 74.8%;每次调用 13.37 秒;每局 3,090 万 token、36.40 美元 | NVIDIA Astranomy;120 个仿真任务 × 5 局,15 Hz,推理期间仿真器暂停 |
| Astra 直接策略 | 成功率 66.3%;每次调用 18.56 秒 | 同上 |
| 最好的专用模型(Lakeside);π0.5 | 51.0%;28.0% | 同上,榜单参考值 |
| π0.5 与 Cosmos3-Nano-Policy 的调用耗时 | 46.96 毫秒;713.76 毫秒 | 同上 |
| Astra 在真实双臂上 | 100 次里完成 7 次;平均进度 46/100;83 次主动放弃 | Robocurve StationeryBench;5 个任务 × 20 次,速度上限 25%,视频里剪掉了思考的停顿 |
| MolmoAct2,同一个基准 | 100 次里 0 次;47 次没离开起始姿态 | 同上;它是零样本跑的,指令也比训练时的长,所以这对 VLA 同样不算公平对决 |
| 人手握力修正;绊倒反射 | 约 74 毫秒;60–140 毫秒 | 转引自 Grid Robotics 综述 |
| Helix System 1;「System 2」 | 200 Hz;111–143 毫秒 | 同上 |
| Astra 导航决策 | 7.7 秒 | 同上 |
| 27B 推理模型 | 单张 GB300 上每秒 747 个 token;每步 1.1–4.3 秒;真机 30 次成 18 次 | Tang 等 |
| π0.5 在 LIBERO-Pro 上 | 95.8 / 96.0 / 20.8 / 0.75% | 同上 |
| 移动机器人上一次 π0.5 调用的时间账 | 97 + 21 + 11 + 10 = 139 毫秒 | RTC 论文 |
| 打滑的算术 | $d = \tfrac12 g t^2$,$g = 9.81$ | 自由落体;5 cm 的手指重叠和 20 cm 的高度是我选的 |
| 厨房台面的 K | 8 个物体、5 个动词、3 种关系 | 我的假设,只为说明量级 |
| 预算模型 | 每个场景每种行为 2 分钟;误差 ∝ N−0.3;听从程度 ∝ log K ÷ log M | 玩具模型;全在滑块上 |
| 组合模型 | 每次重试相互独立;司机每次都能发现失手 | 玩具模型;两条假设都偏乐观 |
机械臂、桌子和桌上的所有东西都是画出来的。这一页上没有任何一段是真实机器人的录像。
Sources
来源
- OpenAI reassigns its robotics researchers (The Decoder, citing Fortune)
- Astranomy: GPT-6 Astra as a native robot policy (NVIDIA)
- StationeryBench: GPT-6 Astra vs MolmoAct2 on bimanual manipulation (Robocurve)
- Envisioning Agentic Robotics (Grid Robotics, arXiv 2611.18359)
- Training Reasoning Models in Simulation to Natively Use Real Robots (Tang et al., arXiv 2611.18562)
- Real-Time Execution of Action Chunking Flow Policies (arXiv 2506.07339)
- Steerable Vision-Language-Action Policies (Chen et al., arXiv 2602.13193)
- π0.7: a Steerable Generalist Robotic Foundation Model (arXiv 2604.15483)
- CAST: Counterfactual Labels Improve Instruction Following (Glossop et al., arXiv 2508.13446)
- Language Conditioned Imitation Learning over Unstructured Data (Lynch & Sermanet, arXiv 2005.07648)
- The Data Wall (this blog)