The Reachability of Steering: One Distribution-Level Law Behind Every VLA Trick

CFGRL, DSRL, test-time search, token-level RL — it looks like a zoo of methods for “steering” a frozen robot policy toward higher reward. Underneath, they’re all the same move: take the base policy’s distribution, multiply it by a non-negative weight, and shove mass toward high value. That move has a hard mathematical ceiling, and it doesn’t care whether you built your policy out of a flow model or an autoregressive token model. This post drops each method onto that one law, and lands on the uncomfortable conclusion: the only knob you actually get to turn is how rich and wide the pretrained base distribution is. Every demo is live — drag the sliders as you read.

The whole post is one inequality, so let me put it up front:

\[\operatorname{supp}\pi^{\star}(\cdot\mid s)\ \subseteq\ \operatorname{supp}\pi_{\mathrm{base}}(\cdot\mid s)\]

The steered policy’s support can never be larger than the base policy’s support. The demo below is just this inequality made draggable. Pull steering strength all the way up and the achieved value only ever creeps up to the in-support ceiling — the gap Δ to the true optimum doesn’t move. The only thing that closes Δ is widening the base.

Reweighting moves mass around inside the support — it can’t manufacture mass where the base put none. The optimum behind the right wall is simply unreachable until the base learns to put a little probability there.

1. Pin the problem down first

Your base policy $\pi_{\mathrm{base}}(a\mid s)$ comes from imitation / pretraining. You want to do RL on top of it to chase reward. Almost every method that promises “improve it without breaking it” is really optimizing the same KL-regularized RL objective:

\[\max_{\pi}\ \mathbb{E}_{a\sim\pi(\cdot\mid s)}\!\big[Q(s,a)\big]\;-\;\alpha\, D_{\mathrm{KL}}\!\big(\pi(\cdot\mid s)\,\|\,\pi_{\mathrm{base}}(\cdot\mid s)\big)\]

That KL term isn’t decoration — it’s where the stability comes from. Drop it and the policy drifts onto state–action pairs the base never visited, the critic extrapolates garbage there, and training falls over. So whether you’re flow or token, everyone keeps this anchor.

Here’s the part people misdiagnose. “Flow matching has no good online-RL method” is a computational complaint — policy gradients want $\log\pi(a\mid s)$ or an importance ratio $\pi_{\text{new}}/\pi_{\text{old}}$, and a flow’s density needs you to integrate the velocity-field divergence along the ODE, which you can’t get in closed form. But DSRL (move RL into noise space), CFGRL (use a guidance weight), and test-time (just pick) all sidestep that density. So the computational wall isn’t the real wall. The real wall is the one below, and it holds for any generative class:

The central question. Without breaking stability (i.e. keeping the KL anchor, or staying inside the base’s support), what is the highest value steering can reach, and what sets it?

2. The core theorem: reweighting can’t leave the support

That objective has a closed-form solution — the familiar Gibbs / Boltzmann form (the same $\pi^\star\propto\pi_{\text{ref}}e^{r/\beta}$ you’ve seen in RLHF / DPO):

\[\pi^\star(a\mid s)=\frac{1}{Z(s)}\,\pi_{\mathrm{base}}(a\mid s)\,\exp\!\Big(\tfrac{1}{\alpha}Q(s,a)\Big)\]

The exponential factor is always positive. It can only reweight mass that’s already there — it can’t conjure mass out of nothing. So:

\[\pi_{\mathrm{base}}(a\mid s)=0\ \Rightarrow\ \pi^\star(a\mid s)=0\quad\Rightarrow\quad \operatorname{supp}\pi^\star\ \subseteq\ \operatorname{supp}\pi_{\mathrm{base}}\]

Take $\alpha\to 0$ (pure exploitation) and the steered policy collapses onto the single best action that the base actually covers:

\[V_{\mathrm{steer}}(s)=\!\!\max_{a\in\operatorname{supp}\pi_{\mathrm{base}}}\!\!Q(s,a)\ \le\ \max_{a}Q(s,a)=V^\star(s),\qquad \Delta(s)=V^\star(s)-V_{\mathrm{steer}}(s)\ge 0\]

and $\Delta(s)=0$ iff the true optimal action $a^\star(s)$ lives in the base support. This is the precise meaning of “the base model matters a lot.” Note the ceiling is set by the support, not by the base’s average performance. You don’t need the base to be good on average — you need it to occasionally emit that good action.

And support-containment is only the asymptotic story. The finite-sample version bites harder. Steering (DSRL and friends) finds good actions by sampling from $\pi_{\mathrm{base}}$ and getting feedback. The chance of hitting a good action of density $\epsilon$ within $N$ draws is about $1-(1-\epsilon)^N$, so

\[N\ \sim\ \frac{1}{\pi_{\mathrm{base}}(a^\star\mid s)},\qquad C^\star=\mathbb{E}_{s\sim d^\star}\!\left[\frac{1}{\pi_{\mathrm{base}}\!\big(a^\star(s)\mid s\big)}\right]\]

$C^\star$ is exactly the coverage / concentrability coefficient from offline RL. The more density the base puts on good actions (the wider and richer it is), the smaller $C^\star$, the cheaper steering gets; as that density → 0, sample complexity blows up exponentially. So the good action doesn’t just have to be “in the support” — it needs non-negligible density. That’s the precise form of “the distribution has to be wide enough.”

3. Every method is one special case of this law

What follows are five methods. The first four are all just different ways to write $\pi=\frac1Z\pi_{\mathrm{base}}\,g(s,a)$ with $g\ge 0$ — only the form of $g$ and the algorithm differ. The fifth, Q-chunking, is orthogonal: it doesn’t touch $g$, it makes $Q$ itself learnable.

CFGRL — classifier-free guidance, ported from image diffusion

Train the flow/diffusion to condition on an “optimality / high-reward” signal $c$; at sampling time crank the guidance weight $w$ and extrapolate along $v_w=v_\varnothing+w(v_c-v_\varnothing)$, pushing toward high-reward modes.

\[\pi_w(a)\ \propto\ \pi_{\varnothing}(a)\Big(\tfrac{\pi_c(a)}{\pi_\varnothing(a)}\Big)^{\!w}\ \propto\ \pi_{\mathrm{base}}(a)\,e^{\,wQ(a)/\beta}\]

On the law: $g=(\pi_c/\pi_{\mathrm{base}})^w\propto e^{wQ/\beta}$, and the guidance weight $w$ is the inverse temperature $1/\alpha$. No matter how big $w$ gets, it’s still just reweighting the unconditional base — the support doesn’t change.

Drag $w$ up: mass piles onto the high-reward mode, entropy drops — but the result is locked inside the base support. Guidance is a knob, not an escape hatch.

DSRL — diffusion steering by doing RL in noise space

Freeze the base. The action is a deterministic function of the input noise, $a=f_{\mathrm{base}}(z,s)$ (given the frozen ODE). So instead of doing RL in action space (density unavailable), learn a policy $\pi_z(z\mid s)$ over the noise $z$ with plain off-policy RL (SAC), treating the base as a black-box decoder. Extremely sample-efficient, forward-pass only.

\[\max_{\pi_z}\ \mathbb{E}_{z\sim\pi_z(\cdot\mid s)}\big[Q\big(s,\,f_{\mathrm{base}}(z,s)\big)\big],\qquad a=f_{\mathrm{base}}(z,s)\]

On the law: the reachable action set is exactly the image of $f_{\mathrm{base}}(\cdot,s)$, which equals $\operatorname{supp}\pi_{\mathrm{base}}$. Changing $z$ is an efficient search inside the support — not an escape from it.

Drag the noise handle around the whole disk and the action stays trapped inside the teal image. The red $a^\star$ outside it is unreachable for every $z$ — DSRL’s optimization can’t go where the base never maps.

Test-time search / Best-of-N

Don’t train a policy at all. At inference, sample $N$ candidate actions from the base, score them with a value / reward model, keep the best (or run a small search / MPC). It’s empirical reweighting by reject-and-select.

\[a^{(N)}=\arg\max_{i}\,Q(s,a_i),\quad a_i\sim\pi_{\mathrm{base}}(\cdot\mid s),\qquad \mathbb{E}\big[Q(a^{(N)})\big]\ \xrightarrow[N\to\infty]{}\ \!\!\max_{a\in\operatorname{supp}\pi_{\mathrm{base}}}\!\!Q\]

On the law: $g\propto \mathbb{1}[a=\arg\max Q]$, selecting only among base samples. It’s the cleanest demonstration of “ceiling + coverage” there is.

The best-of-N curve climbs and then flattens onto the in-support ceiling. It never reaches the global optimum, no matter how big you make N. To raise a good action of density ε you’d need N ∼ 1/ε — there’s your $C^\star$ again.

Token-level / autoregressive RL

For autoregressive VLAs that discretize actions into tokens (the OpenVLA family), you can do PPO / GRPO directly, RLHF-style, with reward at the end of the sequence. This is the “easy” class — token policies have an analytic softmax likelihood, so standard policy gradients just work (unlike flow). In principle it can move mass to any token combination — the full support of the discrete action space.

\[\max_{\theta}\ \mathbb{E}_{a\sim\pi_\theta}\!\big[R(s,a)\big]\;-\;\beta\, D_{\mathrm{KL}}\!\big(\pi_\theta\,\|\,\pi_{\mathrm{base}}\big)\quad\Longrightarrow\quad \pi_\theta^\star\propto\pi_{\mathrm{base}}\,e^{\,Q/\beta}\]

The nuance that matters: to not collapse, everyone adds the $\beta\cdot$KL anchor — which lands you right back on the same Gibbs form and the same near-support bias. Discretization caps precision on top of that. The support bias isn’t a flow thing — it comes from the KL anchor everyone bolts on for stability.

Lower β and the optimum wants to climb to the far, higher peak — but cross the line and you’re in the off-distribution band where the critic lies and training collapses. Raise β and you’re pinned safely near base. Stable vs. surpassing-the-demos is one tension, two ends.

Q-chunking — RL on chunked actions (orthogonal)

Do TD learning in a chunked action space: the policy emits $k$ steps at once and the critic does n-step backups over the chunk. Two payoffs: (1) the effective horizon shrinks by $k$, credit assignment gets shorter, the n-step return is unbiased; (2) committing a whole chunk gives temporally coherent exploration — no more per-step random kicks that cancel out into a jitter in place.

\[\hat Q\big(s_t,\mathbf a_{t:t+k}\big)=\sum_{i=0}^{k-1}\gamma^i r_{t+i}+\gamma^k\max_{\mathbf a'}Q\big(s_{t+k},\mathbf a'\big),\qquad H\ \to\ \lceil H/k\rceil\]

On the law: it doesn’t change $g$, it doesn’t change the support. It fixes the other half — “under long-horizon sparse reward, $Q$ won’t train and exploration is useless” — which is exactly what the steering methods above (they hand you a controllable knob) are missing.

Same start, same step budget. Per-step resampling (steel-blue, H=1) wanders near the origin; committing chunks (teal) sweeps out far bigger loops and usually ends much farther out. Hit “re-roll” a few times to watch the trend — coherent exploration covers ground, per-step noise cancels itself.

4. Tie it together: the only lever is base richness

Write every steering method in one unified form — keep the expressive base, multiply by a non-negative weight that tilts the output toward high reward:

\[\pi(a\mid s)=\frac{1}{Z(s)}\,\pi_{\mathrm{base}}(a\mid s)\,g(s,a),\qquad g(s,a)\ge 0\ \ \Longrightarrow\ \ \operatorname{supp}\pi\subseteq\operatorname{supp}\pi_{\mathrm{base}}\]

Method	Weight $g(s,a)$ / mechanism	Bound by the support ceiling?
CFGRL	$(\pi_c/\pi_{\mathrm{base}})^w\propto e^{wQ/\beta}$	Yes
DSRL	implicit; RL over $z$, $a=f_{\mathrm{base}}(z)$, reachable = flow image	Yes
test-time	$\mathbb{1}[a=\arg\max_i Q]$, $a_i\sim\pi_{\mathrm{base}}$	Yes
RL-token	$e^{Q/\beta}$ (KL anchor; discrete full support but pulled back)	Yes (in practice)
Q-chunking	doesn’t change $g$; makes $Q$ learnable (unbiased n-step, $H\to H/k$, coherent exploration)	Orthogonal

So here’s the whole thing, formalized:

Proposition (reachability of steering). Given a frozen base $\pi_{\mathrm{base}}$ and the true optimum $a^\star(s)=\arg\max_a Q^\star(s,a)$, any steering policy of the form $\pi(a\mid s)=\frac1Z\pi_{\mathrm{base}}(a\mid s)\,g(s,a)$ with $g\ge 0$ satisfies $\operatorname{supp}\pi\subseteq\operatorname{supp}\pi_{\mathrm{base}}$, hence \(V_{\mathrm{steer}}(s)=\!\!\max_{a\in\operatorname{supp}\pi_{\mathrm{base}}}\!\!Q^\star(s,a)\le V^\star(s),\qquad \Delta(s)\ge0,\) \(\Delta(s)=0\iff a^\star(s)\in\operatorname{supp}\pi_{\mathrm{base}};\qquad \text{sample complexity}\ \propto\ C^\star=\mathbb{E}_{s\sim d^\star}\!\Big[\tfrac{1}{\pi_{\mathrm{base}}(a^\star(s)\mid s)}\Big].\)

The takeaway. Under the anchored / regularized paradigm, the only lever that moves $\Delta$ and $C^\star$ is how much coverage density $\pi_{\mathrm{base}}$ puts on useful-but-rare actions. So the goal isn’t to invent yet another flow-RL algorithm (the DSRL line is already enough) — it’s two things:

1. Maximize the base’s coverage width and good-action density — more diverse pretraining data, deliberately higher-entropy / more varied demos. This shoves the hard part back from RL into pretraining.
2. Build a closed loop that can safely, autonomously grow the support — steer to bootstrap → autonomously roll out and harvest the slightly-off-distribution successes → fold them back into the base and grow it → repeat. The prerequisite is an oracle that can stand in for a human on failure detection / safe recovery.

Get those two right, and only then do the five methods above have a ceiling worth talking about.

References

The methods this post collapses onto one law, and the pieces it leans on:

• DSRL — A. Wagenmaker, M. Nakamoto, Y. Zhang, S. Park, W. Yagoub, A. Nagabandi, A. Gupta, S. Levine. Steering Your Diffusion Policy with Latent Space Reinforcement Learning. arXiv:2506.15799 · project
• CFGRL — K. Frans et al. Diffusion Guidance Is a Controllable Policy Improvement Operator. arXiv:2505.23458 · code
• Q-chunking — Q. Li, Z. Zhou, S. Levine. Reinforcement Learning with Action Chunking. arXiv:2507.07969 (NeurIPS 2025)
• Token / autoregressive VLA — M. J. Kim et al. OpenVLA: An Open-Source Vision-Language-Action Model. arXiv:2406.09246 — the OpenVLA-class policies you fine-tune with PPO / GRPO.
• The KL-regularized closed form $\pi^\star\propto\pi_{\mathrm{ref}}\,e^{r/\beta}$ — R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. Manning, C. Finn. Direct Preference Optimization. arXiv:2305.18290 — the same Gibbs form as RLHF / control-as-inference.
• Coverage / concentrability $C^\star$ — J. Chen, N. Jiang. Information-Theoretic Considerations in Batch Reinforcement Learning. ICML 2019, arXiv:1905.00360
• Related — Q-Guided Flow, the test-time value-guided flow I walk through elsewhere on this site: my QGF post · q-guided-flow.github.io

---

$\pi^\star\propto\pi_{\mathrm{base}}\,e^{Q/\alpha}$ · support-containment · the coverage coefficient $C^\star$ — the one line every steering method shares.