Q-Guided Flow, From the Ground Up: Guiding a Flow Policy with a Value Function at Test Time

A from-scratch, step-by-step tutorial whose goal is to fully unpack the Q-Guided Flow (QGF) paper. We first build the background you need (flow models, the critic), then state the core problem, then take the equations apart one by one — with interactive geometric visualizations woven throughout so you don’t just follow the idea, you see it. Every demo is draggable and playable, so play as you read.

Q-guided flow in a 2-D action space: noise particles drift along the reference velocity field while the gradient of $Q$ gently nudges them toward high-value regions. This is exactly the mechanism the whole post unpacks — play with it first to build intuition, then read on.

1. The big picture: what is this paper actually doing?

The setting is reinforcement learning (RL) for continuous control (e.g. robotics): we want a policy $\pi(a \mid s)$ that, given the current state $s$ (what the robot perceives), outputs a good action $a$ (how to drive the motors).

The most expressive continuous-control policies today are diffusion / flow models: they can fit very complex, multi-modal action distributions and are stable to train by imitation. But shoehorning them into RL to “chase high reward” is painful — you either need a bespoke training objective, or you have to backpropagate through the entire denoising process, which is unstable and hard to scale.

QGF’s core claim is: decouple the two things completely, train each separately with the most stable standard method, then stitch them together at test time.

1. A reference flow policy $v_\theta$, trained by behavior cloning (BC) — it only learns “in this state, which actions are reasonable / look like the data,” with no guarantee of being “best.”
2. A critic $Q_\phi(s,a)$ (a value function), trained by standard TD learning — it scores actions, telling you “how high is this action’s expected future return.”

Then at test time (inference), the gradient of $Q$ is used to “guide” the flow policy’s sampling so that it generates high-value actions — with no further policy training at all. This is what we mean by “test-time policy improvement.”

Below is the paper’s “one-figure summary,” which I’ve turned into an auto-looping animated version: one denoising trajectory, four panels each performing its own “way of asking $Q$.” Skim it now for a first impression; we’ll return at the end and explain every detail. Try dragging the center of the blue contours in any panel (that’s the peak of $Q$) — every gradient arrow recomputes live.

Four ways of taking the gradient, side by side. ① pure flow denoising (no $Q$); ② BPTT backprop — the arrow jitters violently once it reaches $a_t$ = high variance; ③ OOD queries $Q$ directly at the noisy point, pointing every which way; ④ QGF jumps one step to $\hat a_1$, takes the gradient there, and carries it back unchanged. Don’t worry if it’s opaque now — this is exactly what the post explains.

2. Background: what is a “flow model”?

A flow model (flow / diffusion) is a generative model: it “sculpts” simple noise into a complex distribution. Here, that target distribution is “the good actions that appeared in the data, given state $s$.”

It works by denoising: start from pure Gaussian noise $a_0 \sim \mathcal{N}(0, I)$ and walk along time $t$ from $0$ to $1$, gradually turning it into a clean action sample $a_1$. Which direction to push at each step is decided by a learned velocity field $v_\theta(s, a_t, t)$. The whole process is solving an ordinary differential equation (ODE):

\[\frac{da_t}{dt} = v_\theta(s, a_t, t)\]

In practice we approximate it with small Euler steps:

\[a_{t+\Delta t} = a_t + v_\theta(s, a_t, t)\cdot \Delta t\]

Think of the velocity field as a body of “flowing water”: every location has an arrow saying “push this way,” and noise particles drift along those arrows, eventually pooling at the data distribution’s “peaks” (each peak = one kind of reasonable action). That is the entire mechanism of flow-policy sampling — keep the picture of “stepping along the velocity field” in mind, because everything below is built on it.

3. The core idea: how to fold $Q$ into the policy

We want an improved policy $\pi$: it generates higher-value actions, but it mustn’t stray too far — otherwise it would find weird actions that $Q$ mistakenly scores as “high.” This “high value, but don’t stray” requirement is written formally as a KL-regularized reward-maximization problem:

\[\max_{\pi}\ \mathbb{E}_{a\sim \pi(\cdot\mid s)}\big[Q(s,a)\big]\ -\ \beta\, D_{\mathrm{KL}}\big(\pi(\cdot\mid s)\,\|\,\hat\pi(\cdot\mid s)\big)\]

The first term wants high value; the second ($\beta$ times the KL divergence) penalizes “straying from the reference policy $\hat\pi$.” This problem has a beautiful closed-form solution:

\[\pi(a\mid s)\ \propto\ \hat\pi(a\mid s)\cdot \exp\!\left(\tfrac{1}{\beta}\, Q(s,a)\right)\]

Intuitively: take the reference distribution and multiply it by a re-weighting factor $\exp(Q/\beta)$ — high-$Q$ actions are amplified, low-$Q$ ones are suppressed. Here $\beta$ is a “temperature”: smaller $\beta$ means more aggressive re-weighting (almost only the highest-$Q$ action survives); larger $\beta$ stays closer to the original reference. Geometrically, it just “pinches” the reference density according to the height of $Q$.

The key step: what diffusion / flow models actually learn is the score function $\nabla_a \log p(a)$. Taking $\nabla_a \log$ of both sides above (the normalizer $Z$ doesn’t depend on $a$, so it vanishes under the derivative):

\[\nabla_a \log \pi(a\mid s)\ =\ \underbrace{\nabla_a \log \hat\pi(a\mid s)}_{\text{reference policy, }v_\theta\text{ already knows it}}\ +\ \underbrace{\tfrac{1}{\beta}\,\nabla_a Q(s,a)}_{\text{guidance term}}\]

This is exactly classifier guidance, except the “classifier” is replaced by the learned $Q$ function. The conclusion is clean: to sample from the improved policy $\pi$, run the reference flow’s denoising as usual, but add the guidance term $\tfrac1\beta\nabla Q$ at every step.

Generalizing it to the noisy action $a_t$ during denoising (those half-finished intermediate states):

\[\nabla_{a_t}\log \pi(a_t\mid s)\ \approx\ \nabla_{a_t}\log\hat\pi(a_t\mid s)\ +\ \tfrac{1}{\beta}\,\nabla_{a_t} Q(s,a_t)\]

The reference-score term is already provided by $v_\theta$. So the only question becomes: where, and how, should the guidance term $\nabla_{a_t}Q$ be computed? This is the crux of the whole paper.

4. The real difficulty: where should $\nabla_a Q$ be computed?

Here’s the trap: $Q$ was only ever trained on the “clean, complete actions” in the data. But the intermediate $a_t$ during denoising (especially early, near pure noise) is very far from that training data — it is out-of-distribution (OOD). On inputs it has never seen, the neural network $Q$ gives unreliable values and gradients. The paper compares three approaches:

• Approach 1 (most direct, but it fails): the OOD gradient $\nabla_{a_t}Q(s,a_t)$. Query $Q$’s gradient directly at the noisy point $a_t$. The problem is the one above: at an OOD noisy point, $Q$’s gradient can point in an arbitrarily wrong direction, and may even “exploit” the critic — finding a region $Q$ mistakenly scores high but is actually bad.

• Approach 2 (more principled, but too expensive / unstable): the BPTT gradient $\nabla_{a_t}Q\big(s,\mathrm{ODE}(a_t)\big)$. Since the flow deterministically maps $a_t$ to a clean action $a_1=\mathrm{ODE}(a_t)$, define $Q$ to always be queried on the clean action (in-distribution, trustworthy). The cost: this gradient needs backpropagation through the entire denoising chain (BPTT = backpropagation through time) — expensive, and extremely sensitive to noise (a tiny perturbation of $a_t$ makes the gradient direction lurch wildly, like a “butterfly effect,” with huge variance).

• Approach 3 (QGF): extrapolate one step to estimate a clean action, and query $Q$ there. Neither run the whole chain (avoid BPTT’s cost and jitter) nor query at the raw noisy point (avoid OOD’s unreliability). Details next section.

Before going further, let’s see “where each of the three approaches queries $Q$.” Drag time $t$ and the noisy point $a_t$ and you’ll see three markers: orange $a_t$ (OOD’s query point), cyan $\hat a_1$ (QGF’s query point, next section), and purple $\mathrm{ODE}(a_t)$ (BPTT’s query point). Notice: early in denoising (small $t$), the orange point often lands in the “untrustworthy OOD region,” while the cyan / purple points already sit on the data peak; only as $t\to 1$ do all three coincide.

The three guidance gradients’ query points, compared. QGF’s trick: use a one-step extrapolation to get an estimate of the clean action, and query $Q$ there — cheap, and inside the trusted region.

5. QGF’s solution: one-step estimate + drop the Jacobian

Step 1: approximate the clean action with a single Euler step. Instead of integrating the whole ODE, stand at $a_t$ and “jump” all the way to $t=1$ in one shot along the reference velocity field $v_\theta$:

\[\hat a_1\ =\ a_t\ +\ v_\theta(s,a_t,t)\cdot(1-t)\]

(Geometrically, this is the cyan arrow in the panel above: from $a_t$, follow the velocity direction for the remaining time $1-t$ and jump to the endpoint estimate.) There’s a lovely property here: for a standard linear-interpolation flow, this one-step estimate exactly equals the model’s posterior mean $\mathbb{E}[a_1\mid a_t]$ — i.e. “the flow model’s best guess of the clean action.” Scoring it with $Q$ is both cheap and lands near the data distribution. Reasonable.

Step 2: write the gradient with the chain rule, then replace the Jacobian with the identity. Expand the gradient of $Q(s,a_1)$ through $\hat a_1$:

\[\nabla_{a_t} Q(s,a_1)\ \approx\ \nabla_{a_t} Q(s,\hat a_1)\ =\ \underbrace{\left(\tfrac{\partial \hat a_1}{\partial a_t}\right)^{\!\top}}_{J}\ \nabla_{\hat a_1} Q(s,\hat a_1)\]

Here $J=\partial\hat a_1/\partial a_t$ is “the Jacobian of the clean estimate with respect to the noisy point.” But it requires differentiating through the neural network $v_\theta$, which in practice is often ill-conditioned and amplifies noise. QGF simply replaces $J$ with the identity matrix $I$:

\[\boxed{\ \nabla_{a_t} Q(s,a_1)\ \approx\ \hat J^{\top}\,\nabla_{\hat a_1} Q(s,\hat a_1)\,,\quad \hat J=I,\ \ \hat a_1=a_t+v_\theta(s,a_t,t)\,(1-t)\ }\]

In other words: the guidance term is just the gradient of $Q$ at the “one-step clean estimate $\hat a_1$,” used directly. Cheap (one extra $Q$-gradient), trustworthy ($\hat a_1$ is a sensible action), and stable (no long chain, no Jacobian).

Adding this guidance term to every denoising step gives the full inference algorithm:

\[\begin{aligned} &a_0\sim\mathcal N(0,I)\\ &\textbf{for } t=0,\tfrac1T,\dots,1-\tfrac1T:\\ &\qquad \hat a_1 \leftarrow a_t+(1-t)\,v_\theta(s,a_t,t) &&\text{// one Euler step: estimate the clean action}\\ &\qquad g \leftarrow \nabla_{\hat a_1} Q_\phi(s,\hat a_1) &&\text{// gradient of }Q\text{ at the clean estimate}\\ &\qquad a_{t+1} \leftarrow a_t+\tfrac1T\big(v_\theta(s,a_t,t)+\tfrac1\beta\,g\big) &&\text{// velocity + guidance, one step together}\\ &\textbf{return } a_T \end{aligned}\]

At each step, the flow velocity $v_\theta$ is responsible for “staying on the data manifold,” and the guidance term $\tfrac1\beta g$ for “bending a little toward high value”; the two are added, and $\beta$ controls how much. The whole pipeline looks like this:

This interactive page lets you drive the mechanism by hand. The setting is 1-D: the data has three peaks (at $a=-2,0,1$), and the true optimal action is $a^*=1$ (true return $=-(a-1)^2$). But the critic $Q$ has learned a flaw — far from the data, around $a\approx 3$, there is a “spuriously high” bump (a classic critic pathology in the OOD region). You can switch among the three guidances, drag the guidance weight $1/\beta$, and watch where the particles end up — tracking their true return and their critic score.

Key observation: crank the weight up and OOD “fools” the samples toward that spurious bump (high critic score, low true return — this is “exploiting the critic”); QGF always delivers samples to the true optimal peak $a^*=1$, because it only queries $Q$ at a sensible clean estimate and never steps into the bump.

Next, let’s draw a whole denoising trajectory frame by frame. The page below plots the trajectory in the “action $a$ (horizontal) × time $t$ (vertical)” plane: it climbs from the bottom (pure noise) step by step to the top (clean action). Note the two dashed lines — the cyan diagonal dashed line’s landing point at the top is $\hat a_1$ (QGF’s query point, which is itself the “one-step approximation”); the top of the orange vertical dashed line is $a_t$ itself (OOD’s query point). Hit “step” or “play” and watch how $\hat a_1$ moves with the steps and how $Q$’s guidance arrow bends the trajectory toward the optimal peak.

Switch guidance to OOD, raise the weight, hit play — the trajectory is “fooled” all the way toward the spurious bump at $a\approx 3$; switch back to QGF and the cyan query point $\hat a_1$ stays glued to the data peak, converging cleanly to the true optimum $a^*=1$.

6. Why the “approximation” beats the “exact” version

QGF makes two seemingly lazy approximations: dropping the Jacobian, and replacing the whole ODE with a single step. Counterintuitively, the paper finds they are not compromises — each one beats its more “exact” counterpart:

First, one-step estimate vs full ODE denoising. Running the whole denoising forces the estimated clean action onto the entire data distribution (it must represent all the modes in the data); the one-step approximation allows small deviations, letting the flow “pick out” one high-value mode rather than being dragged back to represent the whole dataset. So this approximation actually gives the optimization more freedom. (Interestingly, the paper notes that EDP, the best-performing train-time baseline, also uses a one-step approximation.)

Second, identity vs true Jacobian. The true $J$ is easily ill-conditioned (especially early in denoising, when the one-step approximation is already coarse), amplifying noise and making the gradient variance explode; replacing it with $I$ gives a low-variance, clean direction. In practice, a low-variance gradient estimate simply makes a better “optimizer.”

The page below is dedicated to “why imprecise is actually better.” The thing to watch is the stability of the guidance gradient: plot the guidance gradient $g$ as a function of $a_t$ — horizontal axis is the noisy point $a_t$, vertical axis is the guidance gradient computed there. Three curves:

• QGF (the approximation we use): take $Q’$ at $\hat a_1=$ the posterior mean, and treat the Jacobian as $I$ — the curve is smooth.
• BPTT (exact, run the whole chain): $\tfrac{d}{da_t}Q(\mathrm{ODE}(a_t))$ — the curve jitters violently with spikes. The spikes occur at “watersheds”: once $a_t$ crosses some boundary, the whole denoising chain flips to a different peak, the endpoint changes abruptly, and the gradient blows up.
• OOD: $Q’(a_t)$ — smooth, but pointing at the “spurious bump” (wrong direction).

Drag time $t$: the smaller $t$ (the longer the chain), the wilder BPTT’s spikes; as $t\to1$ the three converge. Then drag the probe $a_t$: it shows how much each gradient changes when you nudge the probe by a tiny $\epsilon$ — smaller change = more stable. BPTT jumps at the slightest touch near a watershed (high variance); QGF barely moves (low variance).

The paper quantifies this with two metrics: (1) the gradient’s noise sensitivity — compare the cosine similarity of the gradient at $a_t$ vs $a_t+\epsilon$; closer to 1 is more stable, and QGF has the lowest variance; (2) the ability to optimize $Q$ — treat each gradient as an “optimizer” and look at the final action’s $Q$ value; QGF is best, approaching the best-of-$N$ upper bound, while OOD does poorly because it exploits the critic.

7. Down to the metal: three rings that take BPTT and the Jacobian apart

If you want to really understand “why BPTT jitters, and why QGF’s two cuts work,” let’s take it apart to the lowest level. This section goes ring by ring, from shallow to deep, each ring paired with a hands-on, genuinely-simulated interactive page — all from zero background.

Ring 1: what is a Jacobian?

Start in 1-D. The derivative $f’(x)$ means something simple: push the input $x$ by a tiny $\Delta$ and the output changes by about $f’(x)\cdot\Delta$. It’s the slope of the curve at that point — the “local linear approximation.”

In higher dimensions (input and output are both vectors), this “slope” upgrades to a matrix, the Jacobian $J$:

\[\Delta(\text{output})\ \approx\ J\cdot \Delta(\text{input}),\qquad J_{ij}=\frac{\partial(\text{output}_i)}{\partial(\text{input}_j)}\]

Geometrically, what $J$ does is: turn a tiny “circle” at the input into an “ellipse” at the output — stretching + rotating. The lengths of the ellipse’s two semi-axes are the singular values — the magnification along each direction. If one direction is magnified a lot while the other is nearly squashed flat (a long, thin ellipse), $J$ is said to be ill-conditioned: a tiny wobble of the input gets amplified into a huge swing along some direction — this is exactly the source of “noise being amplified.”

Drag the point on the left; the right side shows in real time the ellipse the little circle maps to. Turn up the “curviness” (mimicking a neural-net-like function $v_\theta$ that bends everywhere) and the ellipse changes shape drastically from place to place, sometimes squashed into a line (ill-conditioned). Remember this picture: $v_\theta$ is a very curvy function, and its Jacobian wanders around and goes ill-conditioned just like this.

Ring 2: the chain rule = a product of Jacobians, so BPTT jitters

Denoising is a chain: $a_t \to a_{t+\Delta}\to \cdots \to a_1$, where each arrow is one Euler step (one function). So the final clean action is a long stack of nested functions: $a_1=\mathrm{ODE}(a_t)$.

BPTT wants $\nabla_{a_t}Q(a_1)$. By the chain rule, that equals multiplying every step’s Jacobian together and then by $\nabla Q$. In 1-D:

\[\frac{d\,\mathrm{ODE}(a_t)}{d a_t}\ =\ \prod_{\text{each step}}\Big(1+\Delta t\,\frac{\partial v}{\partial a}\Big)\]

A long product of numbers (or matrices) is dangerous: if each factor is slightly above 1, the product explodes exponentially; slightly below 1 and it vanishes; worst of all, near a “watershed” (the boundary that decides which peak you finally fall into), a hair’s difference in the start lands the endpoint in a completely different peak — the derivative there is astronomical. This is exactly the source of the spikes on BPTT’s red curve in the previous section. It is both expensive (you must store and backprop the whole chain) and a high-variance, unstable gradient.

Two trajectories start a tiny $\epsilon$ apart, both flowing from bottom to top under the pure flow (no guidance). Most places they stick together (the chain is contracting, the gradient small and stable); but drag the start to the watershed between two peaks and they split off toward different peaks — endpoint gap ÷ start gap is the “amplification,” i.e. the chain Jacobian $\big|\tfrac{d\,\mathrm{ODE}}{da_t}\big|$. Once it explodes, the BPTT gradient explodes.

Ring 3: QGF’s two cuts — one-step approximation + replace the Jacobian with $I$

QGF takes two cuts to remove both problems above at once.

Cut 1: replace the whole chain with “one step.” Instead of running all the denoising steps, use a single Euler step to estimate the clean action $\hat a_1 = a_t+(1-t)\,v_\theta(a_t,t)$. No long chain, so no “product explosion.” Its corresponding Jacobian is:

\[J=\frac{\partial \hat a_1}{\partial a_t}=I+(1-t)\frac{\partial v_\theta}{\partial a_t}\qquad(\text{1-D: } 1+(1-t)\,v')\]

Note that $J$ still requires one differentiation through the velocity field $v_\theta$. And Ring 1 told us: $v_\theta$ is very curvy, so this Jacobian still wanders around and even goes ill-conditioned. So “one step” cut off the worst chain explosion, but $J$ still carries the residual “curviness” of $v_\theta$. The gradient using a one-step estimate + the exact $J$ is $g=J^\top\nabla_{\hat a_1}Q$, which the paper calls QGF-Jacobian.

Cut 2: replace $J$ directly with the identity $I$. The guidance gradient then reduces to $g=\nabla_{\hat a_1}Q(s,\hat a_1)$ — just the direction of “where to get better” of $Q$ at the clean estimate $\hat a_1$, no longer distorted by $J$. Why is this allowed? Because $J$ carries $\partial v_\theta/\partial a_t$ — the very “curviness / ill-conditioning / high variance” we just escaped would sneak back in through it; whereas $I$ is a constant, clean, low-variance direction. The paper finds that dropping $J$ not only saves compute, the resulting gradient is lower-variance and actually better at optimizing $Q$.

All three gradients drawn together; you can peel them back ring by ring: ① full-chain BPTT (red, the most spikes — Ring 2’s chain explosion) → ② cut to one step but keep the exact $J$ (purple, fewer spikes, but $J$’s curviness remains) → ③ drop $J$ = QGF (cyan, smoothest). Drag the probe to watch each one’s jitter $|\Delta g|$; the cyan line is always the steadiest.

Stringing the three rings together gives the complete logic of QGF’s design:

• Ring 1: taking the Jacobian of a very curvy function $v_\theta$ gives a matrix that wanders around and is sometimes squashed into a thin line (ill-conditioned) — it amplifies noise along some direction.
• Ring 2: BPTT must differentiate through the whole denoising chain, i.e. multiply many such Jacobians — so it explodes exponentially at watersheds — expensive and high-variance.
• Ring 3: QGF makes two cuts. Cut 1, “take just one step,” directly kills the chain-product explosion; Cut 2, “$J\to I$,” kills the residual curviness / ill-conditioning from “differentiating $v_\theta$ once.” What remains, $g=\nabla_{\hat a_1}Q$, is cheap, lands near the trustworthy data manifold, and is low-variance.

So, back to that inference algorithm, each line now has its place:

\[\begin{aligned} &\hat a_1 \leftarrow a_t+(1-t)\,v_\theta(s,a_t,t) &&\text{// Cut 1: one-step clean estimate (avoid BPTT's chain explosion)}\\ &g \leftarrow \nabla_{\hat a_1} Q_\phi(s,\hat a_1) &&\text{// Cut 2: }J{=}I\text{, take only }Q\text{'s direction (avoid the ill-conditioned Jacobian)}\\ &a_{t+1}\leftarrow a_t+\tfrac1T\big(v_\theta(s,a_t,t)+\tfrac1\beta\,g\big) &&\text{// velocity + guidance, one step} \end{aligned}\]

The key insight: these two “approximations” are not compromises — each precisely plugs one source of noise — one cut for BPTT’s chain explosion (Ring 2), one for a single Jacobian’s ill-conditioning (Ring 1). With both noise sources removed, what’s left is a gradient that is lower-variance and better at optimizing $Q$ than the “exact” version. That’s why “imprecise” wins.

8. Back to the teaser figure

Now that you have all the background, you can return to the animated teaser at the top and read it again — this time every detail should make sense:

• ① Behavior flow policy: the “chassis” of pure flow denoising, no $Q$ involved, walking from noise along the velocity field $v_t$ all the way to the clean action $a_1$.
• ② BPTT: first roll out (dashed) to $a_1$, take the trustworthy $\nabla Q$ at $a_1$, then multiply a stack of Jacobians to backprop step by step back to $a_t$ — the arrow that finally lands at $a_t$ keeps jittering, the geometric picture of “high variance” (the static original figure can’t show it; animated, it’s obvious at a glance).
• ③ OOD: no rollout at all, the red arrow grows directly on the half-noisy $a_t$, topped with a question mark, pointing every which way — because $Q$ gives no reliable direction on inputs it has never seen.
• ④ QGF: a dashed line jumps one step to $\hat a_1$, takes the blue $\nabla Q$ pointing at the $Q$ peak there, then slides it back to $a_t$ unchanged — this “translate-and-carry” animation is the geometric meaning of $\hat J=I$: no stretching or rotating by any Jacobian, the gradient is copied verbatim.

Drag the center of the contours (the peak of $Q$) and you’ll see BPTT’s endpoint gradient and QGF’s gradient both obediently turn to follow the peak, while OOD’s question-mark arrow does its own thing regardless — “who listens to $Q$, and where” becomes instantly clear.

9. Headline experimental results

• QGF substantially outperforms all prior test-time RL methods, and matches the strongest train-time methods.
• It beats its own Jacobian-keeping variant (QGF-Jacobian) — confirming that dropping $J$ genuinely helps, not just saves compute.
• It gets better as the model grows (the paper reports ~$4\times$ improvement at 3.2M parameters), whereas train-time baselines saturate or even collapse.
• It is insensitive to the critic type — swapping in a stronger TD critic does even better.
• It is orders of magnitude cheaper than best-of-$N$ sampling: on its own it already beats $N{=}4$, and combined it matches $N{=}16$ at far lower compute.

In one sentence

Train a flow policy $v_\theta$ that “generates reasonable actions” by behavior cloning, and a critic $Q_\phi$ that “scores actions” by TD; at test time, at each denoising step, first extrapolate one step to guess the clean action $\hat a_1$, take the gradient of $Q$ on it as the guidance term $\tfrac1\beta\nabla Q(\hat a_1)$, and add it onto the velocity field to step together — so you stay on the data manifold (don’t run off) while bending toward high value (get better), with no further policy training at all. The two “coarse approximations” happen to buy a low-variance, stable, and more $Q$-optimizing guidance gradient — that’s why QGF works.

Paper homepage and code: https://q-guided-flow.github.io/. All interactive demos here are 1-D / 2-D pedagogical simulations, meant to help you “see” the principle; in a real system, a critic’s wild extrapolation on OOD inputs produces exactly the kind of being-fooled-by-a-spurious-bump shown in the demos.