for Flow Policy Steering and Acceleration
The Bottleneck in Flow Policies: Vision-Language-Action (VLA) models increasingly rely on flow or diffusion models to generate complex, multimodal robot behaviors. While highly expressive, these models suffer from two major drawbacks: they require many integration steps to generate an action (leading to slow inference), and they frequently fail when faced with out-of-distribution scenarios without expensive supervised fine-tuning (SFT) on new human demonstrations.
The OTQL Solution: Optimal Transport Q-Learning (OTQL) addresses both issues simultaneously. It allows flow-based policies to improve using the robot's own trial-and-error experience while actively accelerating the inference process. By combining conditional optimal transport (COT) with flow matching (FM), OTQL trains the policy to take straight, direct paths to the highest-quality actions.
Illustration of the Optimal Transport Q-Learning (OTQL) alignment process.
Under the Hood: We first train a critic (Q-function) using a standard temporal difference loss to evaluate the quality of the policy's actions. We then calculate the advantage of these actions and use it as an energy function to weight our target distribution—effectively placing more mass on successful, high-reward behaviors.
Next, we solve an approximate conditional optimal transport problem to explicitly pair random noise samples with these high-advantage actions. Finally, the flow policy is optimized using conditional flow matching with samples from the calculated COT coupling:
Because the flow learns to map noise to optimal actions along straight-line trajectories, the policy requires significantly less integration time. In our real-world evaluations, OTQL increased average task success rates from 36% to 86% while reducing the required neural function evaluations (NFEs) per action from 10 down to just 3.
OTQL efficiently steers Vision-Language-Action (VLA) models.
While pre-trained VLAs like SmolVLA show strong generalization, they often struggle with highly specific or out-of-distribution manipulation tasks even when fine-tuned with limited demonstrations. By applying OTQL, we can fine-tune these models using the robot's own experience without collecting more expert demonstrations.
Our approach enables the VLA to adapt to complex new tasks, increasing the average success rate significantly, while decreasing the required inference steps (from 10 NFE to 3), allowing for smooth, real-time robot control.
SmolVLA fine-tuned with SFT and with OTQL.
We compare the base policy trained using 10 demonstrations and the final OTQL checkpoint (after 50 episodes) across three distinct tasks below.
Figure 3: OTQL training for 50 demonstrations on the stack cups task.
We extensively evaluated OTQL against state-of-the-art offline and online RL methods on challenging benchmarks, demonstrating competitive success rates with fewer inference steps.
Offline RL Results
In offline environments, OTQL effectively leverages suboptimal datasets. As shown below, OTQL achieves a leading average success rate of 0.59 compared to key baselines (FAWAC, DSRL, QAM, and FQL). Crucially, OTQL does this while operating with an Inference NFE of just 2, avoiding the costly multi-step integration required by policies like DSRL or QAM. Additionally, as most offline RL methods utilizing flow and diffusion models require sample generation during training, we measure how many times the flow is evaluated at each training step. Overall, we see that OTQL requires the least number of NFE from the flow policy for each training iteration (2 for generating actions for the critic loss, 2 for generating actions for value estimation, 1 for the flow matching loss and we include an extra evaluation to approximate the OT computation cost).
Online and offline-to-online RL Results
Furthermore, OTQL extends naturally to online fine-tuning and offline-to-online adaptation settings. Our method outperforms exisisting few-step methods in offline-to-online settings and performs as well as 10-step methods in online settings.
Simulation results for offline-online RL at the top and online RL at the bottom.