RL2-VLA: Adaptive RL Latent Compositional Steering with Test-Time Scaling for Vision-Language-Action Models

Derek Ming Siang Tan1,3,† Shailesh Shailesh1,† Srikrishna Iyer3 William Wei Jie Teo1,3 Yuanliang Ju2 Qiao Gu2 Guillaume Sartoretti1
1 National University of Singapore  ·  2 University of Toronto  ·  3 Singapore Technologies Engineering
† Equal contribution in no particular order
Paper Code Models Video
National University of Singapore University of Toronto Singapore Technologies Engineering
Overview of RL2: test-time action error scaling law and real-robot compositional steering
Overview of RL2. RL2 relies on base VLA samples to approach the task object, then adaptively applies RL compositional steering to diversify actions toward successful states — when failure is preemptively detected.
15
Manipulation tasks
across 3 benchmarks
+14.7% (avg +10.1%)
Task success · SIMPLER
π0, OOD prompts / envs
+17.3% (avg +8.9%)
Task success · PolaRiS
π0.5, OOD prompts
+26.7% (avg +19.5%)
Task success · Real robot
π0.5, OOD prompts / envs - PiperX
+28.7% (avg +18.7%)
Scaling samples + diversity
π0, OOD Prompts
Abstract

Despite the impressive visuomotor capabilities of Vision-Language-Action (VLA) models, their performance often degrades on challenging and out-of-domain tasks. We introduce RL2, an adaptive inference-time steering framework that leverages Reinforcement Learning on VLA Latents. We train a lightweight offline RL flow-matching policy conditioned on expressive latents from the VLA action expert, and steers the base VLA by composing their action expert flow velocities during inference. We further discover that inference-time steering follows fundamentally different scaling laws under success and failure states — action diversity is most beneficial when the base VLA is likely to fail — RL2 therefore activates compositional steering only when failure is predicted.

RL2 in Action

Adaptive RL Compositional Steering

The baselines struggle while RL2 succeeds via adaptive steering. The Failure Detection panel shows when the failure probability crosses the conformal (CP) band and applies compositional steering (click on numbered keypoints for explanations).

0:00 / 0:00
Method

Overall architecture

1 / 4
The RL2 pipeline. RL2 leverages expressive VLA action expert latents to perform RL compositional steering adaptively during failure, diversifying action candidates towards successful states for verifier selection.
Results

Consistent gains across simulation and hardware

Key Insight
Diversification is most useful during failure (e.g. RL2, RBF), but may degrade already-accurate actions during success.
Test-Time Scaling Laws
Fitted power law e ≈ a·kb · oracle action error (e) vs. action samples (k)
Failure states
Success states

Test-Time Scaling analysis using oracle verifier & π0 (BridgeV2 Dataset). Compare the panels: RL2 (green) drives error lowest during failure yet stays high during success — motivating adaptive steering.

PCA Heatmap Analysis
Action density vs. ground-truth action · VLA (orange) vs. RL2 steered (green)
PCA analysis heatmaps: VLA (orange, top row) vs RL² steered (green, bottom row) action density relative to the ground-truth action (×)

During failure states, RL2 steers the action distribution (green) closer to the ground-truth action than VLA (orange).

Key Insight
On OOD prompts and environments, adaptive RL2 achieves up to +17.3% in simulation and +26.7% on-robot, over the strongest Rephrase baseline.
Benchmark Results
· success rate (%)
Scaling Quantity and Diversity
π0 · In-Domain SIMPLER (OOD Prompt) · avg success rate (%)

Scaling sample diversity and quantity (8 rephrases × 5 samples) with compositional steering improves success rates by up to +18.6% over single-sample inference.

BibTeX

  

  

    
Acknowledgments

    
We thank our collaborators across the National University of Singapore, the University of Toronto, and Singapore Technologies Engineering for their support and helpful discussions. We are grateful to Sherwin, Karthee, and Dennis from ST Engineering for providing the resources that made our real-robot experiments possible. We also thank the open-source community behind CoVer, RoboMonkey, SAFE, and other related work for making this research possible.