SmoothRL: Efficient Online RL During Asynchronous Execution
Technical ReportSmoothRL is an online reinforcement learning framework designed for real-world robot manipulation under asynchronous inference and deployment. It addresses the temporal mismatch introduced by asynchronous inference in large-scale robot models, while improving motion smoothness during policy updates.
For real-world robots, asynchronous inference is not merely an engineering choice — it is an inevitable deployment paradigm for large-scale models: inference, execution, and environmental feedback are inherently asynchronous. Conventional online RL, built around synchronous inference assumptions, is difficult to directly apply to real-world deployment.
SmoothRL closes the loop across robot interaction, data collection, asynchronous inference, and online optimization. It explicitly partitions action chunks and updates gradients only over the executed region, aligning training with the actual deployment timeline. It further leverages the pretrained policy as a trajectory prior, uses a lightweight sub-policy for efficient action optimization, and supports real-time human intervention.
Real-robot experiments show that SmoothRL enables online RL while the robot continues operating, simultaneously improving task success and motion smoothness — enabling policies to continuously evolve in the real world.
Core Components
SmoothRL addresses the temporal mismatch in online reinforcement learning under asynchronous deployment with an online optimization framework closely aligned with the robot's actual execution process. It consists of three key mechanisms:
Execution-Aware Gradient Updates
Each action chunk is divided into committed, execution, and discarded regions. Gradients are updated only over the actions that are actually executed, strictly aligning optimization with real-world execution.
Full Temporal Value Modeling
Actions in the committed region are incorporated into value estimation, providing the critic with the complete temporal context of the action sequence.
Trajectory Continuity Constraint
Historical trajectories constrain newly generated action chunks, suppressing abrupt changes in policy outputs and ensuring smooth transitions.
Together, these mechanisms align training with the actual execution timeline, enabling stable online RL under asynchronous real-world deployment.
Human Intervention Modes
Human intervention can significantly improve the efficiency of online reinforcement learning in the real world. SmoothRL supports two intervention modes tailored to different task settings, both triggered in real time through teleoperation devices. Intervention does not interrupt the learning loop: the policy continues to infer, while only the final commands sent to the robot are modified. The resulting trajectories can be directly reused for online training.
Absolute Intervention
VR teleoperation directly takes over the robot and fully replaces the policy output, making it suitable for high-precision, fine-grained manipulation.
Residual Intervention
A joystick generates action corrections that are added to the policy output, preserving the original trajectory and temporal structure while enabling real-time adjustments. This is suited for highly dynamic interactive tasks.
Together, the two modes support both precise control and real-time correction, seamlessly incorporating human expertise into the real-world online learning loop.
Experimental Evaluation
We evaluate SmoothRL on the Astribot-S1 mobile bimanual robot. The base policy is a task-finetuned π0.5, with an action output frequency of 30 Hz, an inference frequency fixed at 5 Hz, and a latency budget of 6 frames.
The evaluation covers two scenarios — dynamic manipulation and high-precision bimanual manipulation — across three tasks:
Dynamic Tossing
Grasp an object and throw it into a target container, testing high-speed dynamic manipulation.
Pen Capping
Align a pen with its cap, secure the cap onto the pen, and release the cap, testing precise alignment and bimanual coordination.
Box Opening
Stabilize the box with the left hand, cut the sealing tape with the right hand, and open the lid, testing bimanual coordination and precise manipulation.
Policy Behavior Before and After SmoothRL
Left: The frozen base policy fails to complete the task. Right: The same policy, fine-tuned with SmoothRL, successfully completes the task. Both experiments are conducted under identical environmental conditions.
Evaluation Protocol
The translucent regions in the figure indicate the distribution of initial states sampled for the robot and objects, testing the policy's generalization and robustness across diverse initial conditions.
Success Rate Progress
| Rollout episodes collected | Dynamic Tossing | Pen Capping | Box Opening |
|---|---|---|---|
| 0 | 39% | 8% | 30% |
| 150 | 72% | 67% | 20% |
| 200 | 83% | 75% | 40% |
| 250 | 94% | 83% | 90% |
The curves show the success rate progression of each task throughout online reinforcement learning. The x-axis represents the number of rollout episodes collected. 0 corresponds to the frozen pretrained VLA policy, while the dashed lines indicate the baseline success rates. Each data point reports the average success rate over multiple repeated evaluations, with all performance gains achieved through a single continuous online fine-tuning run.
Conclusion & Future Work
SmoothRL is an online reinforcement learning framework for real-world robots powered by large-scale generalist models. It is designed for asynchronous inference and deployment, with human intervention seamlessly integrated into the learning loop. It addresses the temporal mismatch between policy learning and actual robot execution, continuously optimizing the policy through real-world interaction feedback. SmoothRL improves performance on both high-precision and highly dynamic tasks while maintaining smooth and natural robot motion, providing a practical path toward continuous learning and evolution for robots in the real world.
Looking ahead, we will further explore the joint optimization of pretrained policies and online reinforcement learning, while expanding SmoothRL to a broader range of robots, policy architectures, and real-world deployment scenarios. Our goal is to enable robots to continuously learn from real-world interactions and progressively develop stronger capabilities for autonomous adaptation and continual improvement.
