Dexterous manipulation has seen significant progress through sample-efficient reinforcement learning methods like RLPD and SERL. These approaches fuse expert demonstrations with online exploration data into a shared replay buffer, which is then used to train an offline RL algorithm such as SAC or TD3.
While these methods train policies from scratch, our goal is different: fine-tuning an existing behavior-cloning policy. State-of-the-art behavior-cloning policies like diffusion and flow-matching models are expressive but face challenges in RL fine-tuning due to unstable backpropagation and the lack of an exact log-likelihood, which is essential for likelihood regularization.
The SERNF framework[1] addresses this using normalizing flow policies, which provide multimodal predictions, exact log-likelihoods, and stable backpropagation. Their invertible architecture enables conservative policy updates during RL fine-tuning.
My role involved two key contributions. First, I replaced the coupling layers in SERNF with autoregressive flows, a model class originally developed for image generation. This change improved expressivity and convergence, yielding a ten percent performance gain on the Square task compared to the original coupling-layer architecture.
Second, I evaluated several vision backbones including DINO, R3M, and JEPA, and also studied the effect of fine-tuning DINO. Our results showed that DINO consistently outperformed the other backbones, and that fine-tuning it further boosted performance.
Experiments were conducted both on the Robomimic simulation benchmark and on two real Franka Panda arms equipped with ORCA hands, demonstrating that our contributions generalize beyond simulation to challenging real-world dexterous manipulation tasks.
This work was done at the Soft Robotics Lab at ETH Zürich under the supervision of Chenyu Yang and Denis Tarasov, and built upon their SERNF paper.
Bimanual dexterous manipulation teleoperation, real Franka Panda arms with ORCA hands.
Bimanual dexterous manipulation teleoperation, real Franka Panda arms with ORCA hands.