HandX: A Unified Foundation Framework for Bimanual Interaction Motion Generation

The HandX project constructs a unified foundation framework covering data, annotation, and evaluation, focusing on generating realistic bimanual interaction motions and addressing the shortcomings of existing full-body models in capturing fine finger movements. Through its trinity architecture (data layer integration and creation, LLM-driven decoupling in the annotation layer, and hand-specific metrics in the evaluation layer), this framework provides a complete ecosystem for bimanual interaction motion generation research, with application prospects in robotics learning, VR/AR, animation production, and other fields, and relevant resources have been made open.

Current human motion generation research mainly focuses on large-scale full-body movements (such as walking and running), but ignores key cues like fine control of finger joints, timing of contact, and bimanual coordination, leading to poor performance in fine manipulation scenarios like twisting a bottle cap and tying shoelaces. The root cause lies in the scarcity of high-quality captured data for bimanual interaction motions—existing datasets lack details on finger dynamics or bimanual collaboration scenarios. Additionally, semantic annotation of hand motions is complex, requiring detailed information like the degree of finger bending and contact point positions, which existing annotation systems struggle to meet.

The HandX framework includes three core dimensions:

1. Data Layer: Integrates and filters existing public datasets while creating new datasets covering bimanual interaction scenarios, focusing on fine details like finger joint angles, contact points, and spatial relationships between hands;
2. Annotation Layer: Adopts a decoupling strategy—first extracts quantitative features like contact events and finger bending degrees, then uses large language models (LLMs) to convert these features into rich semantic descriptions (e.g., "The right index finger lightly touches the edge of the cup with its pad, preparing to apply force"), offering strong scalability;
3. Evaluation Layer: Designs hand-specific metrics to comprehensively assess generation quality from dimensions like finger joint angle accuracy, bimanual coordination level, temporal correctness of contact events, and semantic coherence.

Based on HandX data and annotations, benchmark tests were conducted on diffusion models and autoregressive models (covering modes like text description, target pose, and action category control). The results show that the models generate high-quality dexterous hand motions, with significant improvements in all hand-specific metrics. A scaling effect was also observed: when the model parameter size expands from basic to large scale, the finger joint angle error decreases by about 30%, and the consistency of bimanual coordination increases by 25%. Especially in fine manipulation tasks, the motions are more smooth and natural, echoing the Scaling Law of large language models.

HandX brings new possibilities to multiple fields:

• Robotics Learning: Helps robots understand human manipulation skills and learn dexterous grasping strategies;
• VR/AR: Enhances the expressiveness of virtual avatars and enables natural and complex gesture operations;
• Animation Production: Reduces the workload of manual fine hand animation, allowing animators to focus on creativity. The research team has publicly released the HandX dataset (including motion data, semantic annotations, and evaluation tools) to promote progress in the field.

HandX is an important step forward in human motion generation towards fine and complex scenarios. By constructing a complete framework of data, annotation, and evaluation, it lays a solid foundation for bimanual interaction motion generation. The discovery of the scaling effect indicates that expanding the scale of models and data remains an effective way to improve performance in this field. Paper link: http://arxiv.org/abs/2603.28766v1