CoRAL: Adaptive LLM-based Robot Control Framework for Contact-Rich Manipulation

CoRAL (Contact-Rich Adaptive LLM-based Control) is an adaptive LLM-based robot control framework for contact-rich manipulation. Its core design uses LLMs as cost function designers instead of direct controllers, combined with a neuro-symbolic adaptation loop and a retrieval memory mechanism, to achieve zero-shot robot manipulation planning in contact-rich scenarios. Experimental validation shows that this framework increases the success rate by more than 50% compared to existing baselines in unseen contact-rich scenarios and effectively handles the simulation-to-reality transfer problem.

Large Language Models (LLMs) and Vision-Language Models (VLMs) excel in high-level reasoning and semantic understanding, but they face fundamental challenges when directly applied to contact-rich robot manipulation tasks: lack of explicit physical grounding and inability to perform adaptive control. Contact-rich manipulation refers to tasks that require complex physical interactions with the environment (e.g., grasping, pushing, pulling, flipping objects), which demand controllers to respond to force changes in real time. Traditional end-to-end strategies that treat LLMs as black-box controllers perform poorly in dynamic contact scenarios.

CoRAL adopts a modular design that decouples high-level reasoning from low-level control. The key insight is that LLMs should not directly output control commands but act as 'cost designers' to synthesize context-aware objective functions for sampling-based motion planners (MPPI). The advantages of this design include: retaining the semantic understanding and task planning advantages of LLMs; delegating control execution to specialized optimizers to ensure real-time performance and stability; integrating multi-source information such as visual semantics, physical parameters, and interaction feedback through the intermediate representation of cost functions.

CoRAL's neuro-symbolic adaptation loop aims to resolve the ambiguity of physical parameters in visual data. The workflow is as follows: VLMs extract semantic priors (e.g., estimates of object mass and friction coefficients) from visual inputs; online system identification refines and corrects these estimates during real-time interactions; LLMs iteratively adjust the structure of cost functions based on interaction feedback to correct strategy-level errors. This layered processing addresses the uncertainty of pure visual estimation—semantic priors provide initial guesses, and actual physical interactions provide correction signals.

CoRAL integrates a retrieval-based memory unit for storing and reusing successful manipulation strategies. When faced with similar task scenarios, the system can retrieve past experiences to accelerate the planning process and improve success rates. This embodies the principles of experience accumulation and value reuse in agent learning—it can significantly reduce computational overhead for repeated tasks while maintaining adaptability to new scenarios.

CoRAL's validation covers both simulation environments and real hardware. The research team designed challenging novel tasks (e.g., using external contact to flip an object against a wall). Experimental results: In unseen contact-rich scenarios, the average success rate is more than 50% higher than the state-of-the-art VLA models and foundation model-based planner baselines; it effectively handles the simulation-to-reality transfer problem. The key to performance improvement lies in the layered architecture: high-level semantic reasoning understands task objectives, mid-level cost functions integrate multi-source information, and low-level control ensures real-time response.

CoRAL provides technical insights for the intersection of robotics and AI: the application of LLMs in the physical world does not have to be limited to the end-to-end black-box model; a reasonable architecture design can fully leverage their semantic understanding capabilities while meeting the real-time and stability requirements of physical control. Future directions include: expanding the layered architecture to handle more complex manipulation tasks; integrating more physical prior knowledge into cost function design; and improving the system's generalization ability in open-world environments.