Surprise! We can teach ‘bi-manual’ robots how to stir-fry food to pure bliss

robotics is make big strides in all sorts of areas, including some quite unusual ones. Researchers from the Idiap Research Institute in Switzerland, the Chinese University of Hong Kong (CUHK) and Wuhan University (WHU) have now developed a machine learning-based method to teach robots to stir-fry like professional chefs, according to a report TechXplore published Friday.

Intelligent robots that can prepare food

“Our recent work is the joint effort of three labs: the Robot Learning & Interaction group led by Dr. Sylvain Calinon of the Idiap Research Institute and the Collaborative and Versatile Robots lab led by Prof. Fei Chen Cuhk and the lab led by Prof. Miao Li at WHU,” Junjia Liu, one of the researchers who conducted the study, told TechXplore

“Our three labs have been studying and working together for about ten years. We are particularly interested in making intelligent robots that can prepare food for humans.”

The new research hopes to create a robotic chef, something that has been very difficult to achieve until now.

“While domestic service robots have evolved considerably in recent years, creating a robotic chef in the semi-structured kitchen environment remains a major challenge,” said Liu.

“Food preparation and cooking are two crucial household activities, and a robot chef that can follow random recipes and cook automatically would be practical and bring a new interactive entertainment experience.”

To perform such a complex task as stir-frying, Liu and his team first had to train a bimanual coordination model known as a “structured transformer.” They did this with the help of human demonstrations.

“This mechanism sees coordination as a sequence transduction problem between the movements of both arms and uses a combined transformer and GNN model to achieve this,” explains Liu.

“For example, in the online process, the left arm movement is adjusted based on the visual feedback and the corresponding right arm movement is generated by the pre-trained structured transformer model based on the left arm movement.”

Cooking both at home and in public

Liu now hopes that his new and improved model can one day introduce the development of robots that can prepare meals both at home and in public. It can also be used in developing robots that can perform other tasks that require two arms and hands. A good example is this already popular pizza making robot

“We will now introduce higher dimensional information to teach more humanoid movements in kitchen skills, such as visual and electromyographic cues,” Liu concluded.

“The estimation of semi-fluid content in this work was simplified as two-dimensional image segmentation, and we used only the relative displacement as the desired target. Thus, we also intend to propose a more comprehensive framework consisting of both the motions of bimanual manipulators and the change of state of the object.”

The results of the study were published in the magazine IEEE Robotics and Automation Letters.

This letter describes an approach to stir-fry well-known Chinese cuisine on a bimanual robotic system. Stir-frying requires a sequence of highly dynamic coordinated movements, which are usually difficult for a chef to learn, let alone transfer to robots. In this letter, we define a canonical stir-fry motion and then propose a decoupled framework for learning this deformable object manipulation from human demonstration. First, the robot’s double arms are decoupled into different roles (a leader and a follower) and taught separately using classical and neural network-based methods, after which the bimanual task is turned into a coordination problem. To obtain generalized bimanual coordination, we secondly propose a Graph and Transformer based model – Structured-Transformer, to capture the space-time relationship between two-armed motions. Finally, by adding visual feedback of content distortion, our framework can automatically adjust the movements to achieve the desired stir-fry effect. We verify the framework through a simulator and implement it on a real Panda bimanual robot system. The experimental results validate that our framework can realize the bimanual robotic stir fry motion and has the potential to extend to other deformable objects with bimanual coordination.

Leave a Comment

Your email address will not be published.