To read this content please select one of the options below:

Long-term robot manipulation task planning with scene graph and semantic knowledge

Runqing Miao (Beijing University of Posts and Telecommunications, Beijing, China)
Qingxuan Jia (Beijing University of Posts and Telecommunications, Beijing, China)
Fuchun Sun (Tsinghua University, Beijing, China)

Robotic Intelligence and Automation

ISSN: 2754-6969

Article publication date: 25 January 2023

Issue publication date: 28 March 2023




Autonomous robots must be able to understand long-term manipulation tasks described by humans and perform task analysis and planning based on the current environment in a variety of scenes, such as daily manipulation and industrial assembly. However, both classical task and motion planning algorithms and single data-driven learning planning methods have limitations in practicability, generalization and interpretability. The purpose of this work is to overcome the limitations of the above methods and achieve generalized and explicable long-term robot manipulation task planning.


The authors propose a planning method for long-term manipulation tasks that combines the advantages of existing methods and the prior cognition brought by the knowledge graph. This method integrates visual semantic understanding based on scene graph generation, regression planning based on deep learning and multi-level representation and updating based on a knowledge base.


The authors evaluated the capability of this method in a kitchen cooking task and tabletop arrangement task in simulation and real-world environments. Experimental results show that the proposed method has a significantly improved success rate compared with the baselines and has excellent generalization performance for new tasks.


The authors demonstrate that their method is scalable to long-term manipulation tasks with varying complexity and visibility. This advantage allows their method to perform better in new manipulation tasks. The planning method proposed in this work is meaningful for the present robot manipulation task and can be intuitive for similar high-level robot planning.



The project number is 2021B0101410002. The project title is “Autonomous Learning of Complex Skills by multi-degree-of-freedom Agents, Key Components and Demonstration Application in 3C Manufacturing Industry”.


Miao, R., Jia, Q. and Sun, F. (2023), "Long-term robot manipulation task planning with scene graph and semantic knowledge", Robotic Intelligence and Automation, Vol. 43 No. 1, pp. 12-22.



Emerald Publishing Limited

Copyright © 2022, Emerald Publishing Limited

Related articles