Manual assembly actions segmentation system using temporal-spatial-contact features
Robotic Intelligence and Automation
ISSN: 2754-6969
Article publication date: 21 August 2023
Issue publication date: 13 October 2023
Abstract
Purpose
Accurate segmentation of artificial assembly action is the basis of autonomous industrial assembly robots. This paper aims to study the precise segmentation method of manual assembly action.
Design/methodology/approach
In this paper, a temporal-spatial-contact features segmentation system (TSCFSS) for manual assembly actions recognition and segmentation is proposed. The system consists of three stages: spatial features extraction, contact force features extraction and action segmentation in the temporal dimension. In the spatial features extraction stage, a vectors assembly graph (VAG) is proposed to precisely describe the motion state of the objects and relative position between objects in an RGB-D video frame. Then graph networks are used to extract the spatial features from the VAG. In the contact features extraction stage, a sliding window is used to cut contact force features between hands and tools/parts corresponding to the video frame. Finally, in the action segmentation stage, the spatial and contact features are concatenated as the input of temporal convolution networks for action recognition and segmentation. The experiments have been conducted on a new manual assembly data set containing RGB-D video and contact force.
Findings
In the experiments, the TSCFSS is used to recognize 11 kinds of assembly actions in demonstrations and outperforms the other comparative action identification methods.
Originality/value
A novel manual assembly actions precisely segmentation system, which fuses temporal features, spatial features and contact force features, has been proposed. The VAG, a symbolic knowledge representation for describing assembly scene state, is proposed, making action segmentation more convenient. A data set with RGB-D video and contact force is specifically tailored for researching manual assembly actions.
Keywords
Acknowledgements
This project was supported in part by the Ministry of Science and Technology of China under Grant 2018AAA0102900, the New Generation of Artificial Intelligence Technology Innovation 2030 Major Project, and in part by the National Natural Science Foundation of China under Grant U1913206 and Grant 51975021.
Citation
Kang, Z., Cui, J. and Chu, Z. (2023), "Manual assembly actions segmentation system using temporal-spatial-contact features", Robotic Intelligence and Automation, Vol. 43 No. 5, pp. 509-522. https://doi.org/10.1108/RIA-01-2023-0008
Publisher
:Emerald Publishing Limited
Copyright © 2023, Emerald Publishing Limited