To read this content please select one of the options below:

Adaptive sensor fusion labeling framework for hand pose recognition in robot teleoperation

Wen Qi (Department of Electronic, Information and Bioengineering, Politecnico di Milano, Milan, Italy)
Xiaorui Liu (Institute for Future, Qingdao University, Qingdao, China)
Longbin Zhang (KTH Royal Institute of Technology, Stockholm, Sweden)
Lunan Wu (State Key Laboratory of Robotics and System, Harbin Institute of Technology, Harbin, China)
Wenchuan Zang (College of Information Science and Engineering, Ocean University of China, Qingdao, China)
Hang Su (Dipartimento di Elettronica Informazione e Bioingegneria, Politecnico di Milano, Milano, Italy)

Assembly Automation

ISSN: 0144-5154

Article publication date: 15 February 2021

Issue publication date: 22 July 2021




The purpose of this paper is to mainly center on the touchless interaction between humans and robots in the real world. The accuracy of hand pose identification and stable operation in a non-stationary environment is the main challenge, especially in multiple sensors conditions. To guarantee the human-machine interaction system’s performance with a high recognition rate and lower computational time, an adaptive sensor fusion labeling framework should be considered in surgery robot teleoperation.


In this paper, a hand pose estimation model is proposed consisting of automatic labeling and classified based on a deep convolutional neural networks (DCNN) structure. Subsequently, an adaptive sensor fusion methodology is proposed for hand pose estimation with two leap motions. The sensor fusion system is implemented to process depth data and electromyography signals capturing from Myo Armband and leap motion, respectively. The developed adaptive methodology can perform stable and continuous hand position estimation even when a single sensor is unable to detect a hand.


The proposed adaptive sensor fusion method is verified with various experiments in six degrees of freedom in space. The results showed that the clustering model acquires the highest clustering accuracy (96.31%) than other methods, which can be regarded as real gestures. Moreover, the DCNN classifier gets the highest performance (88.47% accuracy and lowest computational time) than other methods.


This study can provide theoretical and engineering guidance for hand pose recognition in surgery robot teleoperation and design a new deep learning model for accuracy enhancement.



Qi, W., Liu, X., Zhang, L., Wu, L., Zang, W. and Su, H. (2021), "Adaptive sensor fusion labeling framework for hand pose recognition in robot teleoperation", Assembly Automation, Vol. 41 No. 3, pp. 393-400.



Emerald Publishing Limited

Copyright © 2020, Emerald Publishing Limited

Related articles