  With the advancement in computing and robotics, it is necessary to develop
fluent and intuitive methods for interacting with digital systems,
augmented/virtual reality (AR/VR) interfaces, and physical robotic systems.
Hand motion recognition is widely used to enable these interactions. Hand
configuration classification and MCP joint angle detection is important for a
comprehensive reconstruction of hand motion. sEMG and other technologies have
been used for the detection of hand motions. Forearm ultrasound images provide
a musculoskeletal visualization that can be used to understand hand motion.
Recent work has shown that these ultrasound images can be classified using
machine learning to estimate discrete hand configurations. Estimating both hand
configuration and MCP joint angles based on forearm ultrasound has not been
addressed in the literature. In this paper, we propose a CNN based deep
learning pipeline for predicting the MCP joint angles. The results for the hand
configuration classification were compared by using different machine learning
algorithms. SVC with different kernels, MLP, and the proposed CNN have been
used to classify the ultrasound images into 11 hand configurations based on
activities of daily living. Forearm ultrasound images were acquired from 6
subjects instructed to move their hands according to predefined hand
configurations. Motion capture data was acquired to get the finger angles
corresponding to the hand movements at different speeds. Average classification
accuracy of 82.7% for the proposed CNN and over 80% for SVC for different
kernels was observed on a subset of the dataset. An average RMSE of 7.35
degrees was obtained between the predicted and the true MCP joint angles. A low
latency (6.25 - 9.1 Hz) pipeline has been proposed for estimating both MCP
joint angles and hand configuration aimed at real-time control of human-machine
interfaces.
