  In-place gesture-based virtual locomotion techniques enable users to control
their viewpoint and intuitively move in the 3D virtual environment. A key
research problem is to accurately and quickly recognize in-place gestures,
since they can trigger specific movements of virtual viewpoints and enhance
user experience. However, to achieve real-time experience, only short-term
sensor sequence data (up to about 300ms, 6 to 10 frames) can be taken as input,
which actually affects the classification performance due to limited
spatio-temporal information. In this paper, we propose a novel long-term memory
augmented network for in-place gestures classification. It takes as input both
short-term gesture sequence samples and their corresponding long-term sequence
samples that provide extra relevant spatio-temporal information in the training
phase. We store long-term sequence features with an external memory queue. In
addition, we design a memory augmented loss to help cluster features of the
same class and push apart features from different classes, thus enabling our
memory queue to memorize more relevant long-term sequence features. In the
inference phase, we input only short-term sequence samples to recall the stored
features accordingly, and fuse them together to predict the gesture class. We
create a large-scale in-place gestures dataset from 25 participants with 11
gestures. Our method achieves a promising accuracy of 95.1% with a latency of
192ms, and an accuracy of 97.3% with a latency of 312ms, and is demonstrated to
be superior to recent in-place gesture classification techniques. User study
also validates our approach. Our source code and dataset will be made available
to the community.
