  Non-invasive brain-computer interfaces help the subjects to control external
devices by brain intentions. The multi-class classification of upper limb
movements can provide external devices with more control commands. The onsets
of the upper limb movements are located by the external limb trajectory to
eliminate the delay and bias among the trials. However, the trajectories are
not recorded due to the limitation of experiments. The delay cannot be avoided
in the analysis of signals. The delay negatively influences the classification
performance, which limits the further application of upper limb movements in
the brain-computer interface. This work focuses on multi-channel brain signals
analysis in the temporal-frequency approach. It proposes the two-stage-training
temporal-spectral neural network (TTSNet) to decode patterns from brain
signals. The TTSNet first divides the signals into various filter banks. In
each filter bank, task-related component analysis is used to reduce the
dimension and reject the noise of the brain. A convolutional neural network
(CNN) is then used to optimize the temporal characteristic of signals and
extract class-related features. Finally, these class-related features from all
filter banks are fused by concatenation and classified by the fully connected
layer of the CNN. The proposed method is evaluated in two public datasets. The
results show that TTSNet has an improved accuracy of 0.7456$\pm$0.1205 compared
to the EEGNet of 0.6506$\pm$0.1275 ($p<0.05$) and FBTRCA of 0.6787$\pm$0.1260
($p<0.1$) in the movement detection task, which classifies the movement state
and the resting state. The proposed method is expected to help detect limb
movements and assist in the rehabilitation of stroke patients.
