  Electrocardiography (ECG) signals can be considered as multi-variable
time-series. The state-of-the-art ECG data classification approaches, based on
either feature engineering or deep learning techniques, treat separately
spectral and time domains in machine learning systems. No spectral-time domain
communication mechanism inside the classifier model can be found in current
approaches, leading to difficulties in identifying complex ECG forms. In this
paper, we proposed a novel deep learning model named Spectral Cross-domain
neural network (SCDNN) with a new block called Soft-adaptive threshold spectral
enhancement (SATSE), to simultaneously reveal the key information embedded in
spectral and time domains inside the neural network. More precisely, the
domain-cross information is captured by a general Convolutional neural network
(CNN) backbone, and different information sources are merged by a self-adaptive
mechanism to mine the connection between time and spectral domains. In SATSE,
the knowledge from time and spectral domains is extracted via the Fast Fourier
Transformation (FFT) with soft trainable thresholds in modified Sigmoid
functions. The proposed SCDNN is tested with several classification tasks
implemented on the public ECG databases \textit{PTB-XL} and \textit{MIT-BIH}.
SCDNN outperforms the state-of-the-art approaches with a low computational cost
regarding a variety of metrics in all classification tasks on both databases,
by finding appropriate domains from the infinite spectral mapping. The
convergence of the trainable thresholds in the spectral domain is also
numerically investigated in this paper. The robust performance of SCDNN
provides a new perspective to exploit knowledge across deep learning models
from time and spectral domains. The repository can be found:
https://github.com/DL-WG/SCDNN-TS
