  Medical image classification has developed rapidly under the impetus of the
convolutional neural network (CNN). Due to the fixed size of the receptive
field of the convolution kernel, it is difficult to capture the global features
of medical images. Although the self-attention-based Transformer can model
long-range dependencies, it has high computational complexity and lacks local
inductive bias. Much research has demonstrated that global and local features
are crucial for image classification. However, medical images have a lot of
noisy, scattered features, intra-class variation, and inter-class similarities.
This paper proposes a three-branch hierarchical multi-scale feature fusion
network structure termed as HiFuse for medical image classification as a new
method. It can fuse the advantages of Transformer and CNN from multi-scale
hierarchies without destroying the respective modeling so as to improve the
classification accuracy of various medical images. A parallel hierarchy of
local and global feature blocks is designed to efficiently extract local
features and global representations at various semantic scales, with the
flexibility to model at different scales and linear computational complexity
relevant to image size. Moreover, an adaptive hierarchical feature fusion block
(HFF block) is designed to utilize the features obtained at different
hierarchical levels comprehensively. The HFF block contains spatial attention,
channel attention, residual inverted MLP, and shortcut to adaptively fuse
semantic information between various scale features of each branch. The
accuracy of our proposed model on the ISIC2018 dataset is 7.6% higher than
baseline, 21.5% on the Covid-19 dataset, and 10.4% on the Kvasir dataset.
Compared with other advanced models, the HiFuse model performs the best. Our
code is open-source and available from https://github.com/huoxiangzuo/HiFuse.
