  Deep convolutional neural networks (CNNs) have been widely used in various
medical imaging tasks. However, due to the intrinsic locality of convolution
operation, CNNs generally cannot model long-range dependencies well, which are
important for accurately identifying or mapping corresponding breast lesion
features computed from unregistered multiple mammograms. This motivates us to
leverage the architecture of Multi-view Vision Transformers to capture
long-range relationships of multiple mammograms from the same patient in one
examination. For this purpose, we employ local Transformer blocks to separately
learn patch relationships within four mammograms acquired from two-view
(CC/MLO) of two-side (right/left) breasts. The outputs from different views and
sides are concatenated and fed into global Transformer blocks, to jointly learn
patch relationships between four images representing two different views of the
left and right breasts. To evaluate the proposed model, we retrospectively
assembled a dataset involving 949 sets of mammograms, which include 470
malignant cases and 479 normal or benign cases. We trained and evaluated the
model using a five-fold cross-validation method. Without any arduous
preprocessing steps (e.g., optimal window cropping, chest wall or pectoral
muscle removal, two-view image registration, etc.), our four-image
(two-view-two-side) Transformer-based model achieves case classification
performance with an area under ROC curve (AUC = 0.818), which significantly
outperforms AUC = 0.784 achieved by the state-of-the-art multi-view CNNs (p =
0.009). It also outperforms two one-view-two-side models that achieve AUC of
0.724 (CC view) and 0.769 (MLO view), respectively. The study demonstrates the
potential of using Transformers to develop high-performing computer-aided
diagnosis schemes that combine four mammograms.
