  Multi-modal Record Linkage is the process of matching multi-modal records
from multiple sources that represent the same entity. This field has not been
explored in research and we propose two solutions based on Deep Learning
architectures that are inspired by recent work in Visual Question Answering.
The neural networks we propose use two different fusion modules, the Recurrent
Neural Network + Convolutional Neural Network fusion module and the Stacked
Attention Network fusion module, that jointly combine the visual and the
textual data of the records. The output of these fusion models is the input of
a Siamese Neural Network that computes the similarity of the records. Using
data from the Avito Duplicate Advertisements Detection dataset, we train these
solutions and from the experiments, we concluded that the Recurrent Neural
Network + Convolutional Neural Network fusion module outperforms a simple model
that uses hand-crafted features. We also find that the Recurrent Neural Network
+ Convolutional Neural Network fusion module classifies dissimilar
advertisements as similar more frequently if their average description is
bigger than 40 words. We conclude that the reason for this is that the longer
advertisements have a different distribution then the shorter advertisements
who are more prevalent in the dataset. In the end, we also conclude that
further research needs to be done with the Stacked Attention Network, to
further explore the effects of the visual data on the performance of the fusion
modules.
