Umeå University's logo

The use of deep neural networks (DNNs) in medical images has enabled the development of solutions characterized by the need of leveraging information coming from multiple sources, raising the Multimodal Deep Learning. DNNs are known for their ability to provide hierarchical and high-level representations of input data. This capability has led to the introduction of methods performing data fusion at an intermediate level, preserving the distinctiveness of the heterogeneous sources in modality-specific paths, while learning the way to define an effective combination in a shared representation. However, modeling the intricate relationships between different data remains an open issue. In this paper, we aim to improve the integration of data coming from multiple sources. We introduce between layers belonging to different modality-specific paths a Transfer Module (TM) able to perform the cross-modality calibration of the extracted features, reducing the effects of the less discriminative ones. As case of study, we focus on the axillary lymph nodes metastasis evaluation in malignant breast cancer, a crucial prognostic factor, affecting patient’s survival. We propose a Multi-Input Single-Output 3D Convolutional Neural Network (CNN) that considers both images acquired with multiparametric Magnetic Resonance and clinical information. In particular, we assess the proposed methodology using four architectures, namely BasicNet and three ResNet variants, showing the improvement of the performance obtained by including the TM in the network configuration. Our results achieve up to 90% and 87% of accuracy and Area under ROC curve, respectively when the ResNet10 is considered, surpassing various fusion strategies proposed in the literature.