Umeå University's logo

Deep learning has revolutionized biomedical research by providing sophisticated methods to handle complex, high-dimensional data. Multimodal deep learning (MDL) further enhances this capability by integrating diverse data types such as imaging, textual data, and genetic information, leading to more robust and accurate predictive models. In MDL, differently from early and late fusion methods, intermediate fusion stands out for its ability to effectively combine modality-specific features during the learning process. This systematic review comprehensively analyzes and formalizes current intermediate fusion methods in biomedical applications, highlighting their effectiveness in improving predictive performance and capturing complex inter-modal relationships. We investigate the techniques employed, the challenges faced, and potential future directions for advancing intermediate fusion methods. Additionally, we introduce a novel structured notation that standardizes intermediate fusion architectures, enhancing understanding and facilitating implementation across various domains. Our findings provide actionable insights and practical guidelines intended to support researchers, healthcare professionals, and the broader deep learning community in developing more sophisticated and insightful multimodal models. Through this review, we aim to provide a foundational framework for future research and practical applications in the dynamic field of MDL.

Higher-Level Gait Disorder (HLGD) is a type of gait disorder estimated to affect up to 6% of the older population. By definition, its symptoms originate from the higher-level nervous system, yet its association with brain morphology remains unclear. This study hypothesizes that there are patterns in brain morphology linked to HLGD. For the first time in the literature, this work investigates whether deep learning, in the form of convolutional neural networks, can capture patterns in magnetic resonance images to identify individuals affected by HLGD. To handle this new classification task, we propose setting up an ensemble of models. This leverages the benefits of combining classifiers instead of determining which network is the most suitable, developing a new architecture, or customizing an existing one. We introduce a computationally cost-effective search algorithm to find the optimal ensemble by leveraging a cost function of both traditional performance scores and the diversity among the models. Using a unique dataset from a large population-based cohort (VESPR), the ensemble identified by our algorithm demonstrated superior performance compared to single networks, other ensemble fusion techniques, and the best linear radiological measure. This emphasizes the importance of implementing diversity into the cost function. Furthermore, the results indicate significant morphological differences in brain structure between HLGD-affected individuals and controls, motivating research about which areas the networks base their classifications on, to get a better understanding of the pathophysiology of HLGD.

The medical field faces significant data shortages due to the high image acquisition and maintenance costs. Data Augmentation aims to mitigate this by increasing data availability and enhancing image generalization. However, traditional DA methods often produce data with limited quality and diversity. Generative Adversarial Networks present a promising alternative, offering potential solutions to data scarcity issues. This paper evaluates the impact of GAN-based data augmentation in medical imaging and provides a benchmark for the efficacy of synthetic data in downstream classification tasks. To this aim, we performed a wide set of tests using three different GAN architectures on six 2D datasets from the standardized MedMNIST biomedical image collection, conducting a total of 696 experiments. Our results reveal that while GAN-based DA methods show promise with low-dimensional datasets, traditional DA methods still outperform them.

Alzheimer's disease (AD) poses significant global health challenges due to its increasing prevalence and associated societal costs. Early detection and diagnosis of AD are critical for delaying progression and improving patient outcomes. Traditional diagnostic methods and single-modality data often fall short in identifying early-stage AD and distinguishing it from Mild Cognitive Impairment (MCI). This study addresses these challenges by introducing a novel approach: multImodal enseMble via class BALancing diversity for iMbalancEd Data (IMBALMED). IMBALMED integrates multimodal data from the Alzheimer's Disease Neuroimaging Initiative database, including clinical assessments, neuroimaging phenotypes, biospecimen, and subject characteristics data. It employs a new ensemble of model classifiers, designed specifically for this framework, which combines eight distinct families of learning paradigms trained with diverse class balancing techniques to overcome class imbalance and enhance model accuracy. We evaluate IMBALMED on two diagnostic tasks (binary and ternary classification) and four binary early detection tasks (at 12, 24, 36, and 48 months), comparing its performance with state-of-the-art algorithms and an unbalanced dataset method. To further validate the proposed model and ensure genuine generalization to real-world scenarios, we conducted an external validation experiment using data from the most recent phase of the ADNI dataset. IMBALMED demonstrates superior diagnostic accuracy and predictive performance in both binary and ternary classification tasks, significantly improving early detection of MCI at a 48-month time point and showing excellent generalizability in the 12-month task during external validation. The method shows improved classification performance and robustness, offering a promising solution for early detection and management of AD.

Lung cancer remains one of the leading causes of cancer-related deaths worldwide, with Non-Small Cell Lung Cancer (NSCLC) accounting for approximately 85% of all cases. Accurate histological subtype classification of NSCLC is crucial for personalized treatment planning and improving patient outcomes. Developing robust classification models for NSCLC subtypes often requires large, diverse datasets, which can be challenging to obtain due to privacy concerns and data silos. This study proposes an approach combining federated learning with triplet loss to address these challenges. We evaluated our method’s performance in classifying NSCLC subtypes using data from multiple institutions while preserving privacy. Our experiments compared the proposed federated learning approach with triplet loss against alternative methods, including local training and softmax loss. Results demonstrated that our federated learning approach with triplet loss consistently outperformed other methods across key metrics. The combination of federated learning and triplet loss showed synergistic effects, leveraging external datasets to improve model performance while maintaining data confidentiality. The source code for the implementation described in this paper is available at https://github.com/aksufatih/federated-triplet-histology.

The automated processing of Electronic Health Records (EHRs) poses a significant challenge due to their unstructured nature, rich in valuable, yet disorganized information. Natural Language Processing (NLP), particularly Named Entity Recognition (NER), has been instrumental in extracting structured information from EHR data. However, existing literature primarly focuses on extracting handcrafted clinical features through NLP and NER methods without delving into their learned representations. In this work, we explore the untapped potential of these representations by considering their contextual richness and entity-specific information. Our proposed methodology extracts representations generated by a transformer-based NER model on EHRs data, combines them using a hierarchical attention mechanism, and employs the obtained enriched representation as input for a clinical prediction model. Specifically, this study addresses Overall Survival (OS) in Non-Small Cell Lung Cancer (NSCLC) using unstructured EHRs data collected from an Italian clinical centre encompassing 838 records from 231 lung cancer patients. Whilst our study is applied on EHRs written in Italian, it serves as use case to prove the effectiveness of extracting and employing high level textual representations that capture relevant information as named entities. Our methodology is interpretable because the hierarchical attention mechanism highlights the information in EHRs that the model considers the most crucial during the decision-making process. We validated this interpretability by measuring the agreement of domain experts on the importance assigned by the hierarchical attention mechanism to EHRs information through a questionnaire. Results demonstrate the effectiveness of our method, showcasing statistically significant improvements over traditional manually extracted clinical features.

Accurate classification of histological subtypes of non-small cell lung cancer (NSCLC) is essential in the era of precision medicine, yet current invasive techniques are not always feasible and may lead to clinical complications. This study presents MINT, a Multi-stage INTermediate fusion approach to classify NSCLC subtypes from CT and PET images. Our method integrates the two modalities at different stages of feature extraction, using voxel-wise fusion to exploit complementary information across varying abstraction levels while preserving spatial correlations. We compare our method against unimodal approaches using only CT or PET images to demonstrate the benefits of modality fusion, and further benchmark it against early and late fusion techniques to highlight the advantages of intermediate fusion during feature extraction. Additionally, we compare our model with the only existing intermediate fusion method for histological subtype classification using PET/CT images. Our results demonstrate that the proposed method outperforms all alternatives across key metrics, with an accuracy and AUC equal to 0.724 and 0.681, respectively. This non-invasive approach has the potential to significantly improve diagnostic accuracy, facilitate more informed treatment decisions, and advance personalized care in lung cancer management.

Predicting pathologic complete response in non-small cell lung cancer is crucial for tailoring effective treatment strategies and to improve patient outcomes. With the increasing application of artificial intelligence in cancer research, machine learning is poised to play a significant role in prognostication and decision-making. This paper presents a novel approach that utilizes named entity recognition and attention mechanisms applied to electronic health records to predict the pathologic complete response. We first employ named entity recognition to extract relevant biomedical entities from unstructured clinical notes within reports. These entities, combined with structured data, are then processed using a hierarchical attention mechanism to generate comprehensive patient representations. This approach captures complex relationships and contextual information within electronic health records compared to traditional methods. The results highlight the potential of advanced natural language processing techniques to enhance clinical decision-making and support personalized treatment planning in oncology.

Background and Objective: In the field of lung cancer research, particularly in the analysis of overall survival (OS), artificial intelligence (AI) serves crucial roles with specific aims. Given the prevalent issue of missing data in the medical domain, our primary objective is to develop an AI model capable of dynamically handling this missing data. Additionally, we aim to leverage all accessible data, effectively analyzing both uncensored patients who have experienced the event of interest and censored patients who have not, by embedding a specialized technique within our AI model, not commonly utilized in other AI tasks. Through the realization of these objectives, our model aims to provide precise OS predictions for non-small cell lung cancer (NSCLC) patients, thus overcoming these significant challenges.

Methods: We present a novel approach to survival analysis with missing values in the context of NSCLC, which exploits the strengths of the transformer architecture to account only for available features without requiring any imputation strategy. More specifically, this model tailors the transformer architecture to tabular data by adapting its feature embedding and masked self-attention to mask missing data and fully exploit the available ones. By making use of ad-hoc designed losses for OS, it is able to account for both censored and uncensored patients, as well as changes in risks over time.

Results: We compared our method with state-of-the-art models for survival analysis coupled with different imputation strategies. We evaluated the results obtained over a period of 6 years using different time granularities obtaining a Ct-index, a time-dependent variant of the C-index, of 71.97, 77.58 and 80.72 for time units of 1 month, 1 year and 2 years, respectively, outperforming all state-of-the-art methods regardless of the imputation method used.

Conclusions: The results show that our model not only outperforms the state-of-the-art's performance but also simplifies the analysis in the presence of missing data, by effectively eliminating the need to identify the most appropriate imputation strategy for predicting OS in NSCLC patients.

Contrast Enhanced Spectral Mammography (CESM) is a dual-energy mammographic imaging technique that first requires intravenously administering an iodinated contrast medium. Then, it collects both a low-energy image, comparable to standard mammography, and a high-energy image. The two scans are combined to get a recombined image showing contrast enhancement. Despite CESM diagnostic advantages for breast cancer diagnosis, the use of contrast medium can cause side effects, and CESM also beams patients with a higher radiation dose compared to standard mammography. To address these limitations, this work proposes using deep generative models for virtual contrast enhancement on CESM, aiming to make CESM contrast-free and reduce the radiation dose. Our deep networks, consisting of an autoencoder and two Generative Adversarial Networks, the Pix2Pix, and the CycleGAN, generate synthetic recombined images solely from low-energy images. We perform an extensive quantitative and qualitative analysis of the model's performance, also exploiting radiologists’ assessments, on a novel CESM dataset that includes 1138 images. As a further contribution to this work, we make the dataset publicly available. The results show that CycleGAN is the most promising deep network to generate synthetic recombined images, highlighting the potential of artificial intelligence techniques for virtual contrast enhancement in this field.