Multimodal Ensemble Learning Framework for Breast Cancer Diagnosis Using Transfer Learning and Clinical Machine Learning Models

Abstract Breast cancer remains one of the leading causes of cancer-related death among women all over the world. Early and accurate diagnosis provides potential for better management. From the literature reviewed, several existing systems relied mostly on single data modalities, while others present infrastructure gaps not compatible with health care settings in low and middle-income countries, especially in Africa. The aim of this paper is a multimodal ensemble learning framework for breast cancer diagnosis using transfer learning and clinical machine learning models. The study used three secondary datasets. This includes the mammogram image dataset from Mendeley. The sample size is 614 mammograms, and the classes are benign with 361 images and malignant with 253 images. The second dataset is the clinical records of the 614 participants with 31 attributes attached alongside the mammogram dataset. These first two datasets considered women in South Africa.  University of Calabar Teaching Hospital (UCTH) breast cancer dataset, publicly available at Kaggle repository with 11 attributes, was the third dataset. The mammogram dataset was applied to train MobileNetV3, EfficientNet-B0, and SqueezeNet, while two independent clinical datasets from Nigeria and South Africa were applied to train Logistic Regression (LR), Decision Tree (DT), Multi-Layer Perceptron (MLP), and XGBoost. Comparative analysis identified MobileNetV3 as the best transfer learning model with 90% accuracy, LR as the best model for the Nigerian dataset with 93.02% accuracy, and MLP as the best clinical classifier on the South African dataset with 97.56% accuracy. These three models were applied to curate the ensemble model using the hard voting technique and then develop the multimodal ensemble learning framework for breast cancer diagnosis. Experimental results when tested with images and clinical data from low-income country healthcare system showed successful classification robustness and reduced error because the strengths of the medical image and clinical data classifiers were leveraged. In conclusion, the study provides a reliable framework that is computationally efficient, compatible for resource constraint environment and suitable for timely diagnosis of breast cancer.