RAMDS: Retrieval Augmented Medical Diagnosis System for Explainable Breast Cancer Classification from Ultrasound Images

Abstract Breast cancer, a leading cause of cancer-related deaths in women, presents a growing challenge in medical diagnostics. Despite the effectiveness of mammography and ultrasound, the ambiguity in non-invasive scans often necessitates invasive procedures. Our primary goal was to create an AI model that could predict breast cancer with high negative predictive value and reduce unnecessary procedures. This study introduces the Retrieval-Augmented Medical Diagnosis System (RAMDS) for breast cancer, a novel approach combining an AI model with a retrieval-augmented mechanism to enhance diagnostic accuracy and explainability. The RAMDS employs a pretrained ResNet 34 model, fine-tuned on breast ultrasound image datasets from four countries. It integrates a similarity-based weighted adjustment mechanism to compare new cases with historical diagnoses. It’s like having an experienced doctor who remembers every case they’ve ever seen and uses that knowledge to make better decisions. RAMDS improved sensitivity by 11%, and negative predictive value by 9% when compared to the base model. Notably, the RAMDS improves explainability by linking AI predictions to similar historical cases, aligning with the medical community’s interest in transparent and understandable AI decisions. A unique feature of this system is its adaptability to varied imaging contexts without retraining, addressing the challenge of dataset variability across medical institutions. In conclusion, the RAMDS offers a significant advancement in breast cancer diagnosis, combining enhanced accuracy, explainability, and adaptability. It holds promise for clinical application, though further research is needed to optimize its performance and integrate multi-modal data.