Santhanu7Z/mci_detection

Listening Between the Lines: An explainable multimodal framework for MCI detection from spontaneous speech. Leverages Selective State Space Models (Mamba) and Gated Fusion to integrate linguistic disfluencies and eGeMAPS biomarkers across multi-corpus benchmarks (Pitt, ADReSS, TAUKADIAL) # Fusion Mamba for Explainable Speech-Based Detection of Dementia Official code for the paper **"Fusion Mamba for Explainable Speech-Based Detection of Mild Cognitive Impairment"** (IEEE JCSSE 2026). > **Key Finding:** Linguistic features statistically dominate predictive performance across all corpora (mean attention weight 88.1% vs 11.9% for audio). Multimodal fusion's primary value is clinical **interpretability** — per-prediction modality attribution — rather than accuracy gains. --- ## Overview This repository implements a multimodal speech-based dementia detection pipeline combining: - **Whisper Large-v3** for clinical-grade automatic transcription with hallucination filtering - **Mamba-130M** as a frozen linguistic encoder for efficient sequence modelling - **eGeMAPS v02** acoustic biomarkers (88 features) extracted via OpenSMILE - **Cross-modal attention fusion** for per-prediction modality attribution - **Multi-level XAI suite** with MC word-level perturbation, feature-category analysis, and FDR-corrected biomarker testing Evaluated across three benchmark corpora: **Pitt (DementiaBank)**, **ADReSS 2020**, and **TAUKADIAL**. --- ## Results Full cross-corpus Weighted F1 scores across all 5 architectures and 4 training conditions (mean over 5-fold stratified CV grouped by participant ID). Bold = best per row. | Training Source | Eval Target | Linguistic | Acoustic | Interaction | Gated Fusion | Attn. Fusion | |----------------|-------------|:----------:|:--------:|:-----------:|:------------:|:------------:| | Pitt | Pitt | 0.928 | 0.667 | 0.923 | 0.919 | **0.932** | | Pitt | ADReSS | 0.812 | 0.563 | 0.813 | **0.825** | 0.814 | | Pitt | TAUKADIAL | 0.519 | 0.513 | 0.491 | 0.503 | **0.556** | | ADReSS | Pitt | **0.624** | 0.460 | 0.437 | 0.602 | 0.619 | | ADReSS | ADReSS | 0.918 | 0.654 | 0.811 | 0.897 | **0.945** | | ADReSS | TAUKADIAL | 0.432 | 0.385 | 0.357 | **0.520** | 0.484 | | TAUKADIAL | Pitt | 0.490 …