codewith-pavel/Cross-Lingual-Knowledge-Distillation

Bridging High-Resource to Low-Resource Language Gaps: Refining Clinical Outcome Prediction via Cross-Lingual Methods and Adaptive Translation Strategies with SHAP # Bridging High-Resource to Low-Resource Language Gaps in Clinical NLP This repository hosts the complete implementation and experimental artifacts for the paper: --- ## 📌 Abstract Low-resource languages like Bengali are often underserved in clinical natural language processing (NLP) tasks due to the lack of annotated datasets and pretrained language models. In this work, we address these challenges through a pipeline incorporating **cross-lingual knowledge distillation (CLKD)** and **adaptive translation strategies**. Our key innovations include: - **Adaptive translation** using Google Translate, AI Sheets™, and Gemini 2.0 to create high-quality Bengali clinical corpora. - **Model interpretability** using SHAP to provide granular insight into feature importance. - **Ensemble teacher models** leveraging ClinicalBERT and dynamic attention-weighted ensembling. - **Cost-sensitive learning frameworks** to mitigate class imbalance in clinical predictions (mortality and length-of-stay). We achieve **state-of-the-art AUC scores**: - **0.8716** for *mortality prediction* - **0.7877** for *length-of-stay prediction* using MIMIC-III. --- --- ## 🧠 Key Components ### 🔁 Translation Workflow We use a multi-source translation mechanism: - **Google Translate API** - **Gemini 2.0 (Google AI)** for contextual clinical translation. - Back-translation evaluation with **BLEU**, **ROUGE**, and **METEOR**. ```python # Gemini 2.0 Translation Prompt generation_config = { "temperature": 0.1, "top_p": 0.4, "top_k": 40, "max_output_tokens": 8192 } ``` --- ### 📘 Cross-Lingual Knowledge Distillation (CLKD) - **Teacher Model**: Ensemble of ClinicalBERTs with task-specific fine-tuning. - **Student Model**: Bengali-supported multilingual transformers like XLM-RoBERTa or custom DistilBERT. - **Attention fusion**: Weighted softmax over logits from each teacher. --- ### 🔍 Model Interpretability with SHAP We use SHAP to understand the influence of translated text segments: ```pytho …