mohammad-gh009/Small-language-models-on-clinical-data-extraction

This repository contains the implementation and supporting code for the research study “Small Language Models for Privacy-Preserving Clinical Information Extraction in Low-Resource Languages”. Small Language Models for Privacy-Preserving Clinical Information Extraction in Low-Resource Languages --- ## TL;DR We benchmark five open-source small language models (SLMs, 1B–8B parameters) on a two-step pipeline for extracting 13 binary clinical features from 1,221 anonymized Persian palliative care transcripts — **no fine-tuning required**. Qwen2.5-7B-Instruct achieves the best overall balance (macro-F1: 0.899, MCC: 0.797). Translating Persian to English with Aya-expanse-8B improves sensitivity and completeness at a slight cost to specificity. --- ## Abstract Extracting structured clinical information from unstructured medical transcripts in low-resource languages remains a significant challenge in healthcare natural language processing (NLP). This study evaluates a two-step pipeline combining Aya-expanse-8B as a Persian-to-English translation model with five open-source small language models (SLMs) — Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, Llama-3.2-3B-Instruct, Qwen2.5-1.5B-Instruct, and Gemma-3-1B-it — for binary extraction of 13 clinical features from 1,221 anonymized Persian transcripts collected at a cancer palliative care call center. Using a few-shot prompting strategy without fine-tuning, models were assessed on macro-averaged F1-score, Matthews Correlation Coefficient (MCC), sensitivity, and specificity to account for class imbalance. Qwen2.5-7B-Instruct achieved the highest overall performance (median macro-F1: 0.899; MCC: 0.797), while Gemma-3-1B-it showed the weakest results. Larger models (7B–8B parameters) consistently outperformed smaller counterparts in sensitivity and calibration. A bilingual analysis of Aya-expanse-8B revealed that translating Persian transcripts to English improved sensitivity, reduced missing outputs, and boosted balanced metrics, though at the cost of slightly lower specificity and precision. Feature-level results showed reliable extraction of physiological symptoms across most models, whereas psy …