sean-sampson-tuks/COS760-Group14-Project

COS760 Group 14 project - Evaluating Multilingual Models as Annotators for Semantic Relatedness in African Languages # COS760-Group14-Project # Exploring Semantic Relatedness in Amharic and Hausa Using Transfer Learning ## Project Overview This project benchmarks the performance of several pre-trained multilingual models on semantic relatedness tasks for two low-resource African languages: Hausa and Amharic. The models are fine-tuned and evaluated against human-annotated scores from the SemRel 2024 dataset. The analysis also explores the effectiveness of ensemble methods to improve performance. ## Methodology ### Data Sources and Preprocessing **Data Source:** The project uses the SemRel dataset from the Semantic Textual Relatedness SemEval 2024 GitHub repository. The specific files used are `hau_train.csv` for Hausa and `amh_train.csv` for Amharic. **Preprocessing:** For both languages, the raw 'Text' column, which contained sentence pairs separated by a newline, was split into 'Sentence1' and 'Sentence2' columns. The resulting dataframes were then partitioned into training (80%) and validation (20%) sets. ### Model Training and Evaluation * **Models:** Four pre-trained multilingual models were selected for fine-tuning: * XLM-ROBERTa (XLM-R) * Multilingual BERT (mBERT) * AfroXLMR * AfriBERTa * **Baseline:** A baseline was established using the `paraphrase-multilingual-MiniLM-L12-v2` model to compute cosine similarity scores without fine-tuning. * **Training:** The models were fine-tuned for a maximum of 15 epochs using a `TransformerRegressor` architecture. The training process utilised the AdamW optimiser, Mean Squared Error (MSE) loss, a batch size of 8, and an early stopping mechanism with a patience of 3 to prevent overfitting. * **Evaluation:** Model performance was measured against the human-annotated ground-truth scores using two metrics: Spearman's rank correlation coefficient and Mean Squared Error (MSE). ## Results The fine-tuned models showed a significant improvement over the baseline for both languages. Models specifically trained on African languages, such a …