haturusinghe/subasa-plm

A framework for adapting Pretrained Language Models (XLM-R, BERT etc.) for Low-Resourced Offensive Language Detection in Sinhala using pretrained models and intermediate tasks. # Subasa - Pretrained Language Models (PLM) A framework for adapting Pretrained Language Models for Low-Resourced Offensive Language Detection in Sinhala using pretrained models and intermediate tasks. ## Features - Two-stage finetuning approach with intermediate tasks - Support for multiple pretrained models (XLM-RoBERTa base/large) - Intermediate tasks: Masked Rationale Prediction (MRP) and Rationale Prediction (RP) - Comprehensive evaluation metrics including AUROC and explainability measures - Integration with Weights & Biases for experiment tracking - LIME-based model explanations ## Setup ```bash # Create and activate virtual environment python -m venv env source env/bin/activate # Install dependencies pip install -r requirements.txt ``` ## Project Structure ``` subasa-llm/ ├── main.py # Main training and evaluation script ├── src/ │ ├── config/ # Configuration files │ ├── dataset/ # Dataset loading and processing │ ├── evaluate/ # Evaluation metrics and explainers │ ├── models/ # Model implementations │ └── utils/ # Helper functions and utilities ├── pre_finetune/ # Pre-finetuning stage outputs └── final_finetune/ # Final stage model outputs ``` ## Training Modes ### 1. Pre-finetuning Stage Train with intermediate tasks (MRP or RP): ```bash python main.py \ --pretrained_model xlm-roberta-base \ --intermediate mrp \ --val_int 250 \ --patience 3 \ --mask_ratio 0.5 \ --n_tk_label 2 \ --epochs 5 \ --batch_size 16 \ --lr 0.00002 \ --seed 42 \ --wandb_project your-wandb-project-name \ --finetuning_stage pre \ --dataset sold \ --skip_empty_rat True ``` ### 2. Final Finetuning Stage Finetune for offensive language detection: ```bash python main.py \ --pretrained_model xlm-roberta-base \ --val_int 250 \ --patience 3 \ --epochs 5 \ --batch_size 16 \ --lr 0.00002 \ --seed 42 \ --wandb_project your-wandb-project-name \ --finetuning_stage final \ --dataset sold \ --num_labels 2 \ --pre_ …