ArnolFokam/NER-Annotations-Density-vs-Pre-trained-Models

A repository for the final project of the course COMS4054A - NLP and entitled "Effects of NER Annotations’ Density on Pre-trained Language Models’ Performance in the Context of African Languages" ## Effects of NER annotations’ density on pre-trained models in the context oflow resourced languages This is repository for the course project COMS4054A - NLP 2021. It contains data and necessary notebooks to get the resutls in the report of the same project. ### Table of Contents - Folder structure - Environment setup - Hardware Requirements - Software Requirements - Training and Evaluation - Reproducibility - Model Cards - BERT - RoBERTa - Multilingual BERT (mBERT) - Multilingual BERT (mBERT) finetuned for NER - License ### Folder structure ``` Project ├── data │ └── {lang} -> Dataset corpus in lang corpus │ ├── train.txt -> train set │ ├── dev.txt -> dev set │ └── test.txt -> test set ├── images -> Images generated from various analysis ├── notebooks. │ ├── cap_training_data.ipynb -> Jupyter notebook to create preprocessed corpus. | ├── results_analysis.ipynb -> Jupyter notebook to extract model evaluation files and generate plots. │ └── train_ner.ipynb -> Jupyter notebook to train all the models and evaluate them. └── README.md -> project description ``` ### Environment setup #### Hardware Requirements - RAM 16 GB or more - GPU with CUDA support (for faster training) #### Software Requirements - Python (>= 3.8) and equivalent pip ### Training and Evaluation All the notebooks to train and evaluate all the models can be found in notebooks directory as outline in the Folder structure section. #### Reproducibility To reproduce the results of the report. Use the following hyperparameters: - **Learning Rate** 5e-5 - **Batch Size** 32 - **Maximum Sequence Length** 164 - **Epochs** 30 ### Model Cards This section contains the description to various pre-trained models used as well as the link to model cards. #### BERT This is the BERT base cased model trained on English text with 12 layers of transformers block with a hidden size of 768, 12 attention heads and 110 parameters. Here are the various model cards for dif …