Lucas-Granucci/MULTI-NER

Multilingual Named Entity Recognition for low-resource and underrepresented languages # MULTI-NER: Multi-Domain Named Entity Recognition ## Overview This is the official code repository for the research project "Breaking Language Barriers: Cross-lingual Transfer Learning and Pseudo-Labeling for Natural Language Processing in Low-Resource Languages" found here -> Paper ## Table of Contents - Introduction - Installation - Usage - Datasets - Model Architectures - Results ## Introduction Named Entity Recognition (NER) is a crucial task in Natural Language Processing (NLP) that involves identifying and classifying entities in text. This project improves performance of NER systems for low-resource languages by transfering knowledge from adjacent languages and by iterativley training on synthethically generated pseudo-labels. ## Installation To get started, clone the repository and install the required dependencies: ```bash git clone github.com cd MULTI-NER conda env create -f environment.yml ``` ## Usage Sections in ```_MAIN_.ipynb``` include the training of the baseline BERT-BiLSTM-CRF model, testing various ratios of high to low-resource language data in cross-lingual transfer learning, and testing various confidence intervals in iterative pseudo-labeling. ## Dataset The data for this project is sourced from the WikiANN multi-lingual NER dataset. The specific language pairs used and their are presented below: | Low-Resource Language | High-Resource Language | Low-Resource Sentence Count | High-Resource Sentence Count | |-----------------------|------------------------|----------------------------|------------------------------| | Malagasy | Indonesian | 300 | 40,000 | | Faroese | Danish | 300 | 40,000 | | Corsican | Italian | 300 | 40,000 | | Upper Sorbian | Polish | 300 …