abrhaleitela/Transferring-Monolingual-Model-to-Low-Resource-Language

# Transferring Monolingual Model to Low-Resource Language: The Case Of Tigrinya: ## Proposed Method: The proposed method transfers a mono-lingual Transformer model into new target language at lexical level by learning new token embeddings. All implementation in this repo uses XLNet as a source Transformer model, however, other Transformer models can also be used similarly. ## Main files: All files are IPython Notebook files which can be excuted simply in Google Colab. - train.ipynb : Fine-tunes XLNet (mono-lingual transformer) on new target language (Tigrinya) sentiment analysis dataset. - test.ipynb : Evaluates the fine-tuned model on test data. - token_embeddings.ipynb : Trains a word2vec token embeddings for Tigrinya language. - process_Tigrinya_comments.ipynb : Extracts Tigrinya comments from mixed language contents. - extract_YouTube_comments.ipynb : Downloads available comments from a YouTube channel ID. - auto_labelling.ipynb : Automatically labels Tigrinya comments in to positive or negative sentiments based on Emoji's sentiment. ## Tigrinya Tokenizer: A sentencepiece based tokenizer for Tigrinya has been released to the public and can be accessed as in the following: from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("abryee/TigXLNet") tokenizer.tokenize("ዋዋዋው እዛ ፍሊም ካብተን ዘድንቀን ሓንቲ ኢያ ሞ ብጣዕሚ ኢና ነመስግን ሓንቲ ክብላ ደልየ ዘሎኹ ሓደራኣኹም ኣብ ጊዜኹም ተረክቡ") ## TigXLNet: A new general purpose transformer model for low-resource language Tigrinya is also released to the public and be accessed as in the following: from transformers import AutoConfig, AutoModel config = AutoConfig.from_pretrained("abryee/TigXLNet") config.d_head = 64 model = AutoModel.from_pretrained("abryee/TigXLNet", config=config) ## Evaluation: The proposed method is evaluated using two datasets: - A newly created sentiment analysis dataset for low-resource language (Tigriyna). Models Configuration F1-Score BERT +Frozen BERT weights 54.91 +Random …