rasyosef/bert-amharic

BERT transformer models pretrained on amharic text # BERT Amharic This repo contains 4 BERT transformer models pretrained on `290 million tokens` of Amharic text. All four models have a context length of `512` tokens and the same tokenizer with a vocabulary size of `28672` tokens. The size of these models ranges from 4 Million to 40 Million parameters. |Model|Size (# params)| Perplexity|Sentiment (F1)| Named Entity Recognition (F1)| |-----|---------------|-----------|--------------|------------------------------| |bert-medium-amharic|40.5M|13.74|0.83|0.68| |bert-small-amharic|27.8M|15.96|0.83|0.68| |bert-mini-amharic|10.7M|22.42|0.81|0.64| |bert-tiny-amharic|4.18M|71.52|0.79|0.54| |xlm-roberta-base|279M||0.83|0.73| |am-roberta|443M||0.82|0.69| ### Models You can download and load the models from HuggingFace using the transformers library. - bert-medium-amharic : rasyosef/bert-medium-amharic - bert-small-amharic : huggingface.co - bert-mini-amharic : huggingface.co - bert-tiny-amharic : rasyosef/bert-tiny-amharic - Amharic BERT collection : huggingface.co ### Finetuning Code - **Sentiment Classification** - Dataset: amharic-sentiment - Code: rasyosef/amharic-sentiment-… - Finetuned Model: bert-medium-amharic-finetuned-sentiment - **Named Entity Recognition** - Dataset: amharic-named-entity-recognition - Code: rasyosef/amharic-named-enti… - Finetuned Model: bert-medium-amharic-finetuned-ner - **News Category Classification** - Dataset: amharic-news-category-classification - Code: rasyosef/amharic-news-categ… # How to use In addition to finetuning, you can use these models directly with a pipeline for masked language modeling: ```python >>> from transformers import pipeline >>> unmasker = pipeline('fill-mask', model='rasyosef/bert-me …