ymitiku/amtokenizers

Set of classes to tokenize Amharic language sentences. # Amharic Language Tokenizers This package contains set of Classes which can be used to encode Amharic language sentences into tokens that could be used by language models. The tokenizers are trained using Contemporary Amharic Corpus (CACO) dataset ## Installing #### Pip installation ```bash pip install -i test.pypi.org amtokenizers==0.0.5 ``` ## Sample Code ### Variable length ```python from amtokenizers import AmTokenizer a = AmTokenizer(10000, 5 , "byte_bpe") encoded = a.encode("አበበ በሶ በላ።") print("encoded", encoded.tokens) # encoded [' ', 'áĬł', 'áīłáīł', 'ĠáīłáĪ¶', 'ĠáīłáĪĭ', 'áį', '¢', ' '] print("decoded:", a.decode(encoded.input_ids)) # decoded: አበበ በሶ በላ። ``` ### Fixed length ```python a = AmTokenizer(10000, 5 , "byte_bpe", max_length=16) encoded = a.encode("አበበ በሶ በላ።") print("encoded", encoded.tokens()) # encoded [' ', 'áĬł', 'áīłáīł', 'ĠáīłáĪ¶', 'ĠáīłáĪĭ', 'áį', '¢', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' '] print(encoded.input_ids) # [0, 337, 3251, 3598, 3486, 270, 100, 2, 1, 1, 1, 1, 1, 1, 1, 1] print("decoded:", a.decode(encoded.input_ids)) # decoded: አበበ በሶ በላ። ``` Disclaimer This package is highly inspired by Hugging Face's How to train a new language model from scratch using Transformers and Tokenizers tutorial.