sefineh-ai/Amharic-Tokenizer
Domaine:
natural language processingType de record:
software
Créateur:
sef
Hôte:
Syllable-aware BPE tokenizer for the Amharic language (አማርኛ) – fast, accurate, trainable.
# Amharic Tokenizer 🇪🇹
**Amharic tokenizer with a GPT-style BPE-like pipeline over decomposed fidel.**
Implements: **cleaning → fidel decomposition → BPE training/application → detokenization**, with a **Cython core for speed**.
---
## What's new in v0.2.6
- Vocab size: 30,000 tokens
- Trained on a larger and more diverse Amharic corpus
- Improved tokenization quality and detokenization accuracy
- Better handling of edge cases and rare words
1. **Pretrained tokenizer loading**
- You can now load a pretrained tokenizer directly:
```python
from amharic_tokenizer import AmharicTokenizer
tok = AmharicTokenizer.load("amh_bpe_v0.2.6")
```
This version includes a pretrained model (`amh_bpe_v0.2.6`) that can be used immediately without any additional setup and training.
2. **Full token-to-ID and ID-to-token functionality**
- Added complete round-trip processing methods:
```python
tokens = tok.tokenize(text)
ids = tok.encode(tokens)
detokenized = tok.detokenize(tokens)
```
The tokenizer now supports seamless conversion between tokens and IDs, ensuring full consistency between tokenization and detokenization.
---
### Test Script: test_roundtrip_basic.py
```python
from amharic_tokenizer import AmharicTokenizer
def test_roundtrip_basic():
"""Load a trained tokenizer, tokenize text, convert to IDs, and detokenize."""
tok = AmharicTokenizer.load("amh_bpe_v0.2.6")
text = (
"የኮሪደር ልማት ገፀ በረከት የሆናቸው የከተማችን ሰፈሮች በነዋሪዎች አንደበት በሰዓት 209 ኪሎ ሜትር የሚጓዘው አውሎ ንፋስ ከጃማይካ ቀጥሎ ኩባ ደርሷል ጠቅላይ" )
tokens = tok.tokenize(text)
ids = tok.encode(text)
detokenized = tok.detokenize(tokens)
print("Original Text: ", text)
print("Tokens: ", tokens)
print("IDs: ", ids)
print("Detokenized Text: ", detokenized)
assert text == detokenized, "Detokenized text does not match the original."
if __name__ == "__main__":
test_roundtrip_basic()
Output:
Tokenizer state loaded from amh_bpe_v0.2.6.json
Original Text: የኮሪደር ልማት ገፀ በረከት የሆናቸው የከተማችን ሰፈሮች በነዋሪዎች አንደበት በሰዓት 209 ኪሎ ሜትር የሚጓዘው አውሎ ንፋስ ከጃማይካ ቀ …