Abdisalam-Badel/somstemmer

Somstem: A Hybrid Stemmer for the Somali Language ## Somstem ------------------------------------------------------------------- This repository is the official implementation of: Somstem: A Hybrid Stemmer for the Somali Language >📋 Somstem Architecture: ## Installation >📋 Follow these steps to download and set up the software: 1. Download the `somstem.zip` file from somstem. 2. Extract the ZIP file. You may see a folder structure like `somstem/Somstem`, be sure to use the correct inner directory when accessing the files. 3. Place the `Somstem` folder in your project directory or any location of your choice. >📋 Install the stemmer: ``` pip install directory/somstem/dist/somstem-0.1.0-py3-none-any.whl ``` >📋 example in my case: ```pip install D:/somali_hate_speech/somstem/dist/somstem-0.1.0-py3-none-any.whl``` ## Install Requirements: ``` transformers == 4.46.3 torch == 2.4.1 datasets sentencepiece protobuf ``` ## Usage ``` from somstem.mt5_last import HybridStemmer if __name__ == "__main__": hybrid_stemmer = HybridStemmer() while True: word_to_stem = input("Enter a word to stem or type 'quit' to exit: ") if word_to_stem.lower() == "quit": print("Exiting the program.") break stemmed_word = hybrid_stemmer.stem(word_to_stem) print(f"Original word: {word_to_stem}") print(f"Stemmed word: {stemmed_word}") ``` >📋 Sample Output (Single Word Stemming): ``` Enter a word to stem or type 'quit' to exit: xarunta Original word: xarunta Stemmed word: xarun Enter a word to stem or type 'quit' to exit: ``` >📋 For stemming a corpus, rewrite the usage code accordingly. ## Adapting to a New Language >📋 If you want to use **Somstem** for any other low-resource language and customize our code, first prepare your data following our format, second edit rules 1-6, then retrain the stemmer by running ``cd code python somstem.py``, If you need any help or further details, contact **Abdisalam** **Badel** at fiicane121@gmail.com. ## 📌 Citation If you find this repository helpful, please cite our paper: ` …