BounharAbdelaziz/Terjman-English-Morocco-Darija-Translation

Terjman is a Transformer model trained for the translation from English to Moroccan darija. # Terjman-English-Morocco-Darija-Translation Terjman is a Transformer model trained for the translation from English to Moroccan darija. This repository provides the necessary and sufficient code used for training. All models are finetuned on a the English-to-Moroccan-Darija dataset, using a **A100-40GB** GPU. Model checkpoints are available on Hugging Face 🤗: - Terjman-Ultra - Terjman-Large - Terjman-Nano Note: Currently developping the second version of Terjman, trained on a larger dataset with translation from many languages to moroccan darija and vice versa. ## Setup and Training Start by installing the necessary dependencies: ```bash pip install -r requirements.txt ``` Then run the training script for the version you would like to train: ```bash python3 train.py ``` Note that the v2 code is probably going to change. ## Usage Using our model for translation is simple and straightforward. You can integrate it into your projects or workflows via the Hugging Face Transformers library. Here's a basic example of how to use the model in Python: ```python from transformers import AutoTokenizer, AutoModelForSeq2SeqLM # Load the tokenizer and model tokenizer = AutoTokenizer.from_pretrained("atlasia/Terjman-Ultra") model = AutoModelForSeq2SeqLM.from_pretrained("atlasia/Terjman-Ultra") # Define your Moroccan Darija Arabizi text input_text = "Your english text goes here." # Tokenize the input text input_tokens = tokenizer(input_text, return_tensors="pt", padding=True, truncation=True) # Perform translation output_tokens = model.generate(**input_tokens) # Decode the output tokens output_text = tokenizer.decode(output_tokens[0], skip_special_tokens=True) print("Translation:", output_text) ``` ## Example Bellow is an example of translating English input to Moroccan Darija in Arabic letters (Ary) using Terjman-Ultra: **Input**: "Hi my friend, can you tell me a joke in moroccan darija? I'd be happy to hear that from you!" **Output**: "أهلا صاحبي، تقدر تقولي مز …