jefftrojan/hausa-to-english-translation

### **Hausa-to-English translator using RNN** --- This project uses data that has been webscraped from twitter and preprocessed to have parallel data which is then used for translation operations. #### **1. Data Preparation** I started by preparing a parallel dataset that contained Hausa-to-English translations of tweets. The dataset included four key columns: `CleanedMainT` (the main tweets), `CleanedReplyT` (replies to tweets), and their corresponding English translations (`Hausa2EngMainT` and `Hausa2EngReplyT`). Before diving into the model training process, I spent time cleaning and preprocessing the text data: - **Text Cleaning**: I applied a custom text preprocessing function to remove unwanted characters such as punctuation, URLs, and extra spaces from both `CleanedMainT` and `CleanedReplyT`. - **Handling Missing Data**: To avoid issues during training, I used `fillna('')` to replace missing values and ensured that all entries were converted to strings. This prevented any non-string data from breaking the model later on. --- #### **2. Model Design** For this task, I decided to use a Recurrent Neural Network (RNN) based **Encoder-Decoder model with Attention** for translating Hausa to English. The choice of model stemmed from its effectiveness in handling sequence-to-sequence problems like translation. - **Encoder**: I used a series of RNN layers to process the input sequence (Hausa tweets) and convert them into fixed-size context vectors. - **Attention Mechanism**: To improve translation accuracy, I incorporated an attention mechanism, which allows the model to focus on relevant parts of the input sequence when generating each word in the output sequence. - **Decoder**: The decoder consists of RNN layers as well, taking the context vectors from the encoder along with the attention output and generating the translated English sentences. --- #### **3. Training Procedure** To train the model, I used the following steps: - **Data Splitting**: The da …