HossamElsrah/Transformer-Building-For-Egyptian-English-Translation

A from-scratch implementation of a Transformer model in PyTorch for translating Egyptian Arabic to English. This project focuses on understanding the intricacies of Transformer architecture and its application to a challenging, low-resource translation task. # Transformer for Egyptian to English Translation ### **Building a Transformer from Scratch** This repository contains a complete implementation of a Transformer neural network, built from the ground up using PyTorch. The primary goal of this project was to gain a deep understanding of the Transformer architecture by manually constructing each of its core components: * **Embedding Layers:** For converting input and output tokens into dense vector representations. * **Positional Encoding:** To provide the model with information about the order of tokens in a sequence. * **Multi-Head Self-Attention:** The core mechanism for capturing relationships between different parts of the input sequence. * **Add & Norm Layers:** Implementing residual connections and layer normalization for stable training. * **Point-wise Feed-Forward Networks:** Applying non-linear transformations to the attention outputs. * **Encoder and Decoder Stacks:** Combining the individual layers to form the complete Transformer model. * **Linear and Softmax Layers:** For generating the final probability distribution over the target vocabulary. ### **Egyptian Arabic to English Translation** This project goes beyond a standard Transformer implementation by applying it to the task of translating from Egyptian Arabic to English. This is a particularly interesting challenge due to: * **Linguistic Differences:** Significant variations in grammar, syntax, and vocabulary between Egyptian Arabic and English. * **Data Scarcity:** Compared to standard Arabic, there is generally less publicly available parallel data for Egyptian Arabic and English. * **Dialectal Variation:** Egyptian Arabic itself has regional and social variations, which can add complexity to the translation task. The decision to tackle this specific translation direction was driven by the desire to explore the model's capabilities in a less-resourced setting and to appreciate the nuances involved in handling diverse linguistic structures. …