AbelAdissu/Cross-Lingual-Question-Answering-for-Amharic-Language-Using-Pretrained-LLMs-

## **INTRODUCTION** 📖 Welcome to the Amharic Text Generation project, a journey into the realm of natural language processing for low-resource languages. This initiative taps into the power of large language models, particularly transformer-based architectures, to understand and generate the Amharic language, which historically had limited exposure in the digital world of AI. Our project distinguishes itself by meticulously constructing transformer models from scratch and adapting pre-trained large language models to grasp the syntactic and semantic nuances of Amharic. This project is unique in its technical approach: we've built a Bigram Language Model and a more complex GPT (Generative Pre-trained Transformer) model from the ground up, specifically tailored for Amharic. The Bigram Model serves as a fundamental step into language modeling, focusing on the prediction of the next character in a sequence based on its predecessor, thereby grasping the basic language patterns. On the other hand, our GPT model, which is the centerpiece of this project, employs a sophisticated multi-head attention mechanism, enabling the model to simultaneously process different parts of the input sequence and learn a richer understanding of context and relationships between words. We've pushed the boundaries further by incorporating advanced transformer architectures in our GPT model. These architectures rely on stacks of decoders—a powerful component that allows the model to generate coherent and contextually relevant text. Additionally, we've explored and adapted various scales of pre-trained models, including: * 'gpt2' with 124M parameters * 'gpt2-medium' with 350M parameters * 'gpt2-large' with 774M parameters * 'gpt2-xl' with 1558M parameters Each of these models offers a different level of complexity and learning capacity, allowing us to experiment and find the most effective approach for the Amharic language. This range of models helps in understanding the scalability and adap …