Fine-Tuning mBART-50 for Akkadian-to-English Translation: A Transfer Learning Approach for Low-Resource

Overview The research paper "Fine-Tuning mBART-50 for Akkadian-to-English Translation" by Frank Morales Aguilera explores a methodology to adapt modern neural machine translation (NMT) techniques to Akkadian, an extinct Semitic language. The project addresses the massive bottleneck in Assyriological research, where millions of cuneiform tablets remain untranslated due to a shortage of human experts. Technical Methodology The study utilizes the mBART-50 architecture, a Transformer-based model with 1.2 billion parameters. Key technical steps include: Vocabulary Expansion: The researchers added a custom language token, [akk_AK], to the model’s tokenizer to recognize Akkadian as a 51st language. Random Initialization: During embedding resizing, the authors used random initialization (mean_resizing=False) because Akkadian is typologically distant from the model’s original 50 languages. Fine-Tuning: The model was fine-tuned on a single NVIDIA T4 GPU for 100 epochs, taking approximately 62.43 seconds to reach convergence. Dataset and Performance The model was trained on a highly specialized parallel corpus of 20 phrase pairs across three distinct domains: Legal Formulas: Including excerpts from the Code of Hammurabi. Astronomical Observations: Descriptions of celestial events (e.g., planet movements through Scorpio). Environmental/Historical: Historical accounts of floods and peaceful dwellings. The model achieved 100% accuracy on this training data, demonstrating a perfect ability to memorize and map the specialized Akkadian vocabulary to English. Key Contributions and Findings Technical Feasibility: The work proves that large-scale multilingual models can be rapidly adapted to extinct, low-resource languages without structural changes to the model architecture. Open Science: The project is fully reproducible, with the complete code available on GitHub and the trained model hosted on the Hugging Face Hub. Strategic Foundation: While the current model primarily performs memorization rather than general translation, it establishes a blueprint for scaling to larger datasets. Limitations and Future Work The paper identifies several hurdles for moving toward a production-ready system: Generalization: The model currently overfits due to the tiny dataset. Future work requires expanding the corpus to 5,000–10,000 sentences. Script Complexity: The dataset uses scholarly transliteration. A robust system would eventually need to integrate cuneiform sign recognition (OCR). Linguistic Features: Future iterations aim to incorporate explicit morphological analysis (verb roots and case markings) to better handle Akkadian’s complex grammar.