k-manaa/Algerian-Arabizi

# Algerian Arabizi to Standard Arabic Dataset ## Description This repository contains the code and datasets used for the research and development of transliteration systems and NLP models for Algerian Arabizi, as detailed in the Master's thesis "Exploring the Complexities of Transliteration: A Study on Algerian Arabizi" by Khadidja Manaa at the University of Zurich, under the supervision of Prof. Dr. Gerold Schneider. The thesis explores the complexities of transliteration, particularly focusing on Algerian Arabizi—a blend of Arabic and Latin letters used in digital communication by millions of users. The research highlights the challenges of phonetic transformation when dealing with non-Latin scripts and reviews various transliteration methods from traditional rule-based systems to advanced neural models. Through detailed analyses, the study examines the efficacy of statistical and neural models in handling the inconsistencies and phonetic ambiguities inherent in Algerian Arabizi. **Very important note:** This dataset was built thanks to the efforts of the respective teams (who will be cited appropriately below). It would greatly benefit from manual annotation and correction! ## Contents ### 1. Code Files: - `final_preprocessing.py`: - Preprocessing script for preparing the datasets, it cleans the data by retaining only the most frequent transliterations for each 'Arabizi' entry and saves the cleaned results to ‘cleaned_arabizi.csv’. - `generate_ngrams.py`: - Script to generate n-grams (from 1 up to the length of the text) for each pair of words, and saves all mappings. An example would be the following for the word pair kayen,كاين: - k,ك - a,ا - y,ي - e,ن - ka,كا - ay,اي - ye,ين - kay,كاي - aye,اين - kaye,كاين - One can already see that the n-grams are of subpar quality, since Arabizi is very inconsistent, and many of the character mappings aren’t directly one to one. - `generate_unique_ngrams.py`: - Script to generate unique n-grams; it counts occurrence …