chaima-Khenafif03/darija-social-listening-pipeline

Multi-model pipeline for clustering and analyzing multilingual social media comments — language detection, relevance filtering, Darija translation, topic clustering, and sentiment analysis. # Darija Social Listening Pipeline A multi-model pipeline that cleans, filters, translates, clusters, and analyzes the sentiment of multilingual social media comments — built to handle the language mix common in Algerian social media (French, English, Arabic, Darija in both Latin and Arabic script) in a single automated flow. Built during an internship at Intaj (July–September 2025), with BENAISSA Faiza as co-developer, supervised by S. Touileb. ## Example output Run on a sample of ~1,000 real YouTube comments about Algerie Telecom, the pipeline automatically grouped and titled clusters like: | Cluster title | Example comment | |---|---| | `unlimited calls djezzy` | "Avec 500 dinars les appels illimités vers Djezzy pour 30 jours et non pas pour 15 jours" | | `kane jizzy` | "كيفاش نحول من جيزي هايلة بزاف الى Bienvenue sur izzy" | | `service available offer` | "واليوتيوب ايليميتي يخلص كي يخلصو 10 جيڨا ولا شهر كامل" | Each cluster also gets a sentiment breakdown (positive / neutral / negative), aggregated into a dataset-level sentiment chart. ## Why this architecture There's no clustering-ready multilingual sentence embedding model trained on Darija, so rather than clustering each language separately, everything gets translated into a common pivot language (English) before embedding. Filtering out irrelevant comments *before* translation avoids wasting translation API calls and compute on text that would get discarded anyway. ## Pipeline steps 1. **Clean** — strip URLs, salutations, and noise from raw comments 2. **Detect language** — French / English / Arabic / Darija Latin / Darija Arabic / other, using a custom Darija detector with a fallback 3. **Filter relevance** — a fine-tuned XLM-R classifier discards off-topic/irrelevant comments before any expensive processing happens 4. **Transliterate** — Darija Latin → Darija Arabic (seq2seq), so both Darija scripts converge to one form 5. **Translate to English** — MarianMT for French/Arabic, Google Translate AP …