MELAI-1/polarization-identification-nlp
Domaine:
natural language processingCréateur:
MEL
Hôte:
NLP project for multi-task classification of polarization in English and Swahili social media text. Includes in-depth EDA and modeling strategy.
# Polarization Identification and Analysis (Multi-Task Classification)
## 1. Project Overview
This repository documents the solution for a multi-task NLP challenge focused on identifying and characterizing polarization in social media text from two distinct languages: **English** and **Swahili**.
The project is divided into three core multi-label/binary classification tasks:
| Task | Objective | Label Columns |
|:-----|:----------|:--------------|
| **Task 1** | **Polarization Identification** | Binary: Polarized (1) vs. Non-Polarized (0) |
| **Task 2** | **Category Classification** | Multi-Label: Political, Racial/Ethnic, Religious, Gender/Sexual, Other |
| **Task 3** | **Manifestation Identification** | Multi-Label: Stereotype, Vilification, Dehumanization, Extreme Language, etc. |
## 2. Key Findings from Exploratory Data Analysis (EDA)
The EDA revealed critical linguistic and structural differences essential for the modeling strategy:
| Language | Task 1 Imbalance | Task 2 Dominance | Key Linguistic Feature | Strategic Action |
|:---------|:-----------------|:-----------------|:-----------------------|:-----------------|
| **English** | High (64% Non-Polarized) | Political (66% of tags) | Polarization often uses **complex ideological/political terms**. Text length is a **useful** feature. | Use class weighting; Use `text_len` as a feature. |
| **Swahili** | Balanced (~50% Polarized) | Racial/Ethnic (68% of tags) | Polarization often uses **generic, vulgar slang** shared across all categories. Text length is **not** a useful feature. | Use class weighting for rare categories; Treat Task 2/3 as independent binary classifiers. |
## 3. Repository Structure
```
/polarization-identification-nlp
├── .gitignore
├── README.md
│
├── /data
│ └── /raw # Original task data (eng.csv, swa.csv for T1, T2, T3)
│
├── /notebooks
│ ├── 01_Exploration.ipynb # Full analysis notebook (where all insights and plots were generated)
│ └── 02_Subtask_1_eng. …