chidubemnkemdilim5-blip/Nigeria-News-Topic-Modeling
Domaine:
natural language processingType de record:
dataset
Créateur:
chi
Hôte:
NLP topic modeling of Nigeria news articles
# Nigerian News Topic Modeling
## Project Overview
This project uses Natural Language Processing (NLP) and Latent Dirichlet Allocation (LDA) to discover hidden topics in Nigerian news articles.
The project covers the complete workflow from data cleaning and text preprocessing to exploratory data analysis, feature engineering, topic modeling, model evaluation, topic interpretation, and visualization.
After preprocessing, **20,159 articles** were used for the final topic modeling analysis.
The final LDA model identified **7 interpretable topics** covering major themes such as governance, politics, terrorism, crime, industrial incidents, and regional issues.
---
## Objectives
The main objectives of this project are to:
- Clean and preprocess Nigerian news text.
- Explore the characteristics of the dataset.
- Analyze common words and article lengths.
- Identify missing values and data quality issues.
- Convert text into numerical representations using TF-IDF and Count Vectorization.
- Apply LDA topic modeling.
- Compare different numbers of topics.
- Evaluate the topic model using coherence and perplexity.
- Manually inspect and interpret discovered topics.
- Assign meaningful names to the topics.
- Visualize topic distributions and topic trends over time.
---
## Dataset
The project uses the `Nigeria2019_Newsfeed.csv` dataset.
The raw dataset is stored in:
```text
data/raw/Nigeria2019_Newsfeed.csv
---
## Data Cleaning and Preprocessing
The news articles were cleaned and prepared for Natural Language Processing before applying topic modeling.
The preprocessing workflow included:
- Removing invalid records and unwanted text artifacts.
- Removing duplicate articles.
- Cleaning extra spaces and formatting issues.
- Removing unwanted characters and URLs where appropriate.
- Tokenization.
- Stop-word removal.
- Lemmatization.
- Creating cleaned text and tokenized representations.
The main processed text columns include:
- `Newsfeed_Description2_Clean`
- ` …