Yoshi728-PNG/sentiment-annotation-project

Sentiment annotation quality evaluation on an English-Swahili social media dataset # Sentiment Annotation Quality Evaluation ### English–Swahili Social Media Dataset A hands-on AI data annotation project demonstrating dataset creation, sentiment labeling, data-quality validation, and inter-annotator agreement analysis on a bilingual (English–Swahili) social media comment dataset. --- ## Project Summary This project simulates a real-world annotation pipeline used to prepare training/evaluation data for sentiment analysis models. It focuses on a Kenyan social media context, where comments frequently code-switch between English and Swahili — a pattern often under-represented in mainstream NLP datasets. **Dataset:** 200 social media–style comments (YouTube-comment style, X/Twitter-style, and manually written examples) **Labels:** `Positive` · `Negative` · `Neutral` | Comment | Label | |---|---| | "This song is amazing!" | Positive | | "Sijapenda hii video." | Negative | | "It was uploaded yesterday." | Neutral | --- ## Repository Structure ``` sentiment-annotation-project/ ├── data/ │ ├── raw_comments.csv # Unlabeled comments (annotator input) │ ├── annotated_comments.csv # Final gold-standard annotated dataset │ └── dual_annotation.csv # Two annotators' labels (for agreement analysis) ├── annotation_guidelines.pdf # Labeling standard, edge cases, examples ├── quality_check.py # Data-quality validation script ├── agreement_analysis.py # Inter-annotator agreement + confusion matrix ├── confusion_matrix.png # Visualization of annotator agreement ├── README.md └── report.pdf # Full project write-up and results ``` --- ## Tools Used | Purpose | Tool | |---|---| | Text annotation | Label Studio | | Data cleaning & validation | Python + Pandas | | Agreement metrics | scikit-learn (Cohen's Kappa, confusion matrix) | | Visualization | Matplotlib | | Documentation | Markdown | | Version control | Git & GitHub | --- ## Workflow 1. **Create the dataset** — 200 comments compiled i …