somali-nlp/somali-dialect-classifier

An Open, End-to-End Dialect Classifier # Somali Dialect Classifier A production-ready data pipeline for collecting, processing, and preparing high-quality Somali language text from multiple sources. Built for dialect classification and NLP research with MLOps best practices including structured logging, automated quality metrics, deduplication, and ethical web scraping. ## What's New (v2.0) **Enhanced Filter Observability & Analytics** - Four major enhancements transform the project from operational to enterprise-grade: 1. **Dynamic Filter Catalog Loading** - Dashboard automatically loads filter labels from Python catalog (zero manual sync) 2. **CI Metrics Anomaly Alerts** - Automated GitHub Issues for metrics calculation bugs (prevents production incidents) 3. **Filter Coverage Regression Tests** - Automated tests prevent filter telemetry regressions (100% detection coverage) 4. **Historical Filter Export** - Export filter metrics to Parquet for trend analysis and data warehouse integration See Filter Analytics Guide and CI Monitoring Guide for details. ## Key Features - **Multi-source data collection** - Wikipedia (~50k articles), BBC Somali news, HuggingFace MC4 (~100k-200k web docs), Språkbanken (66 academic corpora), TikTok comments (social media) - **Production-grade MLOps** - Structured logging, **MLFlow experiment tracking**, automated metrics, quality reports, crawl ledger for resume capability - **Quality-first architecture** - Two-tier deduplication (exact + near-duplicate), pluggable quality filters, unified silver dataset schema - **Advanced filter telemetry** - Per-filter breakdown tracking, historical analytics, regression prevention, automated anomaly detection - **Ethical web scraping** - Rate limiting, robots.txt compliance, conditional requests, configurable scraping policies - **Reproducible pipelines** - Timestamped artifacts, manifest versioning, Parquet storage with metadata sidecars - **Comprehensive testing** - 1,200+ tests with CI/CD on Ubuntu and macOS (Python 3.9- …