laleye/langquality

Language Quality Toolkit for Low-Resource Languages. # LangQuality - Language Quality Toolkit for Low-Resource Languages A modular, extensible Python toolkit for analyzing the quality of text and audio datasets for low-resource languages. LangQuality helps researchers and developers ensure high-quality datasets for training NLP models (ASR, machine translation, language models) across diverse languages. ## ✨ Key Features ### 🌍 Multi-Language Support via Language Packs - **Language-agnostic architecture**: Works with any language through configurable Language Packs - **Pre-built packs**: Fongbe, French, English, and more - **Easy customization**: Create your own Language Pack in minutes - **Community-driven**: Share and discover Language Packs from the community ### 🔍 Comprehensive Quality Analysis - **Structural Analysis**: Sentence length distribution, outlier detection, statistical metrics - **Linguistic Analysis**: Readability scores, lexical complexity, morphological features - **Diversity Analysis**: Vocabulary richness (TTR), n-gram distributions, duplicate detection - **Domain Analysis**: Thematic balance, under/over-represented categories - **Gender Bias Detection**: Gender representation, stereotype detection, balance metrics ### 🔌 Extensible Plugin System - **Custom analyzers**: Add your own analysis modules without modifying core code - **Automatic discovery**: Drop plugins into a directory and they're automatically loaded - **Language-specific analyzers**: Create analyzers tailored to specific languages ### 📊 Rich Output Formats - **Interactive Dashboard**: Beautiful HTML visualizations with Plotly - **Actionable Recommendations**: Prioritized suggestions based on best practices - **Multiple Exports**: JSON, CSV, PDF reports, execution logs - **Per-sentence annotations**: Quality scores and flags for each sentence ## 🚀 Quick Start ### Installation ```bash # Install from PyPI pip install langquality # Install with all optional dependencies pip install langquality[all] # Download language m …