Kevinobote/swahili-stopwords-nltk

# Swahili Stopwords for NLTK A research-grade pipeline for extracting and validating Swahili (Kiswahili) stopwords for integration into the NLTK framework. ## Overview This project implements a statistically grounded, linguistically informed methodology for constructing a Swahili stopword corpus. The pipeline combines frequency analysis, Zipf's law, TF-IDF validation, and Swahili-specific linguistic rules to produce a high-quality stopword list suitable for NLP applications. **Research Context:** This work is part of a peer-reviewed publication for AAIAC 2026 (IEEE Xplore indexed). ## Features - **Multi-domain corpus support**: Wikipedia, Mozilla Common Voice, news sources - **Statistical methods**: Frequency analysis, Zipf's law, TF-IDF - **Linguistic filtering**: Swahili-specific rules for agglutinative morphology - **NLTK compatibility**: Output formatted for direct NLTK integration - **Reproducible pipeline**: Modular, well-documented, research-grade code - **Evaluation framework**: Downstream task validation and metrics ## Project Structure ``` swahili-stopwords-nltk/ │ ├── data/ # Corpus data │ ├── raw/ # Raw corpus files │ │ ├── wikipedia/ │ │ ├── common_voice/ │ │ └── news/ │ ├── cleaned/ # Preprocessed corpus │ └── samples/ # Test samples │ ├── src/ # Source code │ ├── preprocessing/ # Text preprocessing │ │ ├── normalize.py # Normalization (lowercasing, punctuation) │ │ ├── tokenize.py # NLTK-compatible tokenization │ │ └── clean.py # Quality filtering │ ├── analysis/ # Statistical analysis │ │ ├── frequency.py # Frequency distribution │ │ ├── tfidf.py # TF-IDF analysis │ │ └── zipf.py # Zipf's law analysis │ ├── linguistics/ # Linguistic filtering │ │ ├── pos_filter.py # P …