Silva3012/xhosa-nlp-dataset

My attempt at building a xhosa NLP dataset --- language: - xh - en license: other multilinguality: translation size_categories: 100K ⚠️ **XhosaNavy License Notice** > The XhosaNavy corpus was sourced from > OPUS > and originated from research at Stellenbosch > University (Herman Engelbrecht, Dept. of E&E > Engineering). OPUS explicitly states it does > not own the source text and cannot guarantee > redistribution rights. License confirmation > for commercial redistribution is currently > being sought from the original author. > **Commercial users should contact the > maintainer before using records where > `source == "xhosanavey"`.** ## 🎯 Intended Use Cases This dataset is designed specifically for: 1. **Language Modeling:** Training or continuing pre-training of Xhosa language models. 2. **Multilingual LLMs:** Fine-tuning multilingual models (e.g., AfroXLMR, AfriBERTa) to improve Xhosa comprehension. 3. **Machine Translation:** Building high-fidelity Xhosa-English and English-Xhosa translation systems. 4. **Sentiment Analysis:** Training commercial sentiment classifiers and customer feedback analyzers in Xhosa. 5. **Named Entity Recognition:** Teaching systems to correctly identify entities in Xhosa text. 6. **Commercial African Tech:** Providing training data for products targeting Xhosa speakers in the South African and broader African markets. ## 🤝 How to Contribute We welcome contributions from researchers and developers! If you have scripts for scraping additional Xhosa data, notice data quality issues, or want to contribute new parallel sets, please: 1. Fork the repository. 2. Set up your Python 3.13 environment. 3. Add or update scraper files inside the `/scrapers` directory using existing shared utilities. 4. Ensure text encoding fixes and language verification steps are applied. 5. Open a Pull Request detailing your additions or fixes. ## 📝 Citation If you use this dataset in a research publication or project, please cite it using the following format: ```bibtex @dataset{xhosa_nlp_dataset_ …