Norbert49/webtext-crawler-nlp

High-volume multilingual web scraping and dataset curation pipeline for low-resource languages (Swahili, Yoruba, etc.)—cleaned, deduplicated, timestamped text datasets ready for training large language models and NLP research # Multilingual Language Model Dataset Pipeline This repository contains a scalable, production-grade web scraping and data cleaning pipeline designed for multilingual textual data acquisition—specifically targeting Swahili and Yoruba as low-resource languages. ## Features - 🔎 Language-aware web crawling using `fastText` and `langdetect` - 🧹 HTML + JS stripping, tag removal, and text normalization - 🚫 Filtering of adult, extremist, and low-density content - 🧠 Paragraph-level deduplication and language scoring - 📁 Organized JSONL output with structured metadata and timestamps - ⚙️ Built in Python using `requests`, `BeautifulSoup`, and `asyncio` ## Ideal For - Pretraining or fine-tuning LLMs (Large Language Models) - Corpus creation for NLP tasks in African languages - Cross-lingual transfer learning research ## Sample Output Format Each `.jsonl` line represents a single cleaned document with metadata fields including `source`, `lang`, `text`, `time`, `meta` (title, author, crawl_date, confidence score, etc.). --- Would you like me to generate this full GitHub repo layout (with `README.md`, `requirements.txt`, `pipeline.py`, etc.) for you in a ZIP file or directly push to a GitHub repo on your account?