OsalauYemi/Data_scrapping_Yoruba

# Yoruba Text Dataset Collection This repository contains a growing dataset of authentic Yoruba-language documents intended for natural language processing (NLP), linguistic research, and cultural preservation. ## Structure - `documents/` – Contains `.txt` or `.pdf` files of each document. - `metadata_batch_1.jsonl` – Metadata for Batch 1 (50 documents), one JSON object per line. ## Metadata Format (`.jsonl`) Each entry includes: - `id`: MD5 hash of the document content. - `title`: Title of the document. - `author`: Author's name if available. - `content`: Full raw Yoruba text. - `url`: Source URL (if available). - `word_count`: Total word count. - `language`: ISO code (`yo` for Yoruba) ## Example JSON Line ```json { "id": "0f2b5f2a34ecac1ce76c2b4b44c5bd60", "title": "Gbogbogbo Oro Yoruba", "author": "Unknown", "content": "Osu me̩rin pere ni Alaga Arabinrin Rekiya...", "url": "iroyinowuro.com.ng", "word_count": 162, "language": "yo" } ``` ## Goals - Build a 200+ document Yoruba dataset in clean, text-based format. - Enable applications in speech recognition, translation, and literacy tools. ## How to Contribute You can submit `.txt` files in the `documents/` folder and add entries to the `metadata_batch_X.jsonl` files accordingly.