LahadMbacke/Scraper-Wikipedia-Wolof

# Wolof Wikipedia scraper This repository contains tools to fetch and clean content from the Wolof Wikipedia (wo.wikipedia.org). Purpose ------- Retrieve all (or a subset of) main-namespace pages from Wolof Wikipedia, store the raw wikitext in a TSV file, and provide a cleaner that converts that wikitext to plain text suitable for training or analysis. Main files ---------- - `scrap.py`: script that iterates over pages via the MediaWiki API and writes each page as a TSV line `title\t `. - `clean.py`: script that cleans a TSV produced by `scrap.py` and produces a TSV `title\t `. - `wolof_corpus.txt`: example corpus produced by `scrap.py`. - `wolof_corpus_clean.txt`: example cleaned output produced by `clean.py`. Prerequisites ------------- - Python 3.8+ - Python dependency: ``` pip install requests ``` Usage ----- 1) Fetch pages from Wolof Wikipedia `scrap.py` calls the API `wo.wikipedia.org` and writes each page as a single TSV line: ``` title\twikitext_with_\\n_escapes\n ``` Examples: Fetch 100 pages into `wolof_corpus.txt`: ``` python3 scrap.py --out wolof_corpus.txt --limit 100 ``` Fetch everything (warning: this is long and heavy — use `--sleep` and `--resume`): ``` python3 scrap.py --out wolof_corpus.txt --limit 0 --sleep 1.0 --resume ``` Important `scrap.py` options: - `--out`: output file path (default `wolof_corpus.txt`) - `--limit`: max pages to fetch (0 = all) - `--sleep`: seconds between page fetches to reduce server load (default 0.5) - `--resume`: append to existing output and skip titles already present 2) Clean the corpus `clean.py` reads the TSV produced by `scrap.py` and writes a cleaned TSV `title\tcleaned_text`. Default cleaning behavior: - removes HTML comments and ` ` tags - attempts to remove templates `{{...}}`, tables and images - converts wiki links `[[A|B]]` → `B` and `[[A]]` → `A` - converts external links `[http... label]` → `label` - strips remaining HTML tags and bold/italic markup - keeps alp …