uzumstanley/Nigeria-LLM

# 🇳🇬 Naigeria LLM WebScrap Dataset for LLM Training NaijaWeb is a web scraping project inspired by the FineWeb paper and the WebText dataset, including the OpenWebText dataset. The scraping was performed on **Google Colab** due to its **free memory** and **easy integration with Google Drive**. Several notebooks (`clean_download_data.ipynb`, `clean_download_data_403.ipynb`, `download_webpages.ipynb`, `webscrape_403.ipynb`, and `webscrape_nairaland.ipynb`) were run across 9 different Colab notebooks (3 notebooks per Colab account) to expedite the process. All notebooks were linked to a single Google Drive folder: `/content/drive/MyDrive/nairaland_webtext`, where the files were saved. The dataset is available on Hugging Face: NaijaWeb Dataset. ## Notebooks and Process ### webscrape_nairaland.ipynb This notebook scrapes and extracts posts from a specific **section** on Nairaland. The script first collects all **post links** for each section and saves them to a pickle file. It then downloads the individual posts. Due to Colab's limited runtime, the process was distributed across 9 notebooks, where the long list of post links was split into smaller chunks to speed up the downloads. ### extract_outboundlinks.ipynb This notebook extracts all outbound links from the downloaded posts, filters out certain domains, performs basic cleaning (removing full stops at the end of links and consecutive full stops), and saves the cleaned links to a CSV file. ### download_webpages.ipynb This notebook downloads webpages from the outbound links extracted in the previous step. The links are downloaded in batches of 1,000 and saved as pickle files. This process was also run across 9 notebooks to save time. ### clean_download_data.ipynb This notebook uses Trafilatura (as inspired by the FineWeb paper) to extract and clean the downloaded webpages. Pages that returned a "403 Forbidden" response were saved for later handling. ### webscrape_403.ipynb This notebook redownloads webpages th …