Aman-byte1/african-data-cleaning-pipeline

# African Language Data Cleaning Pipeline This repository contains a robust, checkpoint-enabled data cleaning pipeline designed to filter and improve the quality of a large-scale African language dataset. The pipeline is implemented in Python and utilizes the Hugging Face ecosystem for data loading, model inference, and checkpoint management. ## 1. System Requirements and Setup The pipeline is resource-intensive, requiring a machine with sufficient RAM and a GPU (NVIDIA recommended) for efficient model inference. ### 1.1. Prerequisites Ensure the following are installed on your server: * **Python 3.8+** * **Git** * **NVIDIA Drivers and CUDA Toolkit** (for GPU acceleration) ### 1.2. Installation 1. **Clone the Repository** ```bash git clone cd african_data_cleaning ``` 2. **Install Dependencies** The required libraries, including `torch`, `transformers`, `datasets`, and `sentence-transformers`, are listed in `requirements.txt`. ```bash pip install -r requirements.txt ``` ## 2. Pipeline Configuration The core configuration is managed within the `pipeline.py` script. The following parameters are pre-set based on your request: | Parameter | Value | Description | | :--- | :--- | :--- | | `HF_TOKEN` | `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` | Your Hugging Face Write Token for checkpointing and pushing results. | | `SOURCE_REPO` | `amanuelbyte/finetranslations-sentence-level` | The source dataset repository on Hugging Face. | | `TARGET_REPO` | `amanuelbyte/finetranslations-sentence-level-cleaned` | The target repository where cleaned data and checkpoints are saved. | | `BATCH_SIZE` | `25000` | Number of rows processed before a checkpoint is saved. | | `TARGET_GOAL` | `5,000,000` | The target number of clean pairs to collect for each language configuration. | ## 3. Filtering Stages and Thresholds The pipeline applies a rigorous, multi-stage filtering process to ensure high data quality. ### 3.1. Rule-Based Filtering and Deduplication | Rule | Threshol …