LunganiGcabashe/IsiZulu-Text-Processing-Research

# IsiZulu Text Processing Research ## Overview This project focuses on collecting, cleaning, preprocessing, and analyzing isiZulu textual data extracted from public Facebook pages for Natural Language Processing (NLP) and Large Language Model (LLM) research purposes. The research explores methods for preparing low-resource African language datasets suitable for machine learning and language model training. --- ## Research Objective The primary objective of this project is to investigate the accumulation and preprocessing of isiZulu textual data from public Facebook pages to support future development of language technologies and Large Language Models for African languages. --- ## Features - Facebook data extraction using Apify - JSON data processing and structuring - Dataset merging and preprocessing - Text cleaning and normalization - Emoji and URL removal - Sentence segmentation - Language detection using LangID - Filtering English content - English stopword detection and removal - Data visualization using Matplotlib - Preparation of NLP-ready datasets --- ## Technologies Used - Python - Google Colab - Apify - Pandas - NumPy - Matplotlib - LangID - Scikit-learn - JSON --- ## Project Workflow ### 1. Data Collection Textual data was collected from public Facebook pages using Apify and stored in JSON format. ### 2. Data Combination Multiple extracted JSON files were merged into a single structured dataset. ### 3. Text Cleaning The preprocessing stage included: - Converting text to lowercase - Removing emojis - Removing URLs - Removing hashtags - Removing HTML tags - Normalizing punctuation and spaces ### 4. Sentence Segmentation Long text entries were separated into sentence-level structures for easier NLP processing. ### 5. Language Detection Language detection was performed using LangID to identify isiZulu and English text. ### 6. Filtering Non-isiZulu Content English sentences were filtered out to retain isiZulu-focused textual data. ### 7. Stopwo …