HaytamRais/NLP_AtlasChat9B_QLoRA_Adapter

This repo contains the code I used to contribute in a LLM for Darija called AtlasChat-9B using QLoRA due to low computational resources ! (From Scraping scripts -> Final Model) # NLP_AtlasChat9B_QLoRA_Adapter ## Moroccan NLP Story Scraper A specialized Scrapy-based web crawler designed to extract stories and articles from `9esa.com` for Moroccan Arabic (Darija) NLP research and sentiment analysis. ## 🚀 Quick Start To start scraping and save the results to a JSON Lines file: ```bash scrapy crawl qesa -o moroccan_corpus.jsonl ``` --- ## 🏗️ Architecture Overview The project is built on the **Scrapy** framework, following a modular architecture that separates crawling logic, data structures, and post-processing pipelines. ### 🧩 File-by-File Deep Dive #### 1. `qesa_spider.py` (The Engine) Located in `NLP_SCRAPING_DAT/spiders/qesa_spider.py`, this is the core of the scraper. - **Multi-Stage Parsing**: - `parse`: Scans the main archive list to find category/label links. - `parse_story_label`: Navigates through intermediate pages containing lists of individual chapters or stories. It includes custom pagination logic to follow "Next" links sequentially. - `parse_article`: The final extraction stage that pulls the title, raw text, and metadata from individual story pages. - **Encoding Management**: Explicitly handles UTF-8 decoding to ensure Arabic characters are preserved correctly from the web response. #### 2. `items.py` (The Data Schema) Defines the `ArticleItem` class, which acts as a structured container for the scraped data. - Fields include `url`, `title`, `raw_text`, `publish_date`, `category`, and placeholders for processed data like `clean_text` and `sentiment_label`. #### 3. `pipelines.py` (The Processing Factory) Handles data cleaning and enrichment after extraction. - **ArabicCleaningPipeline**: Uses the `PyArabic` library to: - Normalize Hamzas and Ligatures. - Remove diacritics (Harakat) for NLP uniformity. - Strip punctuation and non-Arabic characters while preserving the core script. - **SentimentClassificationPipeline**: A placeholder for integrating machine learning models. It demonstrates how to classify text into a 7 …