MabeyaG/scoutai

ScoutAI is a modular, end-to-end platform for scraping, processing, and analyzing African real estate listings. It features robust data collection (Scrapy spiders), normalization (Java processor), and machine learning-based price prediction (Python). The platform includes a CLI, GUI, and is designed for extensibility and automation # ScoutAI: African Real Estate Data Platform ## Overview ScoutAI is a modular platform for robustly scraping, processing, and analyzing African real estate listings. It consists of: - **scraper-python**: Scrapy spiders for collecting property data from multiple sites. - **processor-java**: Java module for validating and normalizing raw property data. - **predictor-python**: Python module for training and running property price prediction models. - **cli-tool**: (Planned) Command-line interface for advanced users and automation. --- ## Step 1: Scrape Property Listings - Navigate to `scraper-python`. - Run the Scrapy spiders to collect property data from supported sites (e.g., Property24, PigiaMe, Jiji, Expat-Dakar, HassConsult). - Output is written as JSON (e.g., `data/property24_listings.json`). - Spiders use robust selectors, proxy rotation, and fallback logic for maximum completeness. ## Step 2: Process & Normalize Data - Use the `processor-java` module to validate and normalize the raw JSON data. - This step ensures all fields (price, bedrooms, bathrooms, area, etc.) are standardized and ready for analysis. - (See `processor-java/README.md` for details.) ## Step 3: Predict Property Prices - Use the `predictor-python` module to train and run machine learning models for price prediction. - Input: Cleaned/normalized data from Step 2. - Output: Price predictions and analytics. - (See `predictor-python/README.md` for details.) ## Step 4: CLI Tool - The `cli-tool` module will provide a command-line interface for power users. - (Planned for future development.) --- ## Data Columns & Feature Engineering Each listing now includes the following fields for robust analysis and ML: | Column | Description | |---------------------|------------------------------------------------------------------| | title | Listing title | | location …