mikek1337/Building-an-Amharic-E-commerce-Data-Extractor-week4

# Building an Amharic E-commerce Data Extractor This project focuses on fine-tuning LLM’s for Amharic Named Entity Recognition (NER) system that extracts key business entities such as product names, prices, and Locations, from text, images, and documents shared across these Telegram channels. The extracted data will be used to populate EthioMart's centralised database, making it a comprehensive e-commerce hub. # Amharic E-commerce Data Extractor This project extracts, preprocesses, and analyzes Amharic e-commerce data from Telegram channels. It is designed to help collect and clean data for downstream analysis and machine learning tasks. ## Project Structure ``` . ├── .env ├── .gitignore ├── channel_scraper.session ├── channels.txt ├── labeled_data.txt ├── README.md ├── data/ │ ├── cleaned_messages.csv │ ├── message_tokens.txt │ └── messages.csv ├── notebooks/ │ ├── __init__.py │ ├── channel_scraper.session │ ├── inital_analysis.ipynb │ └── README.md ├── scripts/ │ ├── __init__.py │ ├── preprocess.py │ ├── README.md │ ├── scrapy.py │ └── __pycache__/ │ └── preprocess.cpython-313.pyc ├── src/ ├── tests/ │ ├── __init__.py │ └── test_preprocess.py └── .github/ └── workflows/ └── ci.yml ``` - `scripts/scrapy.py`: Scrapes messages from Telegram channels and saves them to `data/messages.csv`. - `scripts/preprocess.py`: Provides text preprocessing utilities for cleaning and normalizing messages. - `notebooks/inital_analysis.ipynb`: Jupyter notebook for data exploration and further processing. - `channels.txt`: List of Telegram channel usernames to scrape. - `.env`: Environment variables for Telegram API credentials. ## Setup 1. **Install dependencies** Make sure you have Python 3.10+ and pip installed. Then run: ```sh pip install -r requirements.txt ``` 2. **Configure environment variables** Create a `.env` file in the project root with your Telegram API credentials: ``` APP_KEY=your_telegram_api_hash APP_ID=your_telegram_api_id …