robegetachew/EthioMart-LLM

EthioMart is a Telegram-based e-commerce hub for Ethiopia, centralizing data from independent channels. It fine-tunes LLMs for Amharic NER to extract business entities from text, images, and documents, enabling seamless product discovery and vendor interaction on a unified platform. # EthioMart Data Extraction & Labeling EthioMart focuses on extracting and labeling data from Ethiopian Telegram e-commerce channels to support Named Entity Recognition (NER) tasks. The primary objective is to identify and label entities such as **products**, **prices**, and **locations** from Amharic text within messages, enabling seamless integration with machine learning models. --- ## **Project Overview** The workflow is structured into the following key steps: ### **1. Data Collection** Scrape data from Ethiopian Telegram e-commerce channels using a custom-built Telegram scraper. ### **2. Data Preprocessing** Clean and format the scraped data: - Filter messages containing Amharic text. - Tokenize and normalize the content for consistency. ### **3. Entity Labeling** Manually label a subset of messages to identify and categorize entities using the CoNLL format: - **B-PRODUCT**: Beginning of a product entity. - **I-PRODUCT**: Inside a product entity. - **B-LOC**: Beginning of a location entity. - **I-LOC**: Inside a location entity. - **B-PRICE**: Beginning of a price entity. - **I-PRICE**: Inside a price entity. - **O**: Tokens that do not belong to any entity. ### **4. Output** Save the labeled data in the CoNLL format for future use in machine learning models. --- ## **Prerequisites** Ensure you have **Python 3.7 or higher** installed. ### Required Libraries: - **telethon**: For Telegram scraping - **pandas**: For data manipulation - **re**: For handling regular expressions - **dotenv**: For loading environment variables - **asyncio**: For managing asynchronous operations Install the required libraries using: ```bash pip install -r requirements.txt ``` --- ## **Setup** 1. **Create a `.env` file**: Add your Telegram API credentials. Obtain these credentials by creating a bot via BotFather. Example `.env` file: ``` TG_API_ID=your_api_id TG_API_HASH=your_api_hash PHONE=your_phone_number ``` 2. **Run the Telegram Scraper**: Use the scraper to coll …