achimdesa/week-5

10 Academy Kifiya AI Mastery Training Program Week 5 challenge (Amharic Named Entity Recognition (NER) System for EthioMart) # week-5 # EthioMart NER Project ## Overview ### Business Need EthioMart aims to become the primary hub for Telegram-based e-commerce activities in Ethiopia. With the increasing popularity of Telegram for business transactions, various independent e-commerce channels have emerged, each facilitating its own operations. However, this decentralization presents challenges for both vendors and customers who need to manage multiple channels for product discovery, order placement, and communication. To solve this problem, EthioMart plans to create a single centralized platform that consolidates real-time data from multiple e-commerce Telegram channels into one unified channel. This will provide a seamless experience for customers to explore and interact with multiple vendors in one place. ### Project Focus This project focuses on fine-tuning Large Language Models (LLMs) for an Amharic Named Entity Recognition (NER) system that extracts key business entities such as product names, prices, and locations from text, images, and documents shared across these Telegram channels. The extracted data will populate EthioMart's centralized database, making it a comprehensive e-commerce hub. ## Key Objectives - Real-time data extraction from Telegram channels. - Fine-tuning LLMs to extract entities like product names, prices, and locations. ## Task 1: Data Ingestion and Preprocessing Data Collection: The data was collected from Ethiopian-based e-commerce Telegram channels. For this, a custom scraper was built to capture text messages, product images, and other relevant data in real-time. Steps: 1. Telegram Scraper: A custom message ingestion system was implemented to fetch real-time messages, images, and documents posted in the selected Telegram channels. 2. Data Preprocessing: The messages were processed, focusing on Amharic text. Preprocessing included: o Tokenization: Splitting the text into tokens (words). o Normalization: Handling variations in Amharic spelling and chara …