tegbiye/Amharic-ecommerce-data-extractor

# Building an Amharic E-commerce Data Extractor ## Transform messy Telegram posts into a smart FinTech engine that reveals which vendors are the best candidates for a loan. --- ## Task 1: Data Ingestion and Data Preprocessing 1. Data scraping Scraped data from seven channels namely: - Zemen Express® - NEVA COMPUTER® - HellooMarket - ሞደርን ሾፒንግ ሴንተር MODERN SHOPPING CENTER - qnash.com - ቅናሽ ®️ - አዳማ ገበያ - Adama gebeya - Sheger online-store - Scraped around 37378 of raw data. 2. Data Preprocessing - Checked for the missing values and found some significant amount of missing values fixed - made cleaning of the Amharic text as the task is for Amharic language where removed english text and other punctuations, emojis, tags. - Saved the cleaned message to be used for the task 2 ## Task 2: Label a Subset of Dataset in CoNLL Format Based on the task objective 30–50 messages from the "Message" column of the provided cleaned_message.csv dataset is used to create in CoNLL format for Named Entity Recognition (NER). The entities to be identified and labeled include: • Product: Items being advertised • Price: Monetary values • Location: Place names or addresses • Labels follow the BIO scheme: B- (Beginning), I- (Inside), and O (Outside) for each entity type. 1. Initial Script Development: o Created a Python script (label_conll_amharic.py) to load the dataset, tokenize messages, label entities, and save the output in CoNLL format. o Used NLTK's word_tokenize for tokenization, suitable for Amharic text. o Defined keyword lists for products and locations and a regular expression pattern for prices o Filtered out invalid messages and processed up to 50 messages. 2. Dataset-Specific Updates: o Updated the script to handle the provided cleaned_message.csv, which contains 37,377 rows, with the "Message" column in Amharic. o Refined keyword lists based on the dataset's content for products, and for locations. o Enhanced product detection with contextual keywords to captu …