Dagiayy/Amharic-E-commerce-Data-Extractor

# EthioMart Task 2: Named Entity Recognition (NER) Labeling ## Overview This submission fulfills Task 2 of the EthioMart project, which involves extracting and labeling **30 messages** from the `telegram_data.csv` dataset for Named Entity Recognition (NER). The labeled data identifies entities such as products, prices, and locations in Amharic and English Telegram messages, formatted in the CoNLL standard. --- ## Dataset Description - **Source**: `telegram_data.csv` - **Messages Extracted**: 30 unique messages - **Output File**: `labeled_data_corrected.conll` - **Entities Labeled**: - `B-Product`, `I-Product`: Product names (e.g., ጫማዎች, Hair Straightener) - `B-PRICE`, `I-PRICE`: Prices and their currency units (e.g., 2800, ብር) - `B-LOC`, `I-LOC`: Locations (e.g., ቦሌ, መሰረት ደፋር) - `O`: Non-entity tokens (e.g., phone numbers, Telegram handles, OCR gibberish) --- ## Labeling Process ### Data Selection: - **30 messages** were sampled from `telegram_data.csv` to ensure a diverse set of products, prices, and locations. - Messages with clear entity information were prioritized, while those dominated by OCR gibberish (e.g., ሕዴርውዱክ) were filtered or minimally included to meet the requirement. ### Tokenization: - A custom tokenizer was used to handle Amharic and English text, preserving words, numbers, and Telegram handles. - Compound location terms (e.g., መደሐንያለም) were split into correct forms (e.g., መድኃኔዓለም) using an updated `LOCATION_DICT`. ### Labeling: - Manual and automated labeling was performed to tag entities accurately. - Corrections were made to address issues such as: - Missed product names (e.g., ጫማዎች labeled as `O`, corrected to `B-Product`). - Inconsistent price tagging (e.g., 1450 labeled as `O`, corrected to `B-PRICE`). - Incorrect location tags (e.g., ቦሌ labeled as `O`, corrected to `B-LOC`). - OCR gibberish tokens were consistently labeled as `O`. ### Output Format: - The labeled data is stored in `labeled_data_corrected.conll` in CoNLL format, wi …