YonatanMoges/ethiomart

Fine-tuning a Named Entity Recognition (NER) model to extract key entities like products, prices, and locations from Amharic Telegram messages. # EthioMart NER Project ## Overview This repository contains the code and resources for fine-tuning a Named Entity Recognition (NER) model to extract key entities like products, prices, and locations from Amharic Telegram messages. The project involves data preprocessing, model training, and evaluation, with the fine-tuning process conducted using Google Colab. ## Directory Structure ├── notebooks/ # Contains Jupyter notebooks for demo and model training │ ├── scripts/ # Python scripts for preprocessing │ ├── scraping.py # Script for scraping data from Telegram channels │ ├── requirements.txt # List of required Python packages ├── README.md # Project documentation └── .gitignore # Files to ignore in the Git repo ## Setup Instructions ### 1. Clone the repository ```bash git clone github.com cd ethiomart ``` ### 2. Install dependencies Ensure that you have Python 3.7 or higher installed. Install the required packages by running: ```bash pip install -r requirements.txt ``` ### 3. Directory Details notebooks/: This directory contains Jupyter notebooks demonstrating preprocessing and the fine-tuning process. preprocessing.ipynb: Demonstrates the dataset preprocessing using the preprocessing.py script. fine_tuning.ipynb: This notebook handles fine-tuning the NER model. It includes data loading, tokenization, model training, and evaluation. This notebook was run on Google Colab to take advantage of GPU resources for faster training. scripts/: Contains Python scripts for scraping and preprocessing tasks. scraping.py: A Python script for scraping messages from multiple Ethiopian Telegram e-commerce channels. It fetches text, images, and metadata (e.g., timestamps, sender info) in real-time and stores them for preprocessing. preprocessing.py: A Python script for cleaning, tokenizing, and preparing raw text data for NER model training, as well as combining datasets. requirements.txt: A list of all the Python libraries and version …