miirshe/Somali-NLP-Document-Similarity-using-BERT

The project utilizes Natural Language Processing (NLP) techniques and leverages the BERT model to compute the similarity between documents. The steps include data preprocessing, handling missing values, removing duplicates, removing stop words, exploratory data analysis (EDA), and training and evaluating the BERT model. ### Somali NLP Document Similarity using BERT ========================================== This repository contains the implementation of a document similarity project focused on the Somali language. The project utilizes Natural Language Processing (NLP) techniques and leverages the BERT model to compute the similarity between documents. The steps include data preprocessing, handling missing values, removing duplicates, removing stop words, exploratory data analysis (EDA), and training and evaluating the BERT model. ### Features: - Data reading and preprocessing with Pandas - Handling missing values and removing duplicates - Stop words removal specific to the Somali language - Exploratory Data Analysis (EDA) to understand the dataset - Tokenization and preparation of data for BERT - Training and evaluation of the BERT model for document similarity - Computing document similarity using cosine similarity This project is a part of a graduation project aimed at enhancing NLP applications for the Somali language. ```