VaporFoxLash/LanguageIdentifier

Notebook for the ExploreAI Academy Classification Hackathon, which focuses on language identification for South Africa's 11 official languages. The objective is to identify the language of a given text, leveraging natural language processing techniques. # ExploreAI Academy Classification Hackathon This repository contains the notebook for the ExploreAI Academy Classification Hackathon, which focuses on language identification for South Africa's 11 official languages. The objective is to identify the language of a given text, leveraging natural language processing techniques. ## Overview South Africa is known for its rich linguistic diversity, with 11 official languages. Language plays a crucial role in the country's social, cultural, economic, and political life. With a multilingual population, it becomes essential to develop systems and devices that can effectively communicate in multiple languages. The challenge in this hackathon is to determine the language of a given text written in any of South Africa's official languages. This task is a form of language identification in the field of natural language processing (NLP). Image credit ## Modeling We explore different models for language identification. We train and evaluate various models using different algorithms and techniques. ### Data Preprocessing Before building our models, we need to preprocess the data. This involves cleaning and transforming the text data into a format suitable for modeling. The preprocessing steps include: - **Tokenization:** Splitting the text into individual words or tokens. - **Removing stopwords:** Removing common words that do not contribute much to the language identification task. - **Lemmatization or stemming:** Reducing words to their base or root form. - **Vectorization:** Converting text data into numerical feature vectors. *Model 1:* **_Logistic Regression_** Logistic Regression is a popular algorithm for classification tasks. It works well with linearly separable data and can provide probabilities for each class. *Model 2:* **_Support Vector Machine (SVM)_** Support Vector Machine is a powerful algorithm for both classification and regression tasks. It finds the best separating hyperplane to classify data …