tadiwamark/Shona-Language-Model

A shona language model where a user types in the first five words of a sentence and the model predicts the next 3 words. # Shona Language Model Project ## Overview This project encompasses the development of a language model for the Shona language, focusing predominantly on Jehovah's Witness reading material. The goal is to develop a model capable of understanding and predicting subsequent words in a given sequence of Shona words, utilizing advanced NLP and machine learning techniques. ## Table of Contents - Objective - Dataset - Methodology - Text Extraction - Text Preprocessing - Word Embeddings - Model Development - Training & Testing - Limitations - Solutions and Workarounds - Disclaimer - Conclusion - Further Development ## Objective The primary objective is to construct a proficient model for comprehending and generating text in the Shona language, harnessing modern natural language processing and machine learning methodologies. ## Dataset The dataset used is primarily composed of Shona language reading materials from Jehovah's Witness literature. ## Methodology ### Text Extraction The PyPDF2 library is used for extracting text from PDF files containing the Shona language literary material. The `extract_text_from_pdf` function reads and extracts text from each page of the PDF. ```python import PyPDF2 def extract_text_from_pdf(pdf_path): with open(pdf_path, "rb") as file: reader = PyPDF2.PdfReader(file) text = "".join([page.extract_text() for page in reader.pages]) return text ``` ### Text Preprocessing The text data is tokenized using the Tokenizer class from the keras.preprocessing.text, converting the text into a sequence of tokens (words). ```python from keras.preprocessing.text import Tokenizer tokenizer = Tokenizer() tokenizer.fit_on_texts([shona_text]) ``` ### Word Embeddings Word embeddings are generated using Word2Vec from gensim.models, allowing the conversion of words into numerical vectors, essential for the model to understand the relationships and similarities between different words. ```python from gensim.models import Word2Vec model_gensim = Word2Vec(sente …