vesmunger/Kalenjin-Text-To-Speech

This is a trial run to create speech by finetuning an open source foundation model and inputting open source datasets to generate the speech Kalenjin-Kiswahili Lip-Sync AI Project Project Overview This Google Colab notebook demonstrates an exciting multi-stage project focused on developing AI capabilities for Kalenjin and Kiswahili languages. The primary goal is to leverage natural language processing (NLP) and text-to-speech (TTS) to bridge linguistic gaps and eventually enable advanced applications like lip-sync animated video generation from static images and audio. Currently, the project focuses on setting up the environment, processing a parallel corpus, performing text-to-speech synthesis, and basic video creation. Features Environment Setup: Installs essential Python libraries for NLP, audio processing, and data manipulation, ensuring GPU readiness. Kalenjin-Kiswahili Parallel Corpus Handling: Loads, inspects, and prepares a parallel corpus for linguistic tasks. Text-to-Speech (TTS): Initializes and utilizes a Swahili TTS model (from Hugging Face transformers) to convert text into spoken audio for both Kiswahili and Kalenjin sentences. Interactive Translation & Speech: Allows users to input Swahili phrases, find their Kalenjin translations in the corpus, and hear the translated Kalenjin sentence spoken aloud. Basic Video Generation: Merges generated audio clips with a static background image to create simple MP4 video files, laying groundwork for more complex animation. Technologies Used Python: Primary programming language. Pandas: Data manipulation and analysis. Hugging Face Transformers: For state-of-the-art NLP models, specifically Text-to-Speech (TTS) pipelines (facebook/mms-tts-swh). SoundFile: Reading and writing audio files. MoviePy: Video editing and creation from audio and images. IPython.display: For embedding audio and video directly in the notebook. PyTorch: Underlying deep learning framework. Dataset The project utilizes the Kalenjin-Kiswahili Parallel Corpus, a CSV file (Kalenjin-Kiswahili-Parallel-Corpus.csv) containing paired sentences in both languages. This dataset is crucial …