Reiten-10Academy/Speech_to_text_data_pipeline

This collaboration-based project is to build a data engineering pipeline that allows recording millions of Amharic and Swahili speakers reading digital texts on in-app and web platforms.. # Speech_to_text_data_pipeline **Table of content** - Overview - Install - Data - Folders ## Overview > The purpose of this week’s challenge is to build a data engineering pipeline that allows recording millions of Amharic and Swahili speakers reading digital texts in-app and web platforms. There are a number of large text corpora we will use > We will design and build a robust, large scale, fault tolerant, highly available Kafka cluster that can be used to post a sentence and receive an audio file. By the end of this project, we will produce a tool that can be deployed to process posting and receiving text and audio files from and into a data lake, apply transformation in a distributed manner, and load it into a warehouse in a suitable format to train a speech-t0-text model. ## Install ``` git clone github.com cd Speech_to_text_data_pipeline pip install -r requirements.txt ``` ## Data Data can be found here ## Pipeline > flow of data is shown with the arrows, and the order of execution is shown with the numbers attached to the bottom of the arrows. > - 1: Load original dataset as a csv to from unprocessed folder cleaning and selecting script to be processed by spark > - 2: Load a csv file containing id and text column to interim folder from cleaning script > - 3: Load cleaned data set from interim folder in s3 bucket to producer script that sends one row of data every X seconds to kafka topic > - 4: Send one row of data (sentence and Id) to kafka every X seconds. > - 5: Request for a sentence is sent out to a react frontend > - 6: The GET request is transfered from the react frontend to flask api > - 7: A kafka consumer requests to load latest sentence added to kafka topic > - 8: A kafka Topic responds back by sending a sentence and its id to the consumer > - 9: A flask api responds to the GET request and sends the sentence and id to the react frontend > - …