Speech-to-text-Kafka-Airflow-Spark/StoTkas

Data engineering pipeline that allows recording millions of Amharic and Swahili speakers reading digital texts in-app and web platforms Speech-to-text data collection with Kafka, Airflow, and Spark ## Table of Contents 1. Introduction 2. Project Structure - data - notebooks - scripts - tests - logs - root folder 3. Installation guide ## Introduction Large and quality datasets are critical to ensure the performance, fairness, robustness, reliability, and scalability of ML systems. Data scientists often lack diverse and large datasets to train and test the machine learning models they design. This project focuses on developing a tool that can be deployed to process posting and receiving text and audio files from and into a data lake, apply transformation in a distributed manner, and load it into a warehouse in a suitable format to train a speech-t0-text model. The general objective of the project is to develop a data engineering pipeline using Apache Kafka, Apache Spark and Airflow to allow collection of millions of Amharic and Swahili audio recordings from speakers reading digital text in app and web platforms. These recordings can be used to produce a large and diverse dataset for training and testing speech-to-text processing models. The proposed data pipeline was built on Apache Kafka, an open-source distributed event streaming platform. By combining messaging, storage, and stream processing, the data pipeline allow collection, storage and analysis of real-time audio datasets. The data pipeline consists of the following key components: Data producers Data consumers Apache Kafka cluster Amazon S3 bucket Connectors Apache Spark Stream preprocessors ## Project Structure ### images: - `images/` the folder where all snapshot for the project are stored. ### logs: - `logs/` the folder where script logs are stored. ### data: - `data/` the folder where the dataset files are stored. ### .github: - `.github/`: the folder where github actions and unit-tests are integrated. - `cml.yaml`: the file where the cml configuration is stored. ### .vscode: - `.vscode/`: the folder whe …