lea19019/multimodal-translation-pipeline

Speech-to-speech translation for Efik, Igbo, Swahili, and Xhosa. Fine-tuned NLLB, XTTS, and BLASER models with FastAPI microservices architecture. # Multimodal Translation Pipelines for Low-Resource African Languages ## Executive Summary A research project exploring speech-to-speech translation for four low-resource African languages (Efik, Igbo, Swahili, Xhosa) through model fine-tuning and comprehensive evaluation. All fine-tuned models are publicly available on Hugging Face. **Core Technologies:** PyTorch • Hugging Face Transformers • Whisper (ASR) • NLLB-600M (NMT) • Coqui XTTS (TTS) • FastAPI • BLASER 2.0 **Key Achievements:** - Fine-tuned 6 XTTS model variants + NLLB + 4 custom BLASER encoders - Benchmarked 10+ pipeline combinations with comprehensive evaluation (BLEU, chrF, COMET, MCD, BLASER) - Designed microservice architecture for experimentation - 140+ hours of research and engineering on supercomputing infrastructure (SLURM, multi-GPU) **Jump to:** System Architecture • Technical Stack • Results ## Project Summary This repository documents a research and engineering effort focused on building, evaluating, and scaling speech-to-speech translation pipelines for four low-resource African languages: Efik, Igbo, Swahili, and Xhosa. The project blends hands-on deep learning, large-scale experimentation, and robust engineering: - Fine-tuned multiple versions of Coqui XTTS for TTS on African languages - Fine-tuned Meta NLLB-600M for neural machine translation (NMT) - Developed and integrated custom BLASER 2.0 encoders for speech-to-speech evaluation - Designed and benchmarked 10+ pipeline combinations (ASR, NMT, TTS) - Built a modular microservice architecture for scalable, reproducible experiments - Ran and debugged experiments on supercomputing clusters (SLURM, multi-GPU) - Automated data processing, model training, and evaluation workflows - Explored different data types, model architectures, and evaluation strategies **Full technical report:** Project Report ## System Overview The system uses a **microservices architecture** with HTTP/REST communication. Each service runs independentl …