silsgah/rag-llm-system

RAG system for news portal across the country Ghana # 🚀 Production-Ready RAG System with LLM Fine-Tuning > A complete, production-ready RAG (Retrieval-Augmented Generation) system with custom LLM fine-tuning, following industry best practices and MLOps principles. --- ## 🎯 Project Highlights This is a **full-stack, production-grade AI system** that demonstrates: - ✅ **Complete ML Pipeline**: From raw data ingestion to production deployment - ✅ **Advanced RAG Implementation**: Query expansion, self-querying, and cross-encoder reranking - ✅ **Custom LLM Fine-Tuning**: Supervised Fine-Tuning (SFT) + Direct Preference Optimization (DPO) - ✅ **Production Infrastructure**: Microservices architecture with separate RAG and inference services - ✅ **Multi-Cloud Deployment**: AWS SageMaker, RunPod, and Render.com support - ✅ **Full Observability**: Experiment tracking (Comet ML) and prompt monitoring (Opik) - ✅ **Clean Architecture**: Domain-Driven Design with ~6,000 lines of well-structured code - ✅ **CI/CD Pipeline**: Automated testing, linting, and deployment workflows --- ## 📋 Table of Contents - Features - Architecture - Quick Start - Project Structure - Usage - Deployment - Configuration - Documentation --- ## ✨ Features ### Core Capabilities **Data Pipeline** - Multi-source web crawlers (LinkedIn, Medium, GitHub, news sites) - Automated document processing, chunking, and cleaning - Vector embeddings generation with sentence-transformers - Storage in MongoDB (documents) and Qdrant (vectors) **Advanced RAG System** - **Self-Query**: Automatic metadata extraction (author, topic, platform) - **Query Expansion**: Generates 3 diverse query variations for better recall - **Hybrid Search**: Combines semantic search with metadata filtering - **Cross-Encoder Reranking**: Precision scoring of retrieved documents - **Context-Aware Generation**: Structured prompts with retrieved context **LLM Fine-Tuning** - Automated dataset generation (instruction + preference pairs) - Supervised Fine-Tuning (SFT) on AWS SageMake …