visiontech-102/RealTime-Trending-Topic-Detection-System

This project presents the design and implementation of a real-time trending topic detection system that analyzes Twitter (X) data in both English and Somali languages. The system aims to identify, track, and visualize emerging topics as they occur, enabling timely insights into social media discussions. # Real-Time Trending Topic Detection System for English and Somali Tweets Final Year Project (FYP 2026). A full-stack bilingual topic modeling system that ingests Twitter/X data, trains three competing unsupervised models (LDA, NMF, BERTopic) on the same corpus, evaluates them on identical intrinsic metrics, selects a winner by measured performance, tunes it via hyperparameter search, and deploys it for real-time trending topic output. --- ## System Architecture The full pipeline runs end-to-end in this order: ``` Twitter API v2 → raw_tweets (MongoDB) → corpus_loader (shared — all three models use the same loaded slice) ┌──────────────────────────────────────────────────────────┐ │ LDA training NMF training BERTopic training│ │ (gensim BoW) (sklearn TF-IDF) (SentenceTransformer→UMAP→HDBSCAN→c-TF-IDF) └──────────────────────────────────────────────────────────┘ → Intrinsic evaluation per model (same reference corpus, same TOP_N_WORDS): C_v coherence / U_Mass coherence / Topic Diversity split three ways: English / Somali / Combined → coherence_diversity.json (single source of truth for all downstream steps) → Metrics-driven winner selection (highest mean C_v; diversity tiebreak) → Enhancement: hyperparameter sweep on winner only → before_after.json → Model comparison report (all three models, all metrics) → latest_comparison.json → periodic_winner_training_loop: retrains winner on new tweets every N minutes → detected_trends (MongoDB) — topics with trend_score = 0.6×volume + 0.4×engagement → FastAPI REST endpoints → React / Vite dashboard ``` --- ## Project Structure ``` api/ # Python FastAPI backend ├── app/ │ ├── main.py # FastAPI lifespan; four background loops │ └── routes.py # REST endpoints: auth, trends, jobs, visualizations ├── db/ │ └── connection.py # Motor async MongoDB client, index init, dedup ├── models/ │ └─ …