harsh-mali/llm-monitor

# LLM Monitoring Pipeline > A production-style MLOps observability platform for monitoring LLM performance — with multi-mode querying, multi-dimensional quality scoring, and a live analytics dashboard. --- ## What is this? Most developers call an LLM API and move on — they have zero visibility into whether the model is fast, accurate, or consistent. This project builds the **observability layer** that production AI systems need. Every prompt is automatically: - Routed to **Groq's LLaMA 3.1 8B** for inference - Scored across **three quality dimensions** (length, coherence, relevance) - Evaluated by an **LLM judge** when quality drops below threshold - Logged asynchronously to **Supabase PostgreSQL** without blocking the response - Reflected on a **live dashboard** with metric cards, trend charts, and filterable logs --- ## Three Operating Modes | Mode | What it does | Key feature | |------|-------------|-------------| | **Single** | Send one prompt, get scored response | Full quality breakdown with 4 score bars | | **Batch** | Paste up to 10 prompts, run all at once | `asyncio.gather()` concurrent execution | | **Simulate** | Fire N concurrent users at the same prompt | Latency variance + per-user quality scoring | --- ## Architecture ``` THREE ENTRY POINTS: POST /query → single prompt → background logging (non-blocking) POST /batch → N prompts → asyncio.gather() concurrent execution POST /simulate → 1 prompt × N → concurrent user load simulation FLOW (all three modes): Input prompt(s) │ ▼ ThreadPoolExecutor → call_groq_async() [Groq API — LLaMA 3.1 8B] │ ▼ compute_quality() ├── score_length() [tiered: 0.2 / 0.5 / 0.7 / 0.9 / 1.0] ├── score_coherence() [sentence count + structure bonus] ├── score_relevance() [keyword overlap: prompt ↔ response] └── llm_judge() [only when combined score < 0.7] │ ▼ BackgroundTasks.add_task() → Supabase insert [non-blocking] │ ▼ Return response + quality scores instantly DASHBOARD: G …