nkakonas/prompt-based-interpretability-dialogue

Evaluating how prompt structure affects faithfulness and interpretability in LLM-based dialogue systems across medical, banking, and multi-domain settings under low-resource conditions. # Prompt-Based Interpretability in Dialogue Models > Evaluating how prompt structure affects faithfulness and interpretability in LLM-based dialogue systems across medical, banking, and multi-domain settings under low-resource conditions. --- ## Overview Large Language Models increasingly power dialogue systems in safety-critical domains like medicine and finance. But do structured prompts actually produce more faithful reasoning — or just more coherent-sounding responses? This project systematically benchmarks **5 prompting strategies** across **3 dialogue domains** using Mistral-7B-Instruct, evaluating both task quality (ROUGE-L, BERTScore) and interpretability (Factuality, Relevance, Fluency, Clarity) via a GPT-5.1 judge rubric. **Key finding:** Simpler prompts outperform complex reasoning chains on interpretability. More reasoning does not guarantee clearer or more faithful responses. --- ## Tech Stack | Component | Details | |-----------|---------| | Generator Model | Mistral-7B-Instruct-v0.2 (HuggingFace) | | Evaluation Judge | GPT-5.1 (rubric-based, deterministic) | | Automatic Metrics | ROUGE-L, BERTScore (BERT-F1), BLEU | | Statistical Test | Kruskal-Wallis H-test | | Language | Python | | Datasets | MedDialog, Retail Banking Chatbot, MultiDoGO | --- ## Prompt Families | Prompt Family | Strategy | |---------------|----------| | **Baseline** | Minimal instruction — reveals natural model behavior | | **Chain-of-Thought (CoT)** | Step-by-step reasoning | | **Self-Explain** | Requires brief justification of final answer | | **Domain-Aware** | Incorporates explicit domain guidelines | | **Critic** | Self-critique and revision before final output | --- ## Datasets | Dataset | Domain | Size | |---------|--------|------| | MedDialog (EN) | Medical doctor–patient dialogues | ~5,000 turns | | Retail Banking Chatbot | Customer service dialogues | ~5,000 turns | | MultiDoGO | Multi-domain task-oriented conversations | ~5,000 turns | All datasets subsamp …