SportsOri

# Public Sports Controversy Analysis with LLMs ## Overview This repository contains code, datasets, and analysis for studying public opinion on controversial sports events using Large Language Models (LLMs). The project focuses on three major sports controversies: 1. **Maradona's "Hand of God"** (1986 FIFA World Cup) 2. **Frank Lampard's "Ghost Goal"** (2010 FIFA World Cup) 3. **Luis Suarez's Handball** (2010 FIFA World Cup) The project applies advanced NLP techniques to analyze public sentiment and stance detection on these controversial sporting events, fine-tuning LLMs for improved accuracy in detecting stance in social media comments. ## Datasets ### Frank Lampard Ghost Goal The Frank Lampard Ghost Goal dataset contains annotated social media comments about the controversial moment in the 2010 FIFA World Cup match between England and Germany. The incident occurred when Frank Lampard's shot crossed the goal line but was not awarded as a goal by the officials. Each comment is labeled with one of four stances: - **Favor**: Supporting that the goal should have been awarded - **Against**: Arguing against the importance of the disallowed goal - **Neutral**: Acknowledging the event without taking a stance - **Irrelevant**: Comments unrelated to the controversy Example usage: ```python import pandas as pd # Load the original dataset df = pd.read_excel('data/Dataset/Frank Lampard Ghost Goal Labels.xlsx') # Access comments and their labels comments = df['Comments'] labels = df['Label'] # Distribution of stances stance_distribution = df['Label'].value_counts() print(stance_distribution) ``` ### Maradona Hand of God This dataset contains annotated comments about Diego Maradona's infamous "Hand of God" goal during the 1986 FIFA World Cup quarterfinal between Argentina and England. ### Luis Suarez Handball This dataset contains annotated comments about Luis Suarez's handball incident in the 2010 FIFA World Cup quarterfinal between Ghana and Uruguay, where he deliberately blocked a goal-bound header with his hand. ## Fine-Tuning Scripts The repository includes Jupyter notebooks for fine-tuning Llama 3.1 8B models using the Unsloth library: - `Fine Tuning using Unlsoth script.ipynb`: General fine-tuning pipeline - `frank-lampard-reasoning-unslo.ipynb`: Fine-tuning specifically for the Frank Lampard controversy - `luis-saraez-fine-tune-unsloth.ipynb`: Fine-tuning specifically for the Luis Suarez controversy - `maradona-fine-tuning-unsloth.ipynb`: Fine-tuning specifically for the Maradona controversy ## Analysis Tools The `LLMsKnow` directory contains tools for analyzing LLM behavior and interpretability, based on the research paper "LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations." Key scripts include: - `analyze_frank_lampard.py`: Analyze the Frank Lampard dataset with balanced class handling - `probe_all_layers_and_tokens.py`: Probe model layers to understand internal representations - `probe.py`: General probing utilities for analyzing model behavior ## Prompts The `data/Prompts/prompts.json` file contains carefully designed prompts for stance detection on each controversy. These prompts include: - Background information about the controversy - Example comments with labels - Detailed definitions of stance categories ## Usage Examples ### Loading and Exploring a Dataset ```python import pandas as pd # Load the Frank Lampard dataset df = pd.read_excel('data/Dataset/Frank Lampard Ghost Goal Labels.xlsx') # Print basic statistics print(f"Total comments: {len(df)}") print(f"Stance distribution: {df['Label'].value_counts()}") # Sample comment print(f"Sample comment: {df['Comments'][0]}") print(f"Stance: {df['Label'][0]}") ``` ## Acknowledgements This project builds upon the research framework presented in "LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations" by Orgad et al. (2024) [arXiv:2410.02707](arxiv.org). The project utilizes code from the [LLMsKnow](github.com) repository for model probing and analysis tools, which has been adapted for sports controversy stance detection analysis.