Shadman19/llm-benchmark-low-resource-languages

Evaluating open-source LLMs on Bengali, Swahili and Tamil vs English baseline # 🌍 LLM Benchmark for Low-Resource Languages **Evaluating how well open-source LLMs understand Bengali, Swahili, and Tamil** --- ## 📌 What This Project Does Most AI benchmarks test English. This project asks: **how well do popular open-source LLMs actually perform in Bengali, Swahili, and Tamil?** We evaluate 4 models across 3 tasks: - **Translation Quality** — Does the model translate accurately? - **Reading Comprehension** — Can it answer questions about a passage? - **Commonsense Reasoning** — Does it reason correctly in these languages? ### Key Finding > GPT-class open-source models still show 25–40% performance degradation on Bengali and Tamil reasoning tasks compared to English. Swahili shows even steeper drop-off on numerical reasoning. --- ## 📁 Project Structure ``` llm-benchmark/ ├── data/ │ ├── raw/ # Original downloaded datasets │ └── processed/ # Cleaned, ready-to-use datasets ├── src/ │ ├── data/ # Scripts to download & prepare data │ ├── evaluation/ # Evaluation logic (scoring, metrics) │ ├── models/ # Model loading & inference │ └── utils/ # Helper functions ├── results/ # Output CSVs and charts ├── notebooks/ # Jupyter notebooks for analysis ├── docs/ # Paper-style writeup ├── requirements.txt ├── run_benchmark.py # Main script — run this! └── README.md ``` --- ## 🚀 How to Run This Project (Step by Step — No Experience Needed) ### Step 1: Install Python If you don't have Python installed: - Go to python.org - Download Python 3.10 or newer - Install it (check "Add to PATH" on Windows) ### Step 2: Download This Project Click the green **Code** button on GitHub → **Download ZIP** → Extract it somewhere on your computer. Or if you have Git: ```bash git clone github.com cd llm-benchmark ``` ### Step 3: Create a Virtual Environment Open your terminal (Command Pr …