BosunL/SEA-VQA

We evaluate six VQA models on a locally curated dataset of Kenyan cultural images and questions in Swahili, English, and Ateso, testing how well current VLMs transfer across languages with different levels of digital representation while reasoning about culturally specific visual content. # SEA-VQA: Visual Question Answering for Kenyan Cultures A localized benchmarking framework designed to evaluate and audit modern Vision-Language Models (VLMs) on culturally grounded visual content across **Swahili, English, and Ateso**. This project is a joint research initiative between **Princeton University** and **Maseno University**, developed by Ashley Meier, Olatunbosun Lapite, and Amakalu Vitalis. --- ## About the Project While state-of-the-art Vision-Language Models (VLMs) demonstrate remarkable capabilities in Visual Question Answering (VQA) for widely spoken Western and Asian languages, their ability to reason about localized cultural context—particularly in under-represented African languages—remains drastically underexplored. Web-scale pretraining datasets frequently omit regional cultural knowledge and native African languages, leaving a massive digital gap in AI technologies. **SEA-VQA** addresses this disparity by auditing six prominent open-source VLMs: * BLIP-VQA-Base * BLIP-VQA-CapFilt-Large * Gemma-3-4B-IT * Pangea-7B * Phi-3-Vision-128k * Aya-Vision-8B To achieve this, our team traveled across Kenya for six weeks to curate a specialized dataset of over 1,000 culturally relevant images. From this dataset, we built a multilingual, multiple-choice evaluation pipeline featuring fine-grained captions and complex distractor logic to evaluate cross-lingual transfer and localized visual reasoning. --- ## Project Objectives & Expectations When launching this project, our primary focus was to quantitatively expose the boundary lines of modern multimodal models. We were hoping to extract several key outcomes from this research: * **Audit Cultural Competency:** We aimed to evaluate whether modern VLMs can identify and reason about fine-grained cultural nuances (e.g., traditional attire like *owalo* sisal skirts, regional tools, or local landscapes) when paired with image descriptions. * **Assess Multilingual Transferability:** We sought to measu …