weorold/multilingual-pos-evaluation

Quantifying systematic inequities in NLP infrastructure across languages. Compares POS tagging performance on high-resource (English, Spanish) vs. low-resource (Yoruba, Dominican Spanish) varieties using Universal Dependencies. Python | Stanza | Machine Learning # Multilingual POS Tagging Evaluation **Examining NLP Performance Disparities Across High-Resource and Low-Resource Languages** Author: Justin "Aurelio" Fernandez Sanchez Date: December 16 2025 Course: LNG 3430: Internet Linguistics, CUNY Lehman College --- ## Overview This project evaluates part-of-speech (POS) tagging accuracy across languages with different resource availability levels using Stanza, a neural NLP toolkit. The research reveals systematic performance disparities between high-resource languages (English, Standard Spanish) and low-resource varieties (Yoruba, Dominican Spanish). **Key Finding:** 78.2 percentage point accuracy gap between English (96.5%) and Yoruba (18.3%), with Dominican Spanish achieving only 73.4% compared to Standard Spanish's 92.9%. --- ## Research Questions 1. How do POS tagging models perform across languages with varying resource availability? 2. Do dialectal varieties experience degraded performance compared to standard varieties? 3. Are performance disparities due to data quantity or structural infrastructure bias? --- ## Methodology - **Languages Evaluated:** English, Standard Spanish, Yoruba, Dominican Spanish - **Tool:** Stanza v1.11.0 (neural POS tagger) - **Data:** Universal Dependencies v2.17 test sets - **Metrics:** Accuracy, macro F1, weighted F1, confusion matrices, per-POS accuracy - **Custom Corpus:** Programmatically generated 870-token Dominican Spanish evaluation set --- ## Key Results | Language | Accuracy | F1 (Weighted) | Tokens | |----------|----------|---------------|--------| | English | 96.5% | 0.965 | 25,094 | | Spanish (Standard) | 92.9% | 0.928 | 12,002 | | **Yoruba** | **18.3%** | **0.057** | 8,243 | | **Dominican Spanish** | **73.4%** | **0.752** | 870 | **Error Analysis:** Dominican Spanish errors concentrated on dialectal features: - Phonological contractions (pa', ta): High failure rate - Code-switches: Systematic misclassification - Regional vocabulary: Unrecognized as valid forms …