Dialect in the machine: auditing sentiment analysis models for African American Vernacular English bias

Sentiment analysis systems are deployed at scale in content moderation, social listening, and public health, yet their behaviour on African American Vernacular English (AAVE) is rarely evaluated in a systematic way. We present a reproducible audit framework, implemented as an open Google Colab notebook, that applies three widely-used transformer models (BERT-base, RoBERTa-base, and DistilBERT) to a curated parallel corpus of 1,200 sentence pairs. Each AAVE sentence is matched to a semantically equivalent Standard American English (SAE) sentence. Across all three models, mean predicted negative-sentiment probability is 14.3 percentage points higher for AAVE text than for matched SAE text (p < 0.001, Wilcoxon signed-rank). The disparity is largest for positive-polarity sentences (+18.7 pp), which suggests that AAVE affirmative and emphatic constructions are being read as negative by the models. We further show that this gap is not explained by lexical overlap with hate-speech training data; it is instead driven by morphosyntactic AAVE features, chief among them copula deletion, negative concord, and habitual 'be'. We propose three bias-quantification metrics, the Dialect Sentiment Gap (DSG), the Polarity Flip Rate (PFR), and the Feature-Conditioned Disparity (FCD), and release all code, data, and model outputs publicly.