Benchmarking Classical, Embedding, Transformer, and Large-Language-Model Approaches for Hate Speech Detection in Online Comments

First public release of hate-speech-detection — a reproducible head-to-head benchmark of three eras of NLP for hate-speech classification (TF-IDF + LR, Paragraph2Vec + LR, DistilBERT) on Davidson 2017 and HateXplain, with shared preprocessing, splits, seeds, and metrics.

We address the problem of hate speech detection in online user comments. Hate speech — abusive speech targeting specific group characteristics such as ethnicity, religion, or gender — is an important problem plaguing websites that allow users to leave feedback, with a negative impact on online business and overall user experience.

We benchmark four families of ap‐ proaches on this task:

(i) a TF-IDF + logistic regression baseline,

(ii) Paragraph2Vec (Doc2Vec) comment embeddings followed by a linear classifier, replicating Djuric et al. (2015),

(iii) fine- tuned DistilBERT, and

(iv) zero-shot and four-shot Gemini 2.5 Flash accessed via OpenRouter.

Evaluation is on two public datasets, Davidson et al. (2017) and HateXplain (Mathew et al. 2021), and covers a cross-dataset generalisation study, a threshold-tuning ablation, a target- group bias audit, an inference-cost benchmark, a zero-shot adversarial-obfuscation probe, a two-stage cascade architecture, and a large-language-model comparison. Our findings are: (i) DistilBERT is best in-domain on every cell but only after the decision threshold is calibrated on validation, a step routinely omitted by hate-speech leaderboards; (ii) the best model’s recall on the hate class varies by almost a factor of two across protected attributes, with race-targeted hate detected at recall 0.84 but gender-targeted hate at 0.47; (iii) subword tokenisation does not confer adversarial robustness — all three trained methods lose 11–16 AUC points under realistic character-level obfuscation, with DistilBERT the most affected on F1-hate; (iv) a TF-IDF- prefilter + DistilBERT-verifier cascade matches or beats DistilBERT-alone F1-hate at up to 9× the system throughput; and (v) instruction-tuned Gemini 2.5 Flash is competitive but does not beat fine-tuned DistilBERT on either corpus, with its

If you use this software or the accompanying paper, please cite it as below.