Refusal Is Language-Dependent: A Cross-Lingual (English–Hindi) Refusal Benchmark and Evaluation of Llama-3.1-8B

Safety alignment in large language models is trained and evaluated predominantly in English, yet these models are deployed globally. This record accompanies a study asking a direct question: does a model's tendency to refuse a harmful request depend on the language in which the request is phrased?

To answer it with minimal confounding, we release a parallel English–Hindi red-teaming benchmark of 415 matched request pairs (830 prompts) across seven harm categories, where each unsafe request appears once in English and once as a faithful Hindi translation of the same request. This paired design isolates the effect of language, holding content, intent, and phrasing fixed.

Evaluating Llama-3.1-8B-Instruct over three independent runs, we find the model refuses 87.8% of unsafe requests in English but only 59.8% in Hindi, a stable ~28-percentage-point safety gap present in every run and every harm category. On the matched pairs, the model refuses in English yet complies in Hindi 111 times against only 5 in the reverse direction (exact McNemar's test, p < 10^-26). Automatic judge labels were validated by manual human review in both languages. The finding localises, for an open model and a high-population language, the multilingual safety gap previously reported for closed models, and indicates that safety evaluation should not stop at English.

Contents: the preprint (PDF), the benchmark prompts (parallel English–Hindi, seven categories), the generation, cleaning, and evaluation code, and aggregate results and figures.

Intended use and ethics: this is defensive AI-safety research. The benchmark contains harmful prompts whose sole purpose is to measure and improve model safety, especially closing cross-lingual guardrail gaps in lower-resource languages. It must not be used to elicit or act on harmful content. Harmful model completions are deliberately withheld from this release.

Limitations: results are established for a single model (Llama-3.1-8B) and one language pair; labels rely on an LLM judge with human review. See the paper for full detail.