Elisha-Seme/salama-lm

A 48M bilingual (English/Kiswahili) transformer trained from scratch on one laptop GPU, as a controlled testbed for cross-lingual transfer of safety training # salama-lm *Salama* is Kiswahili for "safe." **A 48M-parameter bilingual (English/Kiswahili) transformer trained from random initialization on one laptop GPU, built to answer a single question: when a model receives safety training in one language, what actually transfers to another?** Everything here was trained from scratch on a single RTX 4060 Laptop GPU (8 GB): the tokenizer, the 1.24B-token corpus, the base model, and twelve alignment conditions. Total cost was roughly 30 GPU-hours of consumer hardware. ## The finding in one table Refusal rate on **hazard topics the model never saw in training**, by language (means over 3 seeds): | Alignment condition | English | Kiswahili | Code-switched | |---|---|---|---| | none (base model) | 0.00 | 0.00 | 0.00 | | English-only, bare refusal | 0.01 | 0.00 | 0.00 | | English-only, refusal + reason | 0.00 | 0.00 | 0.00 | | Bilingual, bare refusal | 0.05 | 0.25 | 0.23 | | **Bilingual, refusal + reason** | **0.18** | **0.49** | **0.58** | English-only safety training produced pure string memorization: flawless on trained topics in English (1.00, including phrasings never seen in training), and nothing anywhere else. It answers Kiswahili requests in English with a memorized compliance template. Bilingual training carries trained topics across languages. Only reason-giving ("process-based") training generalizes to hazards the model has never encountered, and the two ingredients are jointly necessary: reasons without a second language achieve nothing. Two mechanistic results support this: * **Probes.** A linear "is this a hazard?" probe trained on English activations transfers to Kiswahili at 0.91 accuracy in the bilingual-process model, versus 0.64 for English-only training and 0.70 for the untrained base. Concept alignment tracks behavioral generalization. * **Steering.** Ablating the English-derived refusal direction (Arditi et al., 2024) removes English refusal in every model, replicating that mechanism at 48M para …