Elisha-Seme/salama-lm
Domaine:
natural language processingType de record:
model
Créateur:
Eli
Hôte:
A 48M bilingual (English/Kiswahili) transformer trained from scratch on one laptop GPU, as a controlled testbed for cross-lingual transfer of safety training
# salama-lm
*Salama* is Kiswahili for "safe."
**A 48M-parameter bilingual (English/Kiswahili) transformer trained from random
initialization on one laptop GPU, built to answer a single question: when a
model receives safety training in one language, what actually transfers to
another?**
Everything here was trained from scratch on a single RTX 4060 Laptop GPU (8 GB):
the tokenizer, the 1.24B-token corpus, the base model, and twelve alignment
conditions. Total cost was roughly 30 GPU-hours of consumer hardware.
## The finding in one table
Refusal rate on **hazard topics the model never saw in training**, by language
(means over 3 seeds):
| Alignment condition | English | Kiswahili | Code-switched |
|---|---|---|---|
| none (base model) | 0.00 | 0.00 | 0.00 |
| English-only, bare refusal | 0.01 | 0.00 | 0.00 |
| English-only, refusal + reason | 0.00 | 0.00 | 0.00 |
| Bilingual, bare refusal | 0.05 | 0.25 | 0.23 |
| **Bilingual, refusal + reason** | **0.18** | **0.49** | **0.58** |
English-only safety training produced pure string memorization: flawless on
trained topics in English (1.00, including phrasings never seen in training),
and nothing anywhere else. It answers Kiswahili requests in English with a
memorized compliance template. Bilingual training carries trained topics across
languages. Only reason-giving ("process-based") training generalizes to hazards
the model has never encountered, and the two ingredients are jointly necessary:
reasons without a second language achieve nothing.
Two mechanistic results support this:
* **Probes.** A linear "is this a hazard?" probe trained on English activations
transfers to Kiswahili at 0.91 accuracy in the bilingual-process model, versus
0.64 for English-only training and 0.70 for the untrained base. Concept
alignment tracks behavioral generalization.
* **Steering.** Ablating the English-derived refusal direction (Arditi et al.,
2024) removes English refusal in every model, replicating that mechanism at
48M para …