hailaykidu/llama2-amharic-english-mt

# LLAMA3_M — Vocabulary-Extended Llama-2-7B for Amharic → English Translation LoRA fine-tunes `meta-llama/Llama-2-7b-hf` for Amharic-to-English translation, after first extending its tokenizer with Ge'ez-script tokens so Amharic text doesn't get shredded into near-meaningless byte-level fragments by Llama-2's original (Latin-script-oriented) vocabulary. ## Pipeline Everything runs from a single script, finetune_model.py: ### 1. Vocabulary extension - New tokens are sourced from EXLMR's `vocab.json` (`/homes/neumann/teklehaymanot/EXLMR/vocab.json`), a candidate Tigrinya/Ge'ez subword list. Its keys are byte-level-BPE-remapped strings (RoBERTa/GPT-2 style), **not** literal Unicode text — `load_geez_vocab()` reverses that byte remapping first to recover the actual Ge'ez-script subwords, then drops anything that doesn't decode to valid UTF-8 or contains no Ethiopic-script characters, before calling `tokenizer.add_tokens()`. Feeding the raw byte-remapped keys straight into `add_tokens()` (as an earlier version of this vocab-reuse did) would register garbage strings that never occur in real text. - **29,819** tokens survive this filter and get added (vocab 32,000 → 61,819). - Each new embedding row is initialized as the average of a random-normal vector (scaled to the *pretrained* matrix's own std, not unit variance — Llama-2-7b's embedding norms are far smaller than a naive `torch.nn.init.normal_()` default, which would otherwise dominate the softmax and inflate the loss for every prediction) and the mean of all existing pretrained embeddings. ### 2. LoRA fine-tuning - `task_type="CAUSAL_LM"`, `r=16`, `alpha=32`, `dropout=0.05`, targeting `q_proj/k_proj/v_proj/o_proj/gate_proj/up_proj/down_proj`. - `modules_to_save=["embed_tokens", "lm_head"]` — the newly added Ge'ez-script embeddings were just randomly/mean-initialized and need to stay fully trainable; LoRA's frozen-base-weights default would otherwise leave them stuck at their initial values. - Data: Helsinki-NL …