Yabsera-Haile/What-Gets-Left-Behind-Codebase

This is a repository that allows reproducibility for the paper "What Gets Left Behind: Signal-Dependent Distortion and Low-Resource Erosion in Instruction-Data Selection" # What Gets Left Behind — Reproducibility Code, controlled pool, and reference outputs for the audit of instruction-data selectors, the downstream analysis, and the rarity-aware correction described in the paper. The work has three stages, each of which maps to a group of scripts here: 1. **Audit** (§3–4): what each selector keeps, measured as per-group representation ratio. No model training — every quantity is a property of the selected subsets. 2. **Downstream** (§5): fine-tune on selected subsets and measure whether the distortion reaches the trained model, and whether standard benchmarks can see it. 3. **Correction** (§6): a selector-agnostic wrapper that repartitions the budget through quality-gated retention floors, and a map of when each floor is necessary. All model-based scoring uses fixed public models held constant across conditions. Fine-tuning uses LoRA for a fixed number of epochs so any downstream difference reflects the selected data, not the training budget. --- ## Repository layout ``` audit/ Python package (import as `audit.*`) common.py model defaults, chat templates, run metadata metadata/ §3.1–3.2 controlled pool construction + group labels build_metadata.py assemble metadata (resource tier + skill) for a pool inject_muri.py enrich the base sample with native low-resource examples langid.py GlotLID language identification pool_io.py canonical pool read/write export_pilot_jsonl.py export the labeled controlled pool selectors/ §3.3 the eight selectors (one representative per signal family) run_random.py group-neutral control / fair-share reference run_perplexity.py keep-high / keep-low / keep-middle (one scorer, three directions) run_ifd.py instruction-following difficulty run_semdedup.py semantic deduplication (reuses the RDS+ pool embeddings) run_quality.py LLM-judged quali …