Wayy-Research/project-aya

Compression, Equity, and the Architecture of Linguistic Inclusion — Multilingual Mamba-MoE distillation from Cohere Tiny Aya # Project Aya **Compression, Equity, and the Architecture of Linguistic Inclusion** Cohere Labs x Wayy Research | Buffalo, NY | Est. 2026 | Open Science / Open Weights --- ## Overview Most efficient AI models are English-centric. The architecture of language technology is leaving billions behind. Project Aya investigates whether transformer-based multilingual models can be distilled into Mamba architectures while preserving multilingual capability, structured tool use, and cross-lingual reasoning — and whether compression degrades uniformly across language families. ### Core Research Question > *Can transformer-based multilingual models be distilled into Mamba architectures while preserving multilingual capability, structured tool use, and cross-lingual reasoning — and does the compression degrade uniformly across language families?* ### Teacher Model: tiny-aya-global CohereLabs/tiny-aya-global — 3.35B parameters, 70+ languages, 4-layer transformer (3 sliding window attention + 1 global attention), CC-BY-NC license. Loads without quantization on 4GB VRAM. ### Student Architecture: Aetheris (Hybrid Mamba-MoE) A novel student architecture combining selective state spaces with sparse mixture-of-experts, distilled from tiny-aya-global into Aetheris (~500-800M params, Mamba-MoE). | Component | Symbol | Description | |-----------|--------|-------------| | SSM Blocks | Psi | O(n) selective scan with constant memory. Mamba backbone enables linear-time sequence processing | | Sparse MoE | Sigma | 4 expert FFNs with top-1 routing and load balancing. Only one expert fires per token | | Hybrid Design | Delta | SSM blocks on even layers, MoE on odd layers. 24 total layers with weight-tied embeddings and gradient checkpointing | ### 3-Stage MambaInLlama Pipeline 1. **Layer Alignment** — Map transformer attention layers to Mamba SSM blocks with CKA-guided structural correspondence 2. **KL Distillation** — Soft-target training with KL divergence to transfer knowledge …