Eyimofe-y/lzw-huffman-compression

Domain:

natural language processing

Record type:

software

Creator:

Eyi

Host:

Huffman and LZW Compression Visualiser on English and Nigerian Languages (Yoruba, Igbo, Hausa, Nigerian Pidgin) # Huffman and LZW Compression Visualiser on Nigerian Languages A visual and interactive notebook exploring how two classic compression algorithms perform across **English, Yoruba, Igbo, Hausa, Nigerian Pidgin and random text** and what that reveals about the structure of each language. --- ## What it does - **Huffman coding**: builds an optimal binary tree for each language based on character frequency. Frequent characters get shorter codes; rare ones get longer codes. - **LZW compression**: learns repeating patterns as it reads through a text. The faster its dictionary flattens, the more structure (and compressibility) the language has. - **Shannon entropy**: plots the theoretical minimum bits-per-character for each language, so you can see how close each algorithm gets to the ceiling. All four outputs are compared side-by-side in an interactive dashboard. --- ## Why Nigerian languages? English-trained compressors are everywhere. But Yoruba, Igbo and Hausa have different character frequency distributions (different vowel patterns, tonal markers, loanword structures) so a compressor optimised for English will underperform on them. This matters for Nigerian telecoms systems handling multilingual data at scale. The notebook makes that difference visible. --- ## Visualisations included | Visual | What it shows | |---|---| | Huffman tree (interactive) | The actual binary tree for any input text; switch between languages and watch the shape change | | LZW dictionary growth chart | How fast each languages patterns are learned. Random text never flattens, structured languages do | | Compression dashboard | Size saved (%), bits/char and entropy floor across all 6 texts | | Summary report | Printed metrics table: original size, Huffman savings, LZW savings, gap to Shannon floor | --- ### Option 1 — Google Colab (zero setup) Click the **Open in Colab** badge at the top of this README to run the simulation in your browser instantly. ### Option 2 — Local inst …

nacef112/LelapaAI-InkubaLM-Compression

Submission for Zindi's Lelapa AI Buzuzu-Mavi Challenge - Bronze Medalist. Focused model compression

raya-mez/Lost-in-compression

Effect of model compression (quantization) of LLMs on machine translation performance for high vs. l

Incorporating territory compression into population models

The ideal despotic distribution, whereby the lifetime reproductive success a territory's owner achie

Enhancing Cross-Lingual Transfer through Reversible Transliteration: A Huffman-Based Approach for Low-Resource Languages

As large language models (LLMs) are trained on increasingly diverse and extensive multilingual corpo

SOULx1s/Medical-LLM-Fine-Tuning-Model-Compression

A highly efficient LLM fine-tuning and deployment system using QLoRA and 4-bit quantization, optimiz

Intriguing Properties of Compression on Multilingual Models

Multilingual models are often particularly dependent on scaling to generalize to a growing number of

Eyimofe-y/lzw-huffman-compression

Domain:

Record type:

Visit

Languages