gfkaceli/Automated-Interlinear-Glossing-using-Transformers

Automated Glossing For low resource languages # Automated Interlinear Glossing Pipeline This project implements an enhanced pipeline for automated interlinear glossing. The system uses a Transformer-based character encoder with relative positional encodings, an unsupervised morpheme segmentation module with adaptive thresholding and a full forward–backward algorithm (with utility masking), a translation encoder, and a Transformer decoder with cross-attention to generate gloss sequences from source words and their translations. ## Project Architecture - **Encoder.py** Implements a Transformer-based character encoder with relative positional encodings. **Input:** One-hot encoded source characters (shape: `(batch_size, seq_len, input_size)`) and sequence lengths. **Output:** Contextualized embeddings of shape `(batch_size, seq_len, embed_dim)`, which serve as input to the segmentation module. - **MorphemeSegmenter.py** Implements an unsupervised segmentation module that computes segmentation probabilities for each character, predicts an adaptive threshold from rich encoder statistics (max, mean, variance), and applies a forward–backward algorithm with utility masks (via `make_mask_2d` and `make_mask_3d`) to produce a binary segmentation mask and auxiliary outputs (predicted morpheme count and raw segmentation probabilities). **Input:** Encoder outputs and valid sequence lengths (and optionally target morpheme counts). **Output:** Binary segmentation mask, morpheme count, adaptive threshold, and segmentation probabilities. - **GlossingDecoder.py** Implements a Transformer decoder that generates gloss tokens. It uses cross-attention over a memory that is formed by concatenating aggregated morpheme representations (derived via an aggregation function from the segmentation mask) with a translation representation. **Input:** Target gloss token indices (for teacher forcing during training) and a memory tensor (aggregated segments + translation representation). **Output:** Logits over the gloss vocabulary for each t …