d-gurgurov/Adversarial-Training-LowRes-Languages

A repo for adversarial training overview in context of Low-Resource Languages # Adversarial Training for Low-Resource Languages (LRLs) ## Literature Review ### Adversarial Training for LRLs (Lukas Hauzenberger) Fine-tuning a model while controlling for a specific attribute can significantly enhance its robustness. The approach involves: 1. **Adversarial Head with Gradient Reversal Layer (GRL):** - Predicts the languages to _hypothetically_ minimize the influence of a higher-resource language on the low-resource language results and to extract language-agnostic features during classification head training. - Model is discouraged to correctly predict language by the gradients flowing back from an adversarial head being reversed. - Adjusts the magnitude of the gradients using an alpha parameter. - Fine-tunes the model, controlling for specific attributes and increasing its robustness against adversarial examples. 2. **Adversarial Head Goal:** - Achieve 50% accuracy to ensure no extra information is contained in the model. ### Unsupervised Language Adaptation (Rocha and Cardoso, 2019) Adversarial training in a cross-lingual setting aims to make the neural network agnostic to the input language while addressing specific tasks. Key components include: 1. **Components:** - Feature Extractor - multilingual backbone model such as mBERT or XLM-R - Task Classifier - Language Discriminator 2. **Training with GRL:** - GRL is used as presented in Unsupervised Domain Adaptation by Backpropagation by Ganin and Lempitsky (2015). - The goal is to minimize both the task classifier and adversarial component losses. - However, training a neural network with GRL is very unstable. 3. **Alternative Approach:** - Chen et al. (2018) in Adversarial Deep Averaging Networks for Cross-Lingual Sentiment Classification propose minimizing the Wasserstein distance between the distribution of joint hidden features for source and target instances (bilingual examples). - Alpha regularizes the adversarial component, increasing its importance over time. 4. **Shared …