Syntactic Inductive Bias in Transformer Language Models: Especially Helpful for Low-Resource Languages?
Domain:
natural language processingRecord type:
paper
Creator:
GesSch
Host:
A line of work on Transformer-based language models such as BERT has attempted to use syntactic inductive bias to enhance the pretraining process, on the theory that building syntactic structure into the training process should reduce the amount of data needed for training. But such methods are often tested for high-resource languages such as English. In this work, we investigate whether these methods can compensate for data sparseness in low-resource languages, hypothesizing that they ought to be more effective for low-resource languages. We experiment with five low-resource languages: Uyghur, Wolof, Maltese, Coptic, and Ancient Greek. We find that these syntactic inductive bias methods produce uneven results in low-resource settings, and provide surprisingly little benefit in most cases. Accepted at CoNLL 2023
Similar
Empowering Sentiment Analysis in African Low-Resource Languages Through Transformer Models and Strategic Language SelectionProbing Gender Bias in Masked Language Models for Low-Web Data LanguagesNews Classification in Low‐Resource Languages: Insights From Transformer and Baseline ModelsLarge Language Models Adaptation for Low-resource Languages: The Case for African LanguagesEnhancing Sentiment Analysis in Amharic: Leveraging Transformer-Based Language Model for Low-Resource African LanguagesComparative Analysis of Deep Learning and Transformer Models for Low Resource Hausa Language Text Summarization