lgessler/microbert

A tiny BERT for low-resource monolingual models ⚠️ **NOTE**: If you want to train a MicroBERT for your language, please see lgessler/microbert2. # Introduction MicroBERT is a BERT variant intended for training **monolingual** models for **low-resource** languages by **reducing model sizes** and using **multitask learning** on part of speech tagging and dependency parsing in addition to the usual masked language modeling. For more information, please see our paper. If you'd like to cite our work, please use the following citation: ``` @inproceedings{gessler-zeldes-2022-microbert, title = "{M}icro{BERT}: Effective Training of Low-resource Monolingual {BERT}s through Parameter Reduction and Multitask Learning", author = "Gessler, Luke and Zeldes, Amir", booktitle = "Proceedings of the The 2nd Workshop on Multi-lingual Representation Learning (MRL)", month = dec, year = "2022", address = "Abu Dhabi, United Arab Emirates (Hybrid)", publisher = "Association for Computational Linguistics", url = "aclanthology.org", pages = "86--99", } ``` # Pretrained Models The following pretrained models are available. Note that each model's suffix indicates the tasks that were used to pretrain it: masked language modeling (`m`), XPOS tagging (`x`), or dependency parsing (`p`). - `microbert-ancient-greek-m` - `microbert-ancient-greek-mx` - `microbert-ancient-greek-mxp` - `microbert-coptic-m` - `microbert-coptic-mx` - `microbert-coptic-mxp` - `microbert-indonesian-m` - `microbert-indonesian-mx` - `microbert-indonesian-mxp` - `microbert-maltese-m` - `microbert-maltese-mx` - `microbert-maltese-mxp` - `microbert-uyghur-m` - `microbert-uyghur-mx` - `microbert-uyghur-mxp` - `microbert-tamil-m` - `microbert-tamil-mx` - `microbert-tamil-mxp` - `microbert-wolof-m` - `microbert-wolof-mx` - `microbert-wolof-mxp` # Usage ## Setup 1. Ensure submodules are initialized: ``` git submodule update --init --recursive ``` 2. Create a new environment: ```bash conda create --name embur python=3.9 conda activate embur ``` …