UKPLab/conll2020-multilingual-sentence-probing

Code and data for our CoNLL 2020 publication: "How to Probe Sentence Embeddings in Low-Resource Languages: On Structural Design Choices for Probing Task Evaluation" # How to Probe Sentence Embeddings in Low-Resource Languages: On Structural Design Choices for Probing Task Evaluation Code and data for our CoNLL 2020 publication: "How to Probe Sentence Embeddings in Low-Resource Languages: On Structural Design Choices for Probing Task Evaluation" ## Citation ``` @inproceedings{eger-etal-2020-probe, title = "How to Probe Sentence Embeddings in Low-Resource Languages: On Structural Design Choices for Probing Task Evaluation", author = "Eger, Steffen and Daxenberger, Johannes and Gurevych, Iryna", booktitle = "Proceedings of the 24th Conference on Computational Natural Language Learning", month = nov, year = "2020", address = "Online", publisher = "Association for Computational Linguistics", url = "aclweb.org", pages = "108--118", } ``` ## Structure Our implementation is based on Senteval to train and evaluate classifiers, using a given sentence embedding. The following features were added: * Change the size of a dataset, while maintaining its balance * Change the balance between classes in a given dataset * Use the Random Forest and Naive Bayes classifiers from scikit-learn * Automatically tune hyperparameters for MLP and Random Forest * Train various sentence embeddings We also added english datasets, as well as datasets in the languages: * Turkish (tr) * Russian (ru) * Georgian (ka) The following probing and downstream tasks were added to SentEval: | Task | Type | Description | Example | Command Line Argument | |------------------------|------------|---------------------------------------------|----------------------------------|------------------------| | Voice | Probing | Whether sent. contains a passive construct | He likes cats ⟶ False | Voice | | Subject Verb Agreement | Probing | Whether subject and verb agree | They works together ⟶ …