fgaim/GeezSwitch-data

A dataset for language identification in five low-resource Ge'ez based languages # GeezSwitch Dataset A benchmark dataset for Language Identification (LI) between five low-resourced languages that use the Ge'ez script as a writing system: Amharic (አማርኛ), Blin (ብሊን), Ge'ez (ግእዝ), Tigre (ትግረ), and Tigrinya (ትግርኛ). The five languages are spoken by over 42 million people in total, in Ethiopia and Eritrea and the global diaspora of the regions. However, there is a lack of research on automatically distinguishing texts of these languages, and GeezSwitch is one step towards that. > The dataset is described in the paper: *"GeezSwitch: Language Identification in Typologically Related Low-resourced East African Languages"* published at LREC 2022. A Python package that includes the GeezSwitch dataset can be found here. It can identify between 60 languages with ~99% accuracy. ## Dataset __Dataset Overview__ The dataset has three parts [train, test, and development] that are located in the `data` directory, as follows: ``` data/train.tsv data/test.tsv data/dev.tsv ``` Each line in the files contains **three** tab [\t] separated values: | Column | Field | Description | |----------|-----------|----------------| | 1 | Sample ID | A sequential number of samples [1-n]. | | 2 | Language | Name of language, one of: amharic, blin, geez, tigre, tigrinya. | | 3 | Text | A sample text in the given language, which can be a sentence or phrase. | __Data Statistics__ | Language | Family | Branch | Speakers | ISO 639-3 | Samples | Tokens | Vocab. | Char. | |:----------:|:------:|:--------:|:--------:|:---------:|:-------:|:------:|:------:|:----:| | Amharic | Afro-Asiatic | Semitic | 32M | amh | 3k | 49.3k | 20.1k | 277 | | Blin | Afro-Asiatic | Cushitic | 120k | byn | 3k | 35.3k | 11.8k | 217 | | Ge'ez | Afro-Asiatic | Semitic | extinct* | gez | 3k | 48.6k | 14.3k | 219 | | Tigre | Afro-Asiatic | Semitic | 1M | tig | 3k | 53.8k | 15.4k | 208 | | Tigrinya | Afro-Asiatic | Semitic | 9M | tir | 3k | 49.0k | 16.2k | 238 | …