acoli-repo/acoli-glosser

Command-line tools for doing morphological segmentation and morphological glossing # ACoLi Glosser Command-line tools for doing morphological segmentation and morphological glossing. Developed within the project "Linked Open Dictionaries", funded by the BMBF, for applications within the humanities. Note that a requirement in the development was to provide a transparent, knowledge-based and flexible approach. We thus provide a dictionary-based (rather than neuronal) implementation. Beyond mere lookup, we provide an inference mechanism to annotate unseen words, based on frequency and overlap with reference annotations. This is less precise than manual annotation with IGT tools such as FLEx or Toolbox, but less work-intense. The glosser is particularly well-suited for historical languages with defective or unsystematic orthographies (where grammatically relevant phonemes cannot be reliably identified under all circumstances) as it does not require for formalize the mapping from orthography to morphological segmentation, but just exploits existing analyses. An example for such a defective orthography is Sumerian. Below a manual annotation where morphemes that have been inferred by the annotator but that are not directly recognizable from the surface string (marked in square brackets): 2(u) 2(u)[ten] NU _ _ _ sze sze[barley] N _ _ _ gur gur[unit] N _ _ _ lugal lugal[king][-ak][-ø] N.GEN.ABS _ _ _ For those cases, the most probable analysis will be returned (most frequent known analysis, maximum overlap [+frequency] for unknown analyses). Sample call: > cat MY-INPUT.conll | python3 glosser.py MY-DICT.tsv > MY-ANNOTATION.conll Input files must be provided in a one-word-per-line tab-separated format (TSV, i.e., CSV with TAB separators). The first column must contain the word. following columns are copied into the output without modifications. Lines starting with # are treated as comments and will be copied into the output, but not modified. The …