acoli-repo/hammy

HMM tagging for languages without training corpora # Hammy. HMM tagging for languages without training corpora Hammy has been created for didactic purposes and thus does not provide any optimizations as commonly found in real-world HMM implementations. If you nevertheless like to use it, please cite the following paper: Christian Chiarcos (2026), Towards the Morphological Annotation of North Markian (Low German), LREC-2026. The innovative idea is to use an unannotated corpus and to compare it with related languages to obtain transition probabilities - extrapolate emission probability from frequency - observe transition probabilities from related languages - optionally: refine emission probabilities with prefix- / suffix-matching - optionally: tag top k expressions manually We include data for an experiment on Middle High German, using ReM as gold standard and UD corpora as basis build data with $> make - `target/mhd/UD_split`: gold corpus, automatically mapped from original annotation, split: 80-10-10. note that we split by files, not by lines, so, the orthographies are different So far, we have a vanilla HMM implementation: training (over train split, UD version) $> python3 train.py rem_train.model target/mhd/UD_split/train.conllu tagging (and evaluation) $> python3 tag.py rem_train.model target/mhd/UD_split/test.conllu -e 3 To train and tag on full tags (incl. morphosyntactic features), specify the columns $> python3 train.py rem_train.xpos.model target/mhd/UD_split/train.conllu -c 4 $> python3 tag.py rem_train.xpos.model target/mhd/UD_split/test.conllu -e 4 Use `adopt.py` to port a trained model to another language: - input data should be pre-annotated against the same tagset, e.g., from a dictionary - input data can contain ambiguities, marked by |, then count both, weighted by current probabilities for UNKNOWN - input data can contain gaps - replace emission probabilities and word2freq - keep transition probabilities Use `merge.py` to merge two or more models: - average transition probabilities …