fgaim/HornVecs

Subword informed Word Embedding for Templatic Languages: Tigrinya, Amharic, Arabic ... ## Table of contents * Introduction * Resources * Models * Supplementary data * FAQ * Cheatsheet * Requirements * Building hornVecs * Getting the source code * Building hornVecs using cmake (preferred) * Building hornVecs using make * Building hornVecs for Python * Example use cases * Word representation learning * Obtaining word vectors for out-of-vocabulary words * Text classification * Full documentation * References * License ## Introduction HornVecs is a library for efficient learning of word representations and sentence classification of Semitic Languages, particulary Tigrinya and Amahric. It is built on top of the fantastic fastText by Facebook. The main contribution of HornVecs is that it considers the innate **non-concatenative morphology** of the Semitic languages when building the subwords for word embeddings. On the cases of Tigrinya and Amharic, we have observed fairly good performance gains when using HornVecs embeddings over fastText, word2vec, and GloVe. Even though HornVecs can be used for all languages that fastText supports, for languages other than the Semitic family we recommend you to use fastText. ## Resources ### Models - Recent state-of-the-art English word vectors. - Word vectors for 294 languages trained on Wikipedia. - Models for language identification and various supervised tasks. ### Supplementary data - The preprocessed YFCC100M data used in [2]. ### FAQ You can find answers to frequently asked questions on our website. ### Cheatsheet We also provide a cheatsheet full of useful one-liners. ## Requirements We are continuously building and testing our library, CLI and Python bindings under various docker images using circleci. Generally, **hornVecs** builds on modern Mac OS and Linux distributions. Since it uses some C++11 features, it requires a compiler with good C++11 support. These include : * (g++-4.7.2 or newer) or (clang-3.3 or newer) Compilation is carried out using a Makefile, so you will need to have a working …