asahala/pmi-embeddings

State-of-the-art count-based word embeddings for low-resource languages. # pmi-embeddings State-of-the-art count-based word vectors for low-resource languages. 1. ```src/make_embeddings.py``` creates PMI-based word vectors from the source corpus. 2. ```src/explore_embeddings.py``` make simple searches from embeddings. Requires a vector file and a dictionary (included in ```corpora/akkadian.zip```) 3. ```src/hypertune.py``` tests hyperparameters (by using brute force) to find the best settings for the given data set (requires a gold standard). 4. ```corpora/extract_corpus.py``` a script for extracting sense-disambiguated corpora from Korp-Oracc VRT files. 5. ```corpora/akkadian.zip``` a zipped test corpora and a dictionary of Akkadian language (use these to generate new embeddings and to explore them). 6. ```eval/gold.tsv``` an initial version of the Akkadian gold standard. ### What are word embeddings and why are they useful? Word embeddings represent words as real-valued vectors in a multi-dimensional vector space. As the vectors encode words' contextual similarity, they can be used to extract words that show higher interchangeability with each other. Thus, in addition to analogy and similarity tasks, word embeddings can be exploited in almost any NLP application, including sentiment analysis, spam detection and automatic chat moderation, document classification, machine translation etc. This repository contains basic tools for lexicographic analysis, namely exploring vocabularies of historical languages in their own terms. ## Jupyter tutorials For those who like to use Jupyter Notebooks, ```src/jupyter_embeddings.ipynb``` instructs how to build your own word embeddings with just a few lines of code. ```scr/jypyter_explore_embeddings.ipynb``` guides how to make queries from embeddings. For setting up Jupyter environment, please read this guide by Niek Veldhuis. Note that you only need the packages listed below (of which most are likely preinstalled in Conda/Jypyter). (NOTE! AFTER THE RECENT GENSIM UPDATE EXPLORE_EMBEDDINGS DOESN'T …