shrutirij/soft-gazetteers

Code and data for the paper "Soft Gazetteers for Low-resource Named Entity Recognition" # Soft Gazetteers for Low-resource Named Entity Recognition This repository contains the code for experiments in the paper "Soft Gazetteers for Low-Resource Named Entity Recognition". Traditional named entity recognition models use gazetteers (lists of entities) as features to improve performance. Designing such features for low-resource languages is challenging, because exhaustive entity gazetteers do not exist in these languages. To address this problem, we propose a method of “soft gazetteers” that incorporates ubiquitously available information from English knowledge bases, such as Wikipedia, into neural named entity recognition models through cross-lingual entity linking. The paper contains experiments and analysis on four low-resource languages: Kinyarwanda (example sentence below), Oromo, Sinhala, and Tigrinya. ## Creating soft gazetteer features The following process expects input in the CoNLL format: each line contains a word and its label for the NER task (space-separated), with a empty line between sentences. An example in English: ``` Hello O , O Bob B-PER . O I O am O Alice B-PER . O ``` The example is in IOB2 format, but any other tagging scheme will work with the code. We retrieve candidates through an entity linking system and store the created features prior to training the NER model, which reduces the computational overhead at training time. Features for the train, development, and test sets are to be created in the same way. Install the required packages. ``` pip install -r requirements.txt ``` Consider a file, `train.conll`. 1. **Extract spans** up to length *n* from the conll file. ``` python data/get_ngrams.py --n 3 --filenames test.conll --output ngrams_list ``` 2. **Retrieve candidates** for all spans with an entity linking system. In the paper, we use three methods for candidate retrieval. The knowledge base (KB) we use is extracted from Wikipedia: `data/english_knowledge-base.zip`. - Exact match --- String lookup of the span …