cisnlp/ColexificationNet

Crosslingual Transfer Learning for Low-Resource Languages Based on Multilingual Colexification Graphs # ColexificationNet We use Conceptualizer to extract the colexification patterns directly from a parallel corpora. This repositories contain a more effiecient version of **Conceptualizer** (forward pass + backward pass) introduced in Conceptualizer paper. ``` . ├── README.md ├── association_finder.py ├── eva │   ├── baseline_vectors │   │   ├── other_word_vectors_process.ipynb │   │   ├── sentence_classification.ipynb │   │   ├── sentence_retrieval.ipynb │   │   └── train_sentence_ID_script.py │   ├── colexification_patterns │   │   ├── clics_neighbors_dict.pickle │   │   ├── conceptualizer_100_neighbors_dict.pickle │   │   ├── conceptualizer_10_neighbors_dict.pickle │   │   ├── conceptualizer_1_neighbors_dict.pickle │   │   ├── conceptualizer_20_neighbors_dict.pickle │   │   ├── conceptualizer_50_neighbors_dict.pickle │   │   ├── conceptualizer_5_neighbors_dict.pickle │   │   └── eva_colexification.ipynb │   ├── round_trip │   │   ├── round_trip.py │   │   └── round_trip_min_langs.py │   ├── sentence_classification │   │   ├── sentence_classification.ipynb │   │   └── sentence_classification.py │   └── sentence_retrieval │   ├── sentence_retrieval.ipynb │   ├── sentence_retrieval.py │   └── test_ids.txt ├── network_related │   ├── __init__.py │   ├── eflomal_network_builder.py │   ├── eflomal_training.py │   ├── iso2area.pickle │   ├── iso2family.pickle │   ├── network_builder.py │   ├── train_different_min_language_embedding.py │   └── updated │   ├── NetworkAnalysis-Basic.ipynb │   └── NetworkAnalysis-LanguageFamilies-Areas.ipynb ├── processing_concepts.py ├── processing_concepts_eng.py ├── processing_ngrams.py └── processing_parallel.py ``` ## Colexification extraction pipeline (1) Preprocess the parallel data to obtain all ngrams for each verse in all languages: ``` python -u processing_ngrams.py --updated_ngrams true --ignore_case true ``` (2) Create parallel data that are required in the subsequent computation: ``` python -u process …