IndianaTones/dogon_consolidation

This repository contains code used to consolidate dogon datasets # First Release: Dataset merging and normalization phase This phase concerns data processing steps described in the blogpost entitiled "Building a Consolidated Dogon Lexical Dataset" by Hantgan-Sonko A., Kpoglu P.D., and Sagara I. ## Introduction This phase aims to merge four different datasets related to Dogon languages, standardizing their content, and combining them into a single comprehensive dataset. ## Process Overview The data merging process is structured into seven main phases: 1. **Data Standardization**: Raw datasets from the `data/` folder are standardized using a set of scripts that clean and normalize language and part-of-speech (POS) information. 2. **Intermediate Merges**: The cleaned datasets are then merged in two separate stages. 3. **Final Merge**: The results of the intermediate merges are combined into a final, unified dataset. 4. **Concept Standardization**: The final merged dataset is enriched with Concepticon glosses and IDs, and automatically generated POS. 5. **Human Concept/POS Revision Integration**: Human-corrected Concepticon glosses, IDs, and POS values are merged back into the Concepticon-enriched dataset. ## Folder Structure - `data/`: Contains the original, raw datasets (e.g., `fabian.tsv`, `abbie_modified_original.tsv`, `heath_base.tsv`, `reflex.tsv`) and the `language_names.tsv` file used for standardization. It also contains an adhoc file `reduced_concepts` which is a reduction of the concept-standardized output that was used for manual correction. - `intermediary_files/`: Stores the output of the standardization process (cleaned files like `cleaned_fabian.tsv`, `cleaned_abbie.tsv`, `cleaned_heath.tsv`, `cleaned_reflex.tsv`) and the results of the intermediate merging steps (e.g., `merged_fabian_abbie.tsv`, `merged_reflex_heath.tsv`). It also stores the `final_reduced.tsv` file which is just a stage reached in the manual corrections, but with additional columns added. - `final_results/`: Contains the final, fu …