mtanti/gabra-converter

A program for converting data from a Ġabra database dump to a more regular and accessible format. # Ġabra Converter This program converts Ġabra's database dump files, which are for MongoDB, into a more accessible format, as well as cleaning and normalising it. ## How to use To use this program, you will need to have the following command line commands available on your computer: - `tar`: 7-zip archiver - `bsondump`: MongoDB tool Make sure that you install the above applications and then test them in your command line with the following commands: - `tar --version` - `bsondump --version` Once you have these applications available in your command line, you can now download a Ġabra database dump file. Use the exporter by calling `python bin/run_gabra_converter.py` or `gabra_converter.exe` in the command line as follows: `python bin/run_gabra_converter.py --gabra_dump_path --out_path --lexeme_cleaners --wordform_cleaners --lexeme_exporter --wordform_exporter ` Here is a typical example: `python bin/run_gabra_converter.py --gabra_dump_path path/to/gabra --out_path path/to/out --lexeme_cleaners --wordform_cleaners --lexeme_exporter csv --wordform_exporter csv` or with the `gabra_converter.exe`: `gabra_converter --gabra_dump_path path/to/gabra --out_path path/to/out --lexeme_cleaners new_lines --wordform_cleaners --lexeme_exporter csv --wordform_exporter csv` Run `python bin/run_gabra_converter.py --help` or `gabra_converter --help` for more information. ## What is exported All the exported data is based on the official Ġabra schema. Whilst MongoDB is a NoSQL database which allows for leaving fields out completely in database rows (note that rows are called documents and tables are called collections in MongoDB), the exported data is structured as flat tables. All the fields in the schema are used in the export and left empty if unused in a row. On the other hand, any fields that are not mentioned in the schema but still used in the rows, such as `norm_freq`, are left out. A number of files are generated to handle one-to-many relationships. F …