KolwaBrad/massivedataset

Training the MASSIVE dataset by Amazon(english-US, German-DE and Swahili-KE) # MASSIVEDATASET README ## Introduction This project focuses on processing and managing language data. It answers two main questions: 1. **Python3 Development Environment Setup:** Set up a Python3 development environment, install relevant dependencies, and build a project structure similar to PyCharm's. Import a massive dataset and generate language-specific Excel files (en-xx.xlsx) using specific fields (id, utt, and annot_utt). Recursive algorithms are not used to optimize performance. 2. **Working with Files:** Generate separate JSONL files for English (en), Swahili (sw), and German (de) datasets with test, train, and dev partitions. Create a single large JSON file showcasing translations from English to other languages (xx) for the training dataset. ## Prerequisites Before running the project, ensure you have the following prerequisites installed: - Python 3.x - pip (Python package manager) ## Installation You can install the required Python libraries/packages using the following command: ``` pip install jsonlines ``` ``` pip install json ``` ``` pip install os ``` ``` pip install pandas ``` ``` pip install sys ``` ## Project Structure The project structure should resemble the following: ``` project-root/ │ ├── 1.1/data/ │ └── excel │ ├── main.py ├── q2.py │ │ ├── en-xx.xlsx (Generated) ├── en_train.jsonl (Generated) ├── sw_train.jsonl (Generated) ├── de_train.jsonl (Generated) │ └── translations.json (Generated) │ ├── README.md ├── generator.sh └── other_files... ``` ## Running the Project Question 1 Place your dataset file (input_data.xlsx) inside the data/ directory. Run the following command to execute Question 1: ``` ./generate.sh ``` The script will generate language-specific Excel files (en-xx.xlsx) in the results/ directory. Question 2 Make sure you have the English (en), Swahili (sw), and German (de) JSONL files (e.g., en-US.jsonl, de-DE.jsonl, sw-KE.jsonl) in the project directory. Run the following command to execute …