victoriapedlar/isizulu-text-generation

Open-Ended Text Generation in isiZulu: Decoding Strategies for a Morphologically Rich Low-Resource Language # isizulu-text-generation # AWD-LSTM + This code was originally forked from the PyTorch word level language modeling example and is heavily inspired by the original AWD-LSTM implementation LSTM and QRNN Language Model Toolkit + The code in this notebook is available on google colab and on github. This version of AWD-LSTM was created by Gustav Madslund and Mikkel Møller Brusen. ### Core components 1. **[x] - Multi Layer** - We will need to controll what happens in between the layers, therefore, instead of using the multi layer cuDNN lstm implementation, we will create multiple single layer cuDNN lstms. 2. **[x] - Weight drop** using DropConnect on hidden-hidden weights $[U^i, U^f, U^o, U^c]$ before forward and backward pass - makes it possible to use cuDNN LSTM 3. **[x] - Optimization** using SGD and ASGD while training ### Extended regularization techniques 4. **[ ] - Variable sequence length** to allow all elements in the dataset to experience a full BPTT window - **[ ] - Rescale learning rate** to counter the varible sequence lengths favoring short sequences with fixed learning rate 5. **[x] - Variational dropout AKA LockDrop** for everything else than hidden-hidden, such that we use same dropout mask for all input/output in a forward backward pass of LSTM 6. **[x] - Embedding dropout** which is **not** just a dropout applied on the embedding 7. **[x] - Weight tying** to reduce parameters and prevent model from having to learn one-to-one correspondance between input and output 8. **[x] - Embed size** independent from hidden size, to reduce parameters. 9. **[ ] - AR and TAR** - $L_2$-regularization by applying AR and TAR loss on the final RNN layer - can screw stuff up ### Getting started Ensure that all scripts are run from the root directory. Install requirements: `pip3 install -r awd_lstm_requirements.txt` Fetch training data: `./utils/getdata.sh` Minimum arguments to run the model: ``` python3 awd_lstm/main.py \ --data data/nchlt/ …