ktrw1011/sentiment-analysis-for-tunisian-arabizi

# 2nd place Solution Summary AI4D iCompass Social Media Sentiment Analysis for Tunisian Arabizi ## Model and Setting All models are trained in StratifiedKfold at 5fold. The pre-trained model is trained with the following settings - maxlen 198 - Domain-Adaptation - train+test MLM - Label-smoothing - dynamic padding - Padding during training can speed up the training. - In addition, we sorted by token length. This makes it even faster. - There was almost no degression in sorting. - Colab P100 or V100 ### Arabic-Bert (aubmindlab/bert-base-arabertv02) cv: 0.809 - I used the translate script that was shared in the discussion - zindi.africa - The accuracy of the translation is not very good (my teammate confirmed this). But we can get the same or slightly better results than Vanilla-BERT ### Vanilla-BERT (bert-base-uncased) cv: 0.805 - I tried the multilingual model, but the accuracy was almost the same. Therefore, we chose the lighter model. - distillation model was a little underperforming. We cannot use more than two pre-trained models (1GB is the limit). However, we knew that the ensemble would score better and be more robust, so we used a traditional model that could be trained with only the given data set and was fast. ### LSTM cv: 0.799 - sentencepiece (vocab 8000) - Subword Regularization - gensim word2vec pretrain (train+test) (dim=300) ### Fasttext (lightgbm) cv: 0.802 - fastext pretrain (train+test) (dim=300) - stopword remove (en+fr) - lightgbm train ### Catboost cv: 0.789 - text_features=["text"] - no preprocessing ### MultinominalNB cv: 0.795 - tfidf vectorizer - no preprocessing ## Ensemble - Stacking the 1st model with oof - 0.5 * Logistic Regression + 0.5 * lightgbm ## Didn't work - French-BERT - I heard that Arabizi uses more French than English, but it was worse than Vanilla-BERT. - Vocabulary Augmentation - Pseudo Labeling …