haisonle001/LatVis

The codes and results of "LatVis: Large-scale Task-specific Language Model for Low-resource Vietnamese Multi-document Summarization" Bias-aware Curriculum Sampling for Fair Ranking" accepted at TALLIP # LatVis: Large-scale Task-specific Language Model for Low-resource Vietnamese Multi-document Summarization Accepted: ACM Transactions on Asian and Low-Resource Language Information Processing ### Introduction We introduce LatVis, a large-scale task-specific language model that specifically pre-trained for Vietnamese multi-document summarization (MDS) task and a Vietnamese multi-document labeled dataset with ∼10,000 samples. To the best of our knowledge, we introduce one of the very first public larger-size MDS dataset and public language model that designed for Vietnamese MDS task and proves to be a potential approach for natural language processing tasks in less-resourced languages. ### Model We use the official code for PRIMERA [[1]](#1) and use our pipeline to make it be able to work on Vietnamese text summarization datasets. The model can support up to 16K tokens. ### Dataset At the early phase of this project, due to the lack of Vietnamese unlabeled multi-document dataset, we divided the Newscorpus [[7]](#7) into smaller parts and translated Newshead [[8]](#8) using Google Translate API and ultilized it as our pretraining dataset. Below is the statistics of Vietnamese unlabeled dataset: | Dataset | Newscorpus | Newshead | | :----------- | :-----------: | :-----------: | | Total clusters | 5778893 | 314033 | | Total articles | 17847516 | 1065571 | | Average length per article | 154.03 | 2925.55 | | Average number of entities per article | 4.87 | 23.67 | Currently, only three Vietnamese MDS datasets are available, posing a significant challenge for research advancement in this domain. These MDS datasets such as VMDS [[2]](#2), ViMs [[3]](#3), and VLSP [[4]](#4) seem to be quite small. Therefore, to enrich the Vietnamese multi-document datasets resource, we translated WCEP [[6]](#6) - a dataset for MDS consists of short, human-written summaries about news events using gpt-3.5-turbo-16k model from OpenAI and spent hours manually curating the dataset. Bel …