Sinhala Similarity and Semantic Plagiarism Detection Using Modern Embedding Models

Abstract While plagiarism detection has developed to some degree in English and in some high-resource languages, it has not yet been fully developed in Sinhala language. Plagiarism detection is now well developed in English, and in some high resource languages, but there is little practical support for Sinhala language. This situation is worse in learning environments with presence of manual or tool-based checking of Sinhala submissions that doesn't identify semantic rewriting and translated copying. Current Sinhala research demonstrates success in direct matching and sentence similarity measures, yet these are applied on a limited scale between small corpora and/or rely on very basic similarity measurements or on a sentence level model that cannot be scaled to a full detection pipeline. In this paper, a research design that will tackle 2 related tasks in a single system for plagiarism detection system is presented for Sinhala plagiarism. The first one searches for similarity plagiarism within the Sinhala language, and is able to highlight copied and paraphrased content on a sentence level. It is written in English and then translated into Sinhala and is used without a reference. Semantic plagiarism is where English source material is translated and then used without a source. The proposed solution includes the use of a corpus to build the system, annotating the corpus, generating contextual embeddings, using a Siamese neural network, creating bilingual sentence representations, conducting a focused web crawl, and deploying the system as a service. A common framework enables the ingestion of documents, splitting them into sentences, comparing them, ranking them by ‘thresholder' and generating the report. The paper also dissects the data plan, workflow for implementation, progression of the prototypes and the evaluation plan for both components. The research integrates same language detection and cross language detection technologies within a single framework to improve the natural language processing for Sinhala language and at the same time simultaneously support academia in the low resource scenario by using it as a tool on academic integrity.