jixtes/Summerrain-Amharic-news-summarizer

# Summerrain-Amharic-news-summarizer The summarizer is an unsupervised and extractive working on news articles written in Amharic. Although the methods tested in this research can be used for generic summarization, the considered training corpus and test cases are restricted to Amharic news documents. The extractive summarizer is implemented with the goal of selecting the most important sentences from an input document. In order to achieve this, all the sentences should be equally analyzed. The sentences should first be segmented. This can be achieved by detecting common sentence separators found in Amharic language. Each sentences are then analyzed on word level and ranked to calculate their relevance score. The relevance score is the most crucial measurement that determines whether a sentence is fit or not to be included in the final summary. The input text goes through four processes -sentence segmentation, tokenization, TF-IDF scoring, clustering and finally outputting the summary sentences in the order presented in the original text. Given an input the first thing the prototype does is do a sentence segmentation. Sentence segmentation is picking out each and every sentence. The segmentation is done by looking at the patterns behind common sentence separators in Amharic text. The following rules are used for the tokenization and sentence segmentation in a regular expression: 1. ፡፡| ::|።| ? and new line are taken as separators for sentences. 2. There is one tricky situation involving quotations; quotes should be part of a sentence even if they contain the stated delimiters. For example - የ ዲሞክራሲፈንድን በተመለከተአቶዘርዐይ፤ “መንግስትን ሲያብጠለጥሉለሚውሉ ሚዲያዎችእንዴትነ ውየ ዲሞክራሲፈንድየ ሚሰጠው?” ሲሉጠይቀዋል፡፡ this should be considered as one sentence without singling out the quoted text. 3. Any punctuation is removed from terms when counting occurrences or computing score except of compound words and abbreviations like ት/ቤት፣ ቤተ-መጽሃፍት which are identified as one word by looking at the - and / sep …