ntakokevin/Word-Count-and-Phrase-Analysis-of-Swahili-Language-Using-Spark-RDD

# Word-Count-and-Phrase-Analysis-of-Swahili-Language-Using-Spark-RDD In this project, our objective is to develop a word count function that performs phrase analysis using Spark RDD (Resilient Distributed Datasets). The main focus lies within the function "run_phrase_count_on_rdd()" where several transformations are applied to the RDD to generate phrase counts. The key steps involved in this process include filtering the RDD to skip header and bad rows, utilizing map operations to split each line into columns and convert text into n-grams using NLTK functions, employing flatMap to flatten the RDD for subsequent reduction, and finally, choosing between reduce or reduceByKey operations. Detailed information on these steps can be found in the Spark RDD API documentation and the "introducing-spark-APIs" notebook. Before executing the function, it is recommended to test the code for computing phrase counts independently. It is advisable to compute one transformation at a time and verify its correctness before chaining multiple operations together. Additionally, ensure that the function "tokenize_into_ngrams_phrase(text, n=3)" is functioning correctly by testing it before using it in the map operation. To explore the data and perform testing, loading the data into a pandas DataFrame is encouraged. The expected number of columns in the output CSV file is 3, namely phrase, count, and phrase_en, which will be checked during the grading process. ## Converting Texts into N-grams For generating n-grams effortlessly, we recommend using the NLTK module and specifically the "ngrams" function. Additionally, the "sent_tokenize" function from the "nltk.tokenize" module will be useful. Exploring the NLTK documentation will provide a better understanding of how to use these functions effectively. ## Cleaning up Phrases During tokenization, nonsensical and noisy phrases such as 0000 or blank strings may be generated. It is crucial to remove these from the results. You have the fl …