BombayBrownBoy09/Nigeria-News-Detective

# Nigeria-News-Detective Streamlit Application Link **Final Project by Bhargav Shetgaonkar for Duke AIPI 540** ## 1. Problem statement The objective of this project is to train a deep learning model to classify Nigeria News articles on violent events based on sources and predict entities linked to ones mentioned in the text using NLP ## 2. Project Structure The project data and codes are arranged in the following manner: ``` ├── README.md the `data` folder can be downloaded from below link: 1) Download data here - **Training data:** Sourced from ACLED data on Nigeria from Jan 2019 to Apr 2022 - **Validation data:** you can set a 0.20 validation split while training 2) Download trained models here ```sh github.com ``` ## 3. Experimentation We have 2 DL models working in this application: **Model1 (Sentence Transformer Text Classification):** The best performing news source classification model used a pretrained sentence transformer model to generate embeddings and then used a logistic regression classification to obtain results (20 % train acc). The other approach tried was bag of words which resulted in lesser 11.2% train acc --> To get source of news, run the following from the home directory: ```sh python setup/text_classification.py ``` **Model2 (Linked Entity Prediction using Word2vec embeddings and LSTM neural net):** We first used NER using nltk and then got lists of entities for each event. We uniformly make list size to 3 and mask the 3rd entity to create a target variable Y. We then use word2vec to generate embeddings and generate prediction using a 4 layer sequential neural net here To get linked entities of news, run the following from the home directory: ```sh python setup/entity_prediction.py ``` To run the streamlit app ```sh python setup/app.py ``` This will prompt a user input in the command line for a news text and output the top 5 linked entities along with n …