zhrgns/Pump-it-ML-Up-Data-Mining-the-Water-Table

This is a multi-class classification problem regarding water pumps in Tanzania. # Pump it Up: Data Mining the Water Table ## Problem Description This is a multi-class classification problem regarding water pumps in Tanzania, where the goal is to classify water pumps’ status either as: "functional", "needs repair", or "nonfunctional". The data is provided by the Tanzanian Ministry of Water. If a good statistical model is built, pump maintenance can be improved, which would lead to better access to water. ## Exploratory Data Analysis, Preprocessing We started by exploring the dataset and performing basic data cleaning and preprocessing steps. The training data set contains 59,400 observations with the labels and the test set contains 14,850 observations (without the labels). In training data set there were several number of numerical columns containing high number of zero values. So, we decided to handle zero values using categorical location-based data columns like “subvillage”, “lga” and ward etc. These location columns had high cardinality, therefore after imputation of numerical location-based data columns we dropped some of the categorical locations columns because of two main reasons; (1) categorical data is required to be encoded into multiple binary columns thus would hinder the performance of the model (2) having a numerical longitude/latitude data would yield the same result therefore, would cause redundancy. ### Categorical Features #### Longitude, Latitude, and GPS In longitude and gps_height columns there were high number 0 values. For Tanzania, these geographical values can not be 0. Whereas in latitude column, there were data points lying outside of Tanzania, on the ocean. Thus, zero and improper latitude values are replaced by NaN and then imputed by using mean value of each column for each unique grouping of 'basin', 'region', 'lga', 'ward', ‘subvillage’ variables. After value imputation, gps_height plot shows that in lower heights of Tanzania the ratio of non functional pumps are higher than higher ratios. The functi …