WorldFishCenter/peskas.malawi.data.pipeline

# peskas.malawi.data.pipeline The goal of peskas.malawi.data.pipeline is to implement, deploy, and execute the data and modelling pipelines that underpin Peskas in Malawi. \## The pipeline is an R package peskas.malawi.data.pipeline is structured as an R package because it makes it easier to write production-grade software. Specifically, structuring the code as an R package allows us to: - better handle system and package dependencies, - forces us to split the code into functions, - makes it easier to document the code, and - makes it easier to test the code We make heavy use of tidyverse style conventions and the usethis package to automate tasks during project setup and deployment. For more information about the rationale of structuring the pipeline as a package check Chapter 3 in *Engineering Production-Grade Shiny Apps*. The book is focused on Shiny applications but the rationale also applies to data pipelines and production-ready code in general. ## How the pipeline works The pipeline is composed of different modules: 1. Data Collection: On site fishing landing surveys and continuous, solar-powered GPS vessel trackers to collect and send data in near real-time, alongside fishery metadata for a thorough data-gathering process. 2. Pre-processing: Data formatting, shaping, and standardisation to prepare the raw data for analysis. 3. Validation: Outlier detection and error identification, and includes an alert system to maintain data quality. 4. Analytics: Modelling fisheries indicators, nutritional characterization, and data mining to extract valuable insights. 5. Data export: Automated dissemination of processed and analysed fisheries data to ensure accessibility and comprehension. This involves restructuring data for dashboard integration and open publication. 6. isualisation: Tools for data reporting and sharing of insights through a comprehensive dedicated web app dashboard (not hosted in this repository). See Peskas: Automated analyt …