WorldFishCenter/peskas.zanzibar.data.pipeline

Peskas data pipeline for Zanzibar # peskas.zanzibar.data.pipeline The goal of peskas.zanzibar.data.pipeline is to implement, deploy, and execute the data and modelling pipelines that underpin Zanzibar, the small-scale fisheries analytics in Zanzibar. ## The pipeline is an R package peskas.zanzibar.data.pipeline is structured as an R package because it makes it easier to write production-grade software. Specifically, structuring the code as an R package allows us to: - better handle system and package dependencies, - forces us to split the code into functions, - makes it easier to document the code, and - makes it easier to test the code We make heavy use of tidyverse style conventions and the usethis package to automate tasks during project setup and deployment. For more information about the rationale of structuring the pipeline as a package check Chapter 3 in *Engineering Production-Grade Shiny Apps*. The book is focused on Shiny applications but the rationale also applies to data pipelines and production-ready code in general. The best place to learn more about package development is probably the *R packages* book by Hadley Wickham and Jenny Brian. ## The pipeline runs on Github Actions While each step in the pipeline are defined as a function in the package, these functions are deployed and integrated using GitHub Actions. This allow us to take advantages of best practices in continous development and integration (CD/CI) and automatically link the code to execution. However, these workflow functions work almost as scripts because they don’t take parameters and are used for their side effects. Each job in the pipeline is defined in the workflow file: `.github/workflows/data-pipeline.yaml` and can be seen in the figure below. Note that additional workflows exist to test the package in multiple environments and build the documentation website. The figure above illustrate the jobs that are part of the pipeline workflow. Note that not all of them are implemented yet. Genera …