GadAugust/FabricWild-From-Camera-Traps-to-ML-Models

An end-to-end data engineering and machine learning solution in Microsoft Fabric to analyze and classify wildlife imagery from camera traps in the African Savannah. # SavannahML: Wildlife Image Analysis with Microsoft Fabric An end-to-end data engineering and machine learning pipeline for analyzing large-scale wildlife image metadata from camera traps in the African Savannah, built on Microsoft Fabric. ## Project Goals - Ingest and organize JSON metadata into a Fabric Lakehouse - Process and analyze data using Apache Spark - Prepare datasets for machine learning - Train and evaluate classification models to identify wildlife patterns ## Technologies Used - Microsoft Fabric (Lakehouse, Data Factory, Spark, ML) - Apache Spark (PySpark) - Python (Pandas, Scikit-learn) - JSON, Parquet # Project Walkthrough: SavannahML with Microsoft Fabric This walkthrough documents each stage of the project, including screenshots and brief descriptions. --- ## 1. Loading Data into the Lakehouse ### Configure a Data Factory Pipeline to Copy Data Use Data Factory to set up a pipeline that ingests the raw JSON metadata. 📸 **Screenshot:** ![Data Factory Pipeline] --- ### Explore the Data in the Lakehouse After loading, verify and preview the data within the Lakehouse Files section. 📸 **Screenshot:** ![Lakehouse View] --- ### Convert the JSON Files into Parquet Files Use a Spark notebook to read the JSON and convert to Parquet for efficient querying. 📸 **Screenshot:** --- ### 🗃 Load the Parquet Files into Delta Tables Register the Parquet files as Delta tables to enable SQL exploration. 📸 **Screenshot:** --- --- ## 3. Semantic Model (Optional) Build a semantic layer if needed for downstream data consumption or ML feature clarity. --- ## 6. Data Analysis & Transformation with Apache Spark in Fabric Perform transformations such as flattening, feature extraction, and filtering. These steps are documented in the following notebook: github.com Data Analysis & Transformation with Apache Spark in Fabric Dow …