BrianMsane/PySpark-Practical

This practical is intended for IndabaX Eswatini 2025 and we're learning how to download, setup, install and run Apache Hadoop and Spark through the PySpark API. # PySpark Practical This practical is intended for IndabaX Eswatini 2️⃣0️⃣2️⃣5️⃣. It covers how to setup and run **Apache Server**, **Hadoop**, and **Spark** through the **PySpark API** for _Big Data Analytics and Processing_. _**Authors: Prof. Stephen Fashoto, Brian Msane, and Bandile Malaza**_\ _**Reviewer: Prof. Stephen Fashoto**_ ## Getting Started 💻 For getting started with this practical we need a few things which include: - WSL (Windows Subsystem for Linux) - Java 17 - Apache Server - Spark - Hadoop - PySpark - Jupyter Notebook Optional - Virtualbox - Ubuntu 24.04 LTS image If you are going to be using a virtual machine (VM), powered by Ubuntu operating system, you need to follow the VM guide. Once the VM is up an running we can then proceed with the steps below. ## Install WSL and Ubuntu - On Windows, open `PowerShell` as an **Administrator** and run this command. ```powershell wsl --install ``` - _Restart your PC if prompted_ and ensure **virtualization** is enabled in **BIOS**. _(Seek help here!)_ ## Create `hadoop` User Open the Ubuntu terminal and create a user called `hadoop`. You will be prompted to provide a _password_. Ensure to save this password as it is going to be needed later. ```bash sudo adduser hadoop ``` Give privileges to the user. ```bash sudo usermod -aG sudo hadoop ``` Switch `hadoop`. Once you have successfully created the user, you need to switch to that user using the command below. It will prompt you to provide the password you created above. ```bash su - hadoop ``` ## Clone Repo ✨ Clone this repository to ensure that you have the _shell scripts_ needed for the installation. Run the commands below. They will install git, configure your username and email and then clone this repository into your PC, in the `$HOME` directory. This will require that you provide your password for the `hadoop` user. ```bash sudo apt update && sudo apt install dos2unix && sudo apt install git -y ``` Replace **Username** and **Email …