Y-Aladdin/llm-workshop-aims-2025

Materials and notes for the Science and Engineering of Large Language Models workshop at AIMS South Africa, Cape Town, 2025. # The Science and Engineering of Large Language Models (AIMS 2025) **A community-driven repository for the workshop held at the African Institute for Mathematical Sciences (AIMS) in Muizenberg, Cape Town, from March 31 to April 11, 2025.** This repository contains lecture slides, lab notebooks, additional resources, and community notes for the "Science and Engineering of Large Language Models" workshop. > The goal of the course -- apart from upskilling you to the forefront of LLM work -- is to allow different graduate students to get trained together, to work together, to socialize together, and to get to know each other across university boundaries. > > — *Ulrich Paquet, Director of AIMS* --- ## Table of Contents - Workshop Schedule - Assignments - Daily Materials & Lectures - Essential Resources - How to Contribute --- ## Workshop Schedule A high-level overview of the topics covered. For full details, see the **Official PDF Schedule**. --- ## Assignments ### Assignment 1 (Due Week 1) As explained by Amr Khalifa, you can choose one or both of the following tasks. A bonus is awarded for completing both. 1. **Theoretical Assignment:** - **assignment_1_theoretical.pdf** 2. **Practical Assignment:** Convert the GPT-2 style transformer from the lab into a Llama-like architecture. - **assignment_1_practical_GPT-to-Llama.ipynb** - **Key Changes Required:** 1. Implement **RMS Norm** instead of Layer Norm (both pre and post-normalization). 2. Implement the **Gated MLP (SiLU/SwiGLU)** layer instead of the standard GELU activation. - **Recommended Reading:** Llama 3 Architecture in JAX/Flax --- ## Daily Materials & Lectures ### Day 1: Monday, 31 March *Topics: A Brief History of LLMs, Tokenization* * **Lab:** Building a Transformer from Scratch (student version) - **Notebook** - ### Day 2: Tuesday, 1 April *Topics: The Transformer Architecture (Deep Dive)* * *Materials from this day are foundational for the practical part of Assignment 1.* …