Design and Implementation of a Multi-Modal AI-Powered Music Sheet Generation System

This project proposes and documents the design and implementation of a multi-modal artificial intelligence system for automated music sheet generation. The system accepts four distinct input modalities, namely voice recording, audio file upload, video file upload, and natural language text-based song requests, and processes each through a dedicated AI pipeline to produce three simultaneous notation output formats: standard staff notation, chord charts, and guitar and instrument tablature. The core technical contribution of this work is a unified architecture that integrates automatic music transcription (AMT), deep learning-based pitch detection, large language model (LLM) assisted song identification, and multi-format notation rendering within a single accessible web application. Existing tools in the music transcription landscape handle only one input type and produce a single output format. This system addresses that gap by combining all four input modalities and all three output formats in one platform, a combination not currently available in any open or commercial tool. The voice and audio transcription pipelines are built on the CREPE pitch detection model and Spotify's Basic Pitch AMT model respectively. Video inputs are processed by extracting the audio track using FFmpeg before entering the transcription pipeline. The text input pipeline uses an LLM to parse song title requests, retrieves MIDI or audio data from available digital sources, and routes it through the transcription engine to generate notation. All pipelines converge at a MIDI intermediate representation, from which music21 generates the three notation formats for rendering via VexFlow in the browser. The system is designed for musicians, music students, educators, and producers, with particular relevance to self-taught musicians in Nigeria, the Philippines, and other markets where formal music education access is limited and professional transcription services are unaffordable. The project carries both academic and commercial intent, forming the research basis for an ND2 Computer Science project submission at the Federal School of Statistics, Ibadan, as well as the technical foundation for a commercial SaaS product. This preprint uploads the approved project proposal document as a public timestamp of the research concept and system architecture prior to full implementation.