ismailSadouki/mini-infirence-engine

End-to-end LLM inference engine from scratch. Implements KV caching, prefill/decode execution, continuous batching, paged KV memory, block-table management, and inference benchmarking in pure PyTorch. Compares the resulting engine with vLLM and evaluates BF16, GPTQ, and AWQ serving on an Algerian Darija DPO model. # Mini Inference Engine A from-scratch LLM inference engine focused on the core mechanisms behind modern high-throughput serving systems. Implements prefill/decode execution, KV caching, continuous batching, paged KV memory, block-table management, and a reproducible inference benchmark harness in PyTorch. The project then compares the implementation against vLLM and evaluates BF16, GPTQ, and AWQ serving for an Algerian Darija DPO model. ## Why This Project? Most LLM projects focus on **training models** or using existing inference frameworks. This project focuses on a different question: > **What actually happens between an incoming generation request and the tokens produced by an LLM server?** The engine is built incrementally from first principles before studying the corresponding production mechanisms in vLLM. The project emphasizes **correctness, reproducibility, and measured performance**, rather than presenting unexplained throughput numbers. --- ## What It Implements ### Core Inference * Model adapter and generation interface * Explicit prefill/decode execution paths * Pre-allocated KV cache * Cached autoregressive generation * Deterministic sampling * Per-request generation state ### Scheduling * Request queue * Static batching baseline * FCFS continuous batching * Token-level request admission and eviction * Ragged active batches ### Memory Management * Contiguous KV-cache baseline * Fixed-size KV block pool * Free-list allocation * Logical-to-physical block tables * Paged KV attention * KV memory utilization and fragmentation analysis ### Benchmarking Measures: * Time To First Token (TTFT) * Inter-Token Latency (ITL) * Time Per Output Token (TPOT) * Throughput * p50 latency * p95 latency * Error and timeout rates * Latency-throughput curves Every reported benchmark includes a complete workload specification. --- # Architecture ```text +-----------------+ | Generation API | +--------+--------+ | v +-----------------+ | Request Manage …