µGPT: A Minimal Transformer Language Model

Large Language Models (LLMs) have transformed modern Artificial Intelligence due to their remarkable capacity to comprehend and produce natural language. Yet, the majority of existing systems rely on extensive frameworks, significant GPU usage, and high level libraries that obscure the mathematical and algorithmic foundations of the Transformer model architecture. This poses a considerable challenge for anyone seeking to grasp the inner workings of GPTs without a strong foundational un derstanding of the subject. Large Language Models (LLMs) have revolutionized contemporary Artificial Intelligence by showcasing impressive abilities in understanding and generating natural language. However, most current implementations depend heavily on large scale frameworks, GPU-intensive training processes, and highly abstracted libraries that conceal the underlying mathe matical and algorithmic principles of Transformer architectures. This creates a substantial obstacle for students, researchers, and independent developers who are trying to understand the internal workings of Generative Pretrained Transformers (GPTs) from basic principles. This paper presents µGPT, a minimal Trans former based language model developed entirely from scratch using pure Python and NumPy without relying on deep learning frameworks such as PyTorch or TensorFlow. The project re constructs the fundamental elements of GPT style architectures, which include token embeddings, positional embeddings, scaled dot-product self-attention, residual connections, RMS normal ization, multilayer perceptrons, autoregressive next-token predic tion, custom automatic differentiation, gradient backpropagation, Adam optimization, gradient clipping, temperature sampling, top-k sampling, and nucleus sampling. The system is developed through several progressively enhanced versions, starting from a dependency-free autograd-based prototype to a refined NumPy based Transformer training pipeline. This proposed architecture illustrates how contemporary language modeling principles can be replicated using compact and interpretable implementations while preserving conceptual fidelity to large-scale Transformer systems. Experimental evaluation is conducted on a dataset comprising over 32,000 names and additional textual corpora, where the model effectively learns character-level and token-level sequence generation patterns. The study also examines training stability, optimization strategies, inference quality, computational efficiency, and architectural trade-offs in low-resource CPU only environments. Unlike production oriented LLM frameworks that prioritize scalability over interpretability, µGPT emphasizes transparency, educational accessibility, and mathematical clarity. The project functions as a minimal GPT implementation focused on research, as well as a teaching framework designed to help understand the intricate workings of Transformer language models in detail.

Impact Statement: The main effect of this work is to democra tize the comprehension of Transformer architectures by offering a completely transparent, lightweight, and framework-agnostic GPT implementation. µGPT empowers students, educators, and researchers to explore the entire lifecycle of language model de velopment from tokenization and self-attention to optimization and autoregressive generation without the need for specialized hardware or large-scale industrial infrastructure. This initiative promotes explainable and accessible AI education while fostering reproducible research in streamlined language modeling systems.