From Attention to Generative AI: A Decade of Architectural Innovations in Large Language Models

Over the past decade, artificial intelligence has undergone a remarkable transformation, particularly in natural language processing (NLP). The field has progressed from recurrent and convolutional models with limited sequence capacity to the attention based Transformer architecture that revolutionized scalability and context modeling. This breakthrough enabled the emergence of foundation models such as BERT, GPT, and T5, which redefined pretraining and transfer learning paradigms. Building on these foundations, the scaling laws formalized in 2020 demonstrated predictable performance gains with larger models, leading to GPT-3 and the era of few-shot and zero shot learning. The mainstream adoption of Generative AI (GenAI) followed in 2022 with ChatGPT and has since expanded to multimodal and instruction-tuned systems such as GPT-4, GPT-4V, Anthropic’s Claude family, and Google DeepMind’s Gemini series. In parallel, the open-source ecosystem has accelerated innovation through Meta’s LLaMA models, Falcon, Mistral, Mixtral, BLOOM, HuggingFace-led initiatives, Stability AI’s StableLM, and recent entrants such as DeepSeek. This survey provides a comprehensive comparative analysis of these developments across model architecture, parameter growth, context length, benchmarks, efficiency methods, and alignment strategies. More than fifty influential works are consolidated, with tables, charts, and metrics illustrating key milestones. I also discuss open challenges including energy efficiency, interpretability, bias, governance, and responsible open-source deployment. By consolidating both proprietary and community-driven contributions, this paper highlights the opportunities and risks that will shape the next generation of AI research and applications.