CNNGen: A Generator and a Dataset for Energy-Aware Neural Architecture Search

Neural Architecture Search (NAS) techniques look for optimal architectures in a predefined architecture space. One forms this space by deriving variants from a known architecture in which we insert cells. Then, a systematic combination of network functions and connections within these cells results in benchmarks. Cell-based methods yielded hundreds of thousands of trained architectures whose specifications and performance are available to design performance prediction models. Yet, existing NAS approaches and datasets typically focus on performance and ignore the growing environmental and resource impact of training and deploying these models. We cannot ignore these impacts anymore since deep learning models are huge (trillions of parameters for GPT-4) and require hardware inaccessible to most. We contribute to energy-aware NAS with i) a grammar-based Convolutional Neural Network (CNN) generator not requiring a predefined architecture. CNNGen yields diversity for multi-objective search where ideal base architectures are unknown; ii) A dataset of 1,300 architectures generated by CNNGen with their full description and implementation, performance and resource consumption metrics and carbon impact; iii) Three novel performance and energy prediction models. Our experiments demonstrate the diversity of generated architectures and that our predictors outperform the state-of-the-art for performance prediction and accurately predict energy consumption.