Implications of Deep Compression with Complex Neural Networks

Abstract: Deep learning and neural networks have become increasingly popular in the area of artificial intelligence. These models have the capability to solve complex problems, such as image recognition or language processing. However, the memory utilization and power consumption of these networks can be very large for many applications. This has led to research into techniques to compress the size of these models while retaining accuracy and performance. One of the compression techniques is the deep compression three-stage pipeline, including pruning, trained quantization, and Huffman coding. In this paper, we apply the principles of deep compression to multiple complex networks in order to compare the effectiveness of deep compression in terms of compression ratio and the quality of the compressed network. While the deep compression pipeline is effectively working for CNN and RNN models to reduce the network size with small performance degradation, it is not properly working for more complicated networks such as GAN. In our GAN experiments, performance degradation is too much from the compression. For complex neural networks, careful analysis should be done for discovering which parameters allow a GAN to be compressed without loss in output quality.