MI-Dropout: A New Generalized Dropout Strategy Guided by Mutual Information

Dropout is a commonly used regularization method in deep neural networks, which can effectively alleviate overfitting. However, traditional Dropout usually uses a fixed probability to discard neurons, which fails to fully utilize the importance of the current input features. This study proposes a dynamic Dropout method based on the mutual information idea (Mutual Information-based Dropout, referred to as MI-Dropout), which estimates the information entropy of neuron activation values and dynamically adjusts the discard probability, thereby more selectively retaining information-rich features. Experimental results show that this method significantly improves the generalization ability of the model while maintaining its versatility. More importantly, MI-Dropout does not depend on a specific model structure and is applicable to a variety of neural network architectures, such as MLP, CNN, Transformer, etc.