A Comparative Study of Feature Attention Mechanisms in Text Classification

Abstract:
 Text classification is essential in natural language processing (NLP), involving categorizing text into predefined categories. Traditional machine learning algorithms, like Support Vector Machines (SVM) and Naive Bayes, require extensive feature engineering and struggle with high-dimensional data. Deep learning models, such as Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM) networks, have improved text classification by learning feature representations. However, these models face challenges with long-range dependencies and word importance.
Attention mechanisms, including additive, multiplicative, self-attention, and multi-head attention, address these limitations by allowing models to focus on relevant input parts. This paper compares these mechanisms' impact on text classification performance using multiple datasets and neural network architectures.
Results show that self-attention and multi-head attention outperform additive and multiplicative attention, effectively capturing global dependencies and enhancing feature representation. The study offers insights into each mechanism's strengths and weaknesses, providing guidelines for selecting suitable mechanisms for specific text classification tasks.