A Semi-Supervised Hybrid Model for Anomaly Detection: Leveraging Autoencoder Feature Extraction and Random Forest Classification

Abstract. Anomalies, also known as outliers, are data points that deviate significantly from the norm. Detecting such anomalies is crucial in many real-world applications such as fraud detection, fault diagnosis, cybersecurity, and system health monitoring, where even a single undetected anomaly can lead to critical failures or losses. However, real-world datasets often suffer from extreme class imbalance and overlapping distributions, where anomalies are camouflaged within dense clusters of normal data. Popular anomaly detection algorithms like Isolation Forest, Local Outlier Factor (LOF), and clustering-based methods often struggle in these scenarios, leading to high false negatives. To address this, we propose a hybrid model combining an Autoencoder for reconstruction-based anomaly scoring with a Random Forest classifier trained specifically on ambiguous regions—where reconstruction errors for normal and anomalous instances overlap. This two-stage approach improves decision boundaries by leveraging both unsupervised reconstruction and supervised classification. Experiments on a labeled benchmark dataset demonstrate that our method significantly outperforms traditional models in terms of F1-score and detection accuracy, making it more effective for detecting subtle, disguised anomalies in imbalanced datasets.