Anomaly Detection in Log Records

In recent times complex software systems are continuously generating application and server logs for the events which had occurred in the past. These generated logs can be utilized for anomaly and intrusion detection. These log files can be used for detecting certain types of abnormalities or exceptions such as spikes in HTTP requests, number of exceptions raised in logs, etc. These types of events recorded in the log files are generally used for anomaly prediction and analysis in future. The proposed prototype for anomaly detection assumes that the log records are uploaded as input using a standard apache log format. Next, a prototype is developed to get the number of HTTP requests for outlier detection. Then anomalies in number of HTTP requests are detected using three techniques namely InterQuartileRange method, Moving averages and Median Absolute deviation. Once the outliers are detected, these outliers are removed from the current dataset. This output is given as input to the Multilayer Perceptron model to predict the number of HTTP requests at the next timestamp. This paper presents a web based model to automate the process of anomaly detection in log files.