Clustered Logistic Regression Algorithm for Flight Delay Prediction

Abstract—Cluster computing is a part High Performance Computing (HPC) which become more and more popular and necessary in the recent years. Meanwhile, data mining is a technology that have been growing where its huge benefits are inevitable. Once cluster computing and data mining are combined, it will yield a very powerful machine whereby the processing time in data mining can be accelerated by using the strength of cluster computing. The cost of developing cluster computing is also more efficient compared to buying a computer with very high specs such as server computer or even supercomputer. In this research, we present a simulation of cluster computing in virtual environment while implementing data mining algorithm to perform a prediction of flight delay. The aim of this research is to evaluate the performance improvement of logistic regression for doing the prediction in cluster environment. The result shows that cluster computing can significantly accelerate the computational speed of algorithm, compared to standalone mode. In this experiment, by using 1 master and 3 worker nodes (with identical hardware specifications), the computation time of logistic regression can be decreased up to 27.03%. Attaching more nodes to the cluster will lead to a better computation performance of cluster itself.
Keywords—Cluster computing, clustered logistic regression, logistic regression, flight delay prediction, pyspark, apache spark.