Clustering-Based Filtering of Big Data to Improve Forecasting Effectiveness and Efficiency

New challenges arise with the upsurge of a Big Data era. Huge volumes of data, from the most varied natures, gathered from different sources, collected in different timings, often with high associated uncertainty, make the decision-making process a harsher task. Current methods are not ready to deal with characteristics of the new problems. This paper proposes a novel data selection methodology that filters big volumes of data, so that only the most correlated information is used in the decision-making process in each given context. The proposed methodology uses a clustering algorithm, which creates sub-groups of data according to their correlation. These groups are then used to feed a forecasting process that uses the relevant data for each situation, while discarding data that is not expected to contribute to improving the forecasting results. In this way, a faster, less computationally demanding, and effective forecasting is enabled. A case study is presented, considering the application of the proposed methodology to the filtering of electricity market data used by forecasting approaches. Results show that the data selection increases the forecasting effectiveness of forecasting methods, as well as the computational efficiency of the forecasts, by using less yet more adequate data.