Cost Reduction for Big Data Exploration and Pertinent Knowledge Extraction (Part 1)

It is particularly difficult to analyze digital observations due to the enormous amount of data per any given time period. The volume of data initially presents with the problem of interception followed by the issue of storage and analysis which all needs to fall within an appropriate infrastructure cost. The lack of pertinent information, extracted from big data, deprives both the evaluation of the use and the performance of the network and ultimately, the possibility of offering an effective service. This study looks to explore intercepted big data and how one may extract pertinent knowledge while minimizing the process cost. The process of analyzing big data is seen as an adaptive two-phase sampling design; the first-phase sample can be both stored within the available infrastructure and used as a sampling frame for each type of potential knowledge that can be extracted. Calibration of multiple sets of weights is completed to ensure the consistency of the estimates with constant quantities as well as between estimates using different or same sets of weights. We examine this proposed approach and the results of an illustrative example are presented.