Curing Ultra Large Incomplete Data by Parallel Fractional Hot Deck Imputation

For reliable machine learning and statistical inference with large/big data, curing incomplete data is critical. Fractional hot deck imputation (FHDI) cures multivariate missing data by filling each missing unit with multiple observed values
(thus, hot-deck) without resorting to distributional assumptions. By inheriting the power of FHDI and parallel computing, we developed ultra data-oriented parallel FHDI (named as UP-FHDI) to cure ultra incomplete data with tremendous instances (big-n) and high dimensionality (big-p). We enabled scalable ultra incomplete data curing and also devised variance estimation via a parallel Jackknife method as well as efficient ultra data-oriented parallel linearization techniques. Results confirm that UP-FHDI can cure ultra datasets, up to millions of instances and 10, 000 variables. We validate the accuracy and scalability of UP-FHDI, paving a new pathway for reliable machine learning
and statistical inference with “cured” big data.