Algorithms for data imputation based on entropy

Abstract

Recent advancements in data imputation have focused on various machine learning techniques, including methods like mean, median, and mode imputation, along with more complex approaches like k-nearest neighbors (KNN) and multiple imputation by chained equations (MICE). Research into entropy-based methods offers a promising direction. This method minimizes uncertainty by selecting imputation values that reduce the overall entropy of the dataset. The goal of this work is to develop an algorithm that imputes missing data by minimiz-ing conditional entropy, thus ensuring that the missing values are filled in a way that pre-serves the relationships between the variables. The method is designed for both qualitative and quantitative data, including discrete and continuous variables, aiming to reduce uncer-tainty in classification tasks and enhance the performance of machine learning models. The proposed algorithm is based on conditional entropy minimization, using entropy as a measure of uncertainty in data. For each incomplete row, the algorithm computes the con-ditional entropy for possible imputation values. The value that minimizes conditional entropy is selected, as it reduces uncertainty in the target variable. This process is iterated for each missing value until all missing data is imputed. Three types of tests were performed on two datasets. The analysis showed that the pro-posed algorithms are quite slow compared to other methods and can be improved, for exam-ple, by multiprocessing, as described in our work [15]. The type 1 test showed that the pro-posed algorithms do not give a gain on the RMS deviation metric, but significantly reduce en-tropy (type 2 test). At the same time, these methods show an improvement in classification performance over the baseline models (type 3 test). Thus, the proposed entropy-based imputation methods have shown good results and can be considered by researchers as an additional tool to improve the accuracy of decision mak-ing, but further computational optimisation studies are needed to improve the performance of these methods. The algorithm shows promise in improving classification accuracy by selecting imputa-tion values that minimize conditional entropy. Future research will focus on optimizing the method for large datasets and expanding its application to various domains.