Data distillation in machine learning: mathematical model and optimization methods

Abstract

The article explores the concept of data distillation in machine learning, an approach aimed at creating compact yet efficient datasets without significant performance loss. The increasing volume of data plays a crucial role in modern deep learning, but its processing requires substantial computational resources. Data distillation seeks to reduce dataset size by selecting the most informative samples, optimizing the training process, reducing redundant information, and improving model generalization. The proposed mathematical model formalizes data distillation as an optimization problem that involves selecting a subset that minimizes information loss. Various evaluation criteria are applied, including the gradient-based approach, which analyzes the impact of individual samples on model training through changes in the loss function gradient; the entropy-based approach, which measures model uncertainty concerning specific samples; and the representative subset method, which minimizes the distance between the original and distilled datasets. The study examines key distillation methods, such as generative models (GANs, diffusion models), active learning (data selection based on entropy levels), and clustering methods (K-means, DBSCAN) for determining representative samples. Experimental analysis demonstrates that using a distilled dataset can reduce data volume by a factor of ten while decreasing model accuracy by only about 2%. Additionally, training time is reduced by a factor of eight, significantly improving computational efficiency. The research results confirm the effectiveness of data distillation in machine learning, as it enables a balance between performance and computational resources. However, the authors highlight certain challenges, including the selection of an optimal distillation strategy and the potential loss of critical information when an inappropriate subset is chosen. Thus, data distillation represents a promising research direction that facilitates the development of more efficient and resource-saving models, optimizing the machine learning process. This approach opens new possibilities for using deep neural networks in various practical applications, particularly in resource-constrained learning environments. Moreover, the integration of data distillation techniques with modern deep learning architectures could further enhance their impact by improving transfer learning capabilities, enabling faster convergence, and reducing dependency on large-scale labeled datasets.