An approach to recognizing GUI elements as images based on machine learning

Abstract

Analysis of Recent Research and Publications. In recent years, computer vision tasks, in particular, object detection and classification in images, have undergone significant development thanks to the use of deep learning methods. The most common approaches are the YOLO, RetinaNet, EfficientDet, Cascade Mask R-CNN, and Swin Transformer models, which demonstrate high efficiency in object recognition tasks of varying complexity. At the same time, the issue of applying these models to recognize graphical user interface (GUI) elements in the context of automated testing of web applications remains insufficiently covered. Traditional approaches to UI testing are based on DOM structure analysis, which makes them vulnerable to changes in layout and complicates test support.

Research Objective. The purpose of this work is to analyze modern methods of recognizing objects in images and justify the choice of an effective machine learning model for the task of recognizing graphical user interface elements.

Presentation of the main research material. The paper proposes an approach to recognizing interface elements based on the analysis of web page screenshots as digital images. The proposed process involves obtaining a screenshot using the Selenium tool, saving the image in storage, and then transferring it to a machine learning model for processing. As a result, the model determines the classes of objects, their coordinates, and sizes, generating output data in JSON format. The obtained coordinates are used to interact with interface elements in an automated testing environment (PyTest).

A comparative analysis of modern object detection models is conducted, in particular Swin-L, EfficientDet-D7, Cascade Mask R-CNN, RetinaNet, and YOLO. Particular attention is paid to models of the YOLO family, which provide high image processing speed while maintaining a sufficient level of accuracy. In the context of the interface element recognition problem, this characteristic is important, since it allows integrating the model into the automated testing process without significantly increasing the test execution time. In addition, YOLO provides a convenient format of output data (bounding boxes), which is directly used for interaction with interface elements.

Conclusions. As a result of the conducted research, it was found that the use of computer vision methods is a promising direction for increasing the efficiency of automated UI testing. The proposed approach allows you to abandon the dependence on the DOM structure, which increases the resistance of tests to interface changes and reduces the costs of their support. Among the considered models, the most appropriate for practical application is YOLO, which provides the optimal balance between speed, accuracy and ease of integration. The results obtained can be used for the further development of intelligent web application testing systems.