A method and software tool for video object recognition and analysis

Abstract

The modern scientific community is paying significant attention to the automation of video analytics. Traditional methods based on convolutional neural networks (CNNs), such as Faster R-CNN, demonstrate high accuracy but have limited processing speed for real-time streams. In contrast, single-stage algorithms in the YOLO family achieve the required performance. Research in object tracking (Tracking-by-Detection) highlights DeepSORT and ByteTrack as the most effective algorithms for associating detections across frames. The application of multimodal vision-language models (VLMs) opens up new possibilities for semantic scene description, although their implementation in high-load systems is hindered by computational complexity. Additionally, multidimensional data analysis (OLAP) technologies are considered, specifically the SurvCube and VideoCube models, which integrate video processing results into structured cubes; however, they often offer limited flexibility for creating new semantic hierarchies.

The objective of the research is to increase the analysis speed of ultra-large video datasets by developing a method and software tool for the automated extraction of structured data and its subsequent integration into a multidimensional OLAP model that supports parallel processing and semantic information enrichment.

A method for cascaded video data processing is proposed, combining rapid object detection with the YOLOv26m architecture and dynamic semantic scene enrichment using the Moondream2 multimodal model. During the preprocessing stage, metadata extraction (coordinates, time) and load optimization are performed by downsampling the frame rate to 2–5 frames per second. The YOLOv26m model is used for object detection and classification, achieving an mAP50 of 0.814 at 81.03 FPS. Tracking is implemented by assigning a unique object_id, which minimizes data redundancy. Semantic context enrichment (landscape type, events) is carried out by the multimodal Moondream2 model using text prompts, allowing the system to adapt to new scenarios without retraining the network. The data is integrated into a multidimensional OLAP model using a star schema, where the dimensions are time, space, object type, environment, and event type. The software features a microservices architecture utilizing a message broker (RabbitMQ/AMQP) for asynchronous communication between the detection and semantic analysis clusters.

In the course of this work, a scalable software complex was developed that successfully transforms unstructured video streams into the format of analytical cubes. It was experimentally confirmed that the use of the Moondream2 model provides the best balance between description quality (METEOR: 25.73) and processing speed (0.422 s/frame). Scalability testing demonstrated the architecture's capability for linear speedup: increasing the number of parallel workers to 10 reduced the total video dataset processing time by a factor of 8.95. The proposed solution is effective for high-load monitoring systems and rapid decision-making based on deep analytics.