Mulsemedia data consolidation method
DOI:
https://doi.org/10.34185/1562-9945-6-143-2022-06Keywords:
Software, multimedia, temporal multimodal data, data synchronization method, multithreading.Abstract
The synchronization of multimodal data is one of the essential tasks related to mulse-media data processing. The concept of mulsemedia (MULtiple SEnsorial MEDIA) involves the registration, storage, processing, transmission and reproduction by computer-based tools of multimodal information about a physical object that humans can perceive through their senses. Such information includes audiovisual information (object's appearance, acoustic properties, etc.), tactile information (surface texture, temperature), kinesthetic information (weight, object's centre of gravity), information about its taste, smell, etc. The perception of mulsemedia information by a person is the process that exists over time. Because of this, the registration of mulsemedia data should be carried out with the fixation of the moments of time when the relevant mulsemedia information existed or its perception made sense for a human who supervises the object as mulsemedia data is temporal. This paper presents a method that enables the consolidation and synchronization of mulsemedia data using the principles of multithreading. The universal method was designed to support combining data of different modalities in parallel threads. The application of the proposed method solves problems associated with integrating data of different modalities and formats in the same time interval. The effectiveness of applying this method increases by us-ing multithreaded distributed computing. This method is designed for use in the development of mulsemedia software systems. The modified JSON format (TJSON – Timeline JSON) was proposed in the paper, as well. TJSON-object is a complex data structure for representing the synchronized mulsemedia data and their further processing. The proposed method can be further extended with other approaches and technologies. For example, artificial intelligence methods can be applied to assess the correlation between data from different modalities. This can help improve the method's accuracy and the output files' quality.
References
A User Perspective of Olfaction-Enhanced Mulsemedia. Proceedings of the international conference on management ofemergent digital ecosystems / Ghinea G., Ademoye O. // MEDES’10. – 2010. – P. 277–280.
Semi-automatic synchronization of sensory effects in mulsemedia authoring tools / Abreu, R., Mattos, D., Santos, J. A. D., & Muchaluat-Saade, D. C. // In Proceedings of the 25th Brazillian Symposium on Multimedia and the Web. – 2019. – P. 201-208.
Beyond multimedia adaptation: Quality of experience-aware multi-sensorial media delivery / Yuan, Z., Ghinea, G., & Muntean, G. M. // IEEE Transactions on Multimedia. – 2014. – №17(1). – P. 104-117.
An evaluation of Heart Rate and ElectroDermal Activity as an objective QoE evaluation method for immersive virtual reality environments / Egan, D., Brennan, S., Barrett, J., Qiao, Y., Timmerer, C., & Murray, N. // In 2016 eighth international conference on quality of multimedia experience (QoMEX) IEEE. – 2016. – P. 1-6.
Towards a reference ontology on mulsemedia system / Saleme, E. B., Santos, C. A., Falbo, R. A., Ghinea, G., & Andres, F. // In Proceedings of the 10th International Conference on Management of Digital EcoSystems. – 2018. – P. 23-30.
Development of directional olfactory display / Kim, D. W., & Ando, H. // In Proceedings of the 9th ACM SIGGRAPH Conference on Virtual-Reality Continuum and its Applications in Industry. – 2010 – P. 143-144.
Temporal Multimodal Data-Processing Algorithms Based on Algebraic System of Aggregates / Pester, A., Sulema, Y., Dychka, I., & Sulema, O. // Algorithms. – 2023. – №16(4). – P 186.
Foundations of JSON schema / PEZOA, Felipe, et al. // In: Proceedings of the 25th international conference on World Wide Web. – 2016. – P. 263-273.
Multiple Sensorial Media Advances and Applications / GHINEA, George; ANDRES, Frederic; GULLIVER, Stephen. // New Developments in. – 2012.
Mulsemedia vs. Multimedia: State of the art and future trends / Y. Sulema // International Conference on Systems, Signals and Image Processing (IWSSIP), Bratislava, Slovakia. – 2016. – P. 1-5.
