Modeling and experimental evaluation of priority–based message processing in OLAP systems under peak loads
DOI:
https://doi.org/10.34185/1562-9945-5-166-2026-05Keywords:
OLAP system, analytical pipeline, micro–batching, prioritization, weighted tardiness, SLO, machine learning, modeling, experimental evaluationAbstract
A priority–based message–processing model for OLAP systems under peak loads is proposed. The method combines value, urgency, and processing cost and reduces losses caused by delayed data delivery.
The paper studies the experimental verification of a controlled service discipline for OLAP systems, where the value of information, its ageing, and the limited capacity of the analytical pipeline are critical. An analysis of recent studies shows that stream processing, classical scheduling, and analytical–warehouse internals are usually considered separately, without a unified control loop. The aim of the study is to develop and experimentally evaluate a controlled priority discipline that combines the business value of messages with the current technical state of the warehouse. A model of the OLAP pipeline is proposed that includes a broker, ETL, micro–batching, writes to the warehouse, materialized views, and background merge operations. To evaluate losses caused by untimely data delivery, a weighted tardiness criterion is used, while the selection of the next micro–batch is based on an integral scoring function that combines the expected impact on analytical decisions, urgency, and processing cost. The main experiment covers four uneven–load scenarios (Uniform, Short peak, Long peak, Mixed load) under the medium SLO profile and capacity = 4.0; additional robustness checks are performed for different capacity levels and SLO profiles. On average, the proposed method achieved the smallest aggregated loss criterion, J = 351.231 million conventional units, which is 2.5% better than static priorities, 22.2% better than EDF, and 26.5% better than FIFO. The method also provided 98.4% HP SLO fulfilment and 83.5% All SLO fulfilment. The results confirm the feasibility of cross–layer control of message processing in OLAP systems under peak loads and provide a reproducible basis for further validation on production traces.
References
1. Abadi, D. J., Ahmad, Y., Balazinska, M., Çetintemel, U., Cherniack, M., Hwang, J.–H., Lindner, W., Maskey, A. S., Rasin, A., Ryvkina, E., Tatbul, N., Xing, Y., & Zdonik, S. (2005). The design of the Borealis stream processing engine. In Proceedings of the Conference on Innovative Data Systems Research (CIDR 2005).
2. Abadi, D. J., Carney, D., Çetintemel, U., Cherniack, M., Convey, C., Lee, S., Stonebraker, M., Tatbul, N., & Zdonik, S. (2003). Aurora: A new model and architecture for data stream management. The VLDB Journal, 12(2), 120–139. https://doi.org/10.1007/s00778–003–0095–z
3. Apache Software Foundation. (n.d.). Apache Kafka documentation. https://kafka.apache.org/documentation/
4. Babcock, B., Babu, S., Datar, M., Motwani, R., & Widom, J. (2002). Models and issues in data stream systems. In Proceedings of the Twenty–First ACM SIGMOD–SIGACT–SIGART Symposium on Principles of Database Systems (pp. 1–16). https://doi.org/10.1145/543613.543615
5. Bennett, J. C. R., & Zhang, H. (1996). WF2Q: Worst–case fair weighted fair queueing. In Proceedings of IEEE INFOCOM 1996 (Vol. 1, pp. 120–128). https://doi.org/10.1109/INFCOM.1996.497885
6. Bernet, Y., Blake, S., Grossman, D., & Smith, A. (2002). An informal management model for Diffserv routers (RFC 3290). Internet Engineering Task Force. https://doi.org/10.17487/RFC3290
7. Burges, C. J. C. (2010). From RankNet to LambdaRank to LambdaMART: An overview (Microsoft Research Technical Report MSR–TR–2010–82). Microsoft Research.
8. Chaudhuri, S., & Dayal, U. (1997). An overview of data warehousing and OLAP technology. SIGMOD Record, 26(1), 65–74. https://doi.org/10.1145/248603.248616
9. ClickHouse, Inc. (n.d.–a). Materialized views. https://clickhouse.com/docs/materialized–views
10. ClickHouse, Inc. (n.d.–b). Part merges. https://clickhouse.com/docs/merges
11. Confluent, Inc. (n.d.). Monitor consumer lag. https://docs.confluent.io/platform/current/monitor/monitor–consumer–lag.html
12. Cormode, G., & Muthukrishnan, S. (2005). An improved data stream summary: The Count–Min Sketch and its applications. Journal of Algorithms, 55(1), 58–75. https://doi.org/10.1016/j.jalgor.2003.12.001
13. Demers, A., Keshav, S., & Shenker, S. (1989). Analysis and simulation of a fair queueing algorithm. SIGCOMM Computer Communication Review, 19(4), 1–12. https://doi.org/10.1145/75246.75248
14. Flajolet, P., Fusy, É., Gandouet, O., & Meunier, F. (2007). HyperLogLog: The analysis of a near–optimal cardinality estimation algorithm. In AofA 2007: Proceedings of the 2007 Conference on Analysis of Algorithms (pp. 127–146). https://doi.org/10.46298/dmtcs.3545
15. Gray, J., Chaudhuri, S., Bosworth, A., Layman, A., Reichart, D., Venkatrao, M., Pellow, F., & Pirahesh, H. (1997). Data cube: A relational aggregation operator generalizing group–by, cross–tab, and sub–totals. Data Mining and Knowledge Discovery, 1, 29–53. https://doi.org/10.1023/A:1009726021843
16. Harinarayan, V., Rajaraman, A., & Ullman, J. D. (1996). Implementing data cubes efficiently. In Proceedings of the 1996 ACM SIGMOD International Conference on Management of Data (pp. 205–216). https://doi.org/10.1145/235968.233333
17. Kaul, S., Yates, R., & Gruteser, M. (2012). Real–time status: How often should one update? In Proceedings IEEE INFOCOM (pp. 2731–2735). https://doi.org/10.1109/INFCOM.2012.6195689
18. Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., & Liu, T.–Y. (2017). LightGBM: A highly efficient gradient boosting decision tree. In Advances in Neural Information Processing Systems 30 (pp. 3146–3154). https://doi.org/10.5555/3294996.3295074
19. Liu, C. L., & Layland, J. W. (1973). Scheduling algorithms for multiprogramming in a hard–real–time environment. Journal of the ACM, 20(1), 46–61. https://doi.org/10.1145/321738.321743
20. Mao, H., Alizadeh, M., Menache, I., & Kandula, S. (2016). Resource management with deep reinforcement learning. In Proceedings of the 15th ACM Workshop on Hot Topics in Networks (pp. 50–56). https://doi.org/10.1145/3005745.3005750
21. Moroz, B. I. (1992). Metody i zasoby orhanizatsii protsesiv obrobky informatsii za kryteriiamy tsinnosti ta starinnia v systemakh avtomatyzovanoho upravlinnia ta informatsiinoho obsluhovuvannia [Methods and means of organizing information processing by the criteria of value and ageing in automated control and information service systems] (Doctoral dissertation). Dnipropetrovsk [in Ukrainian].
22. O’Neil, P., Cheng, E., Gawlick, D., & O’Neil, E. (1996). The log–structured merge–tree (LSM–tree). Acta Informatica, 33, 351–385. https://doi.org/10.1007/s002360050048
23. Pinedo, M. (2016). Scheduling: Theory, algorithms, and systems (5th ed.). Springer. https://doi.org/10.1007/978–1–4614–2361–4
24. Smith, W. E. (1956). Various optimizers for single–stage production. Naval Research Logistics Quarterly, 3(1–2), 59–66. https://doi.org/10.1002/nav.3800030106
25. Sviridov, V., & Moroz, B. (1992). Orhanizatsiia protsesiv obrobky informatsii za kryteriiamy tsinnosti ta starinnia v avtomatyzovanykh systemakh upravlinnia [Organization of information processing by the criteria of value and ageing in automated control systems]. Kharkiv: Osnova [in Ukrainian].
26. Tatbul, N. (2006). Load shedding techniques for data stream management systems (Doctoral dissertation, Brown University).
27. Zaharia, M., Das, T., Li, H., Hunter, T., Shenker, S., & Stoica, I. (2013). Discretized streams: Fault–tolerant streaming computation at scale. In Proceedings of the Twenty–Fourth ACM Symposium on Operating Systems Principles (pp. 423–438). https://doi.org/10.1145/2517349.2522737
28. Zhao, W., Rusu, F., Dong, B., & Wu, K. (2017). Incremental view maintenance over array data. In Proceedings of the 2017 ACM International Conference on Management of Data (pp. 139–154). https://doi.org/10.1145/3035918.3064041
Downloads
Published
Issue
Section
License
Copyright (c) 2026 System technologies
This work is licensed under a Creative Commons Attribution 4.0 International License.
