JOINT LLM RECONCILIATION OF MULTIPLE ASR TRANSCRIPTS FOR SCRIPT–COMPLIANCE CUE DETECTION IN CONTACT CENTERS
DOI:
https://doi.org/10.34185/1562-9945-5-166-2026-08Keywords:
automatic speech recognition, ASR system combination, contact–center quality assurance, script compliance detection, large language models, LLM–as–judge, hypothesis–level fusion, word error rate, evaluation protocolAbstract
Contact–center quality assurance audits script compliance – whether the agent greeted the caller, identified the company, asked for the caller’s name, and closed politely – using automatic speech recognition (ASR) transcripts. The cues that carry these checks are short formulaic phrases, and they are exactly what an imperfect recognizer drops or garbles, so the operative metric is recall of the cue rather than word error rate. Classic ASR system combination votes at the token layer and can delete a word only if one engine heard it, which is precisely the word a compliance check needs. This paper therefore asks where the ensemble should live for a detection task and compares four fusion points under a single LLM compliance judge: single–ASR, token–level, decision–level, and joint–hypothesis–level reconciliation. On a disagreement–enriched stratum of a 3057–call contact–center corpus, joint reconciliation is the best arm (macro–F1 0.913), significantly above single–ASR, decision–level OR and ROVER for both judges, and the ordering survives re–scoring on a stratum containing only human–set labels. Three controls show that the gain requires a nonidentical, call–matched second transcript, and attribute it partly to text visible only in that transcript.
References
1. Fiscus, J. G. (1997). A post–processing system to yield reduced word error rates: Recog-nizer output voting error reduction (ROVER). Proc. IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), 347–354. https://doi.org/10.1109/ASRU.1997.659110
2. Galimzianov, D., & Vyshegorodtsev, V. (2024). Text–based detection of on–hold scripts in contact center calls. Machine Learning and Applications: An International Journal, 11(2), 1–9. https://doi.org/10.5121/mlaij.2024.11201
3. Hillard, D., Hoffmeister, B., Ostendorf, M., Schlüter, R., & Ney, H. (2007). iROVER: Im-proving system combination with classification. Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volume, Short Papers, 65–68. https://doi.org/10.3115/1614108.1614125
4. Ingle, D., Kumar, A., Gogineni, K., & Vepa, J. (2022). Real–time monitoring of silences in contact center conversations. Interspeech 2022, 1951–1952. https://www.isca–archive.org/interspeech_2022/ingle22_interspeech.pdf
5. Jia, F., Majumdar, S., & Ginsburg, B. (2021). MarbleNet: Deep 1D time–channel separable convolutional neural network for voice activity detection. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 6818–6822. https://doi.org/10.1109/icassp39728.2021.9414470
6. Jitsi. (n.d.). JiWER: Evaluate your speech–to–text system with similarity measures. https://github.com/jitsi/jiwer.
7. Jung, J., Brahman, F., & Choi, Y. (2025). Trust or escalate: LLM judges with provable guarantees for human agreement. International Conference on Learning Representations (ICLR). https://doi.org/10.48550/arXiv.2407.18370
8. Kamo, N., Ochiai, T., Delcroix, M., & Nakatani, T. (2025). MOVER: Combining multiple meeting recognition systems. Interspeech 2025, 3424–3428. https://doi.org/10.21437/interspeech.2025–1614
9. Kano, T., Ogawa, A., Delcroix, M., & Watanabe, S. (2021). Attention–based multi–hypothesis fusion for speech summarization. IEEE Automatic Speech Recognition and Under-standing Workshop (ASRU), 487–494. https://doi.org/10.1109/asru51503.2021.9687977
10. Kohli, G. S. (2026). Nine judges, two effective votes: Correlated errors undermine LLM evaluation panels. arXiv.org. https://doi.org/10.48550/arXiv.2605.29800
11. National Institute of Standards and Technology. (n.d.). SCTK: NIST scoring toolkit. https://github.com/usnistgov/SCTK.
12. Płaza, M., Pawlik, Łukasz, & Deniziak, S. (2021). Call transcription methodology for con-tact center systems. IEEE Access, 9, 110975–110988.
https://doi.org/10.1109/access.2021.3102502
13. Prakash, J., Kumar, B., Hacioglu, K., Sharma, B., Gopalan, S., Chetlur, M., Venkatesan, S., & Stolcke, A. (2025). Better pseudo–labeling with multi–ASR fusion and error correction by SpeechLLM. Interspeech 2025, 579–583. https://doi.org/10.21437/interspeech.2025–1707
14. Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I. (2023). Robust speech recognition via large–scale weak supervision. Proceedings of the 40th Interna-tional Conference on Machine Learning (ICML), Proceedings of Machine Learning Re-search, 202, 28492–28518. https://doi.org/10.48550/arXiv.2212.04356
15. Rastorgueva, E., Lavrukhin, V., & Ginsburg, B. (2023). NeMo forced aligner and its ap-plication to word alignment for subtitle generation. Interspeech 2023, 5257–5258. https://www.isca–archive.org/interspeech_2023/rastorgueva23_interspeech.pdf
16. Schwenk, H., & Gauvain, J.–L. (2000a). Combining multiple speech recognizers using voting and language model information. 6th International Conference on Spoken Language Processing (ICSLP 2000), 2, 915–918. https://doi.org/10.21437/icslp.2000–419
17. Schwenk, H., & Gauvain, J.–L. (2000b). Improved ROVER using language model infor-mation. ISCA ITRW ASR2000: Automatic Speech Recognition — Challenges for the New Mil-lennium, 47–52. https://www.isca–archive.org/asr_2000/schwenk00_asr.pdf
18. Sekoyan, M., Koluguri, N. R., Tadevosyan, N., Żelasko, P., Bartley, T. M., Karpov, N., Balam, J., & Ginsburg, B. (2025). Canary–1B–v2 & Parakeet–TDT–0.6B–v3: Efficient and high–performance models for multilingual ASR and AST. arXiv.org. https://doi.org/10.48550/arXiv.2509.14128
19. Shi, X., Wang, X., Guo, Z., Wang, Y., Zhang, P., Zhang, X., Guo, Z., Hao, H., Xi, Y., Yang, B., Xu, J., Zhou, J., & Lin, J. (2026). Qwen3–ASR technical report. arXiv.org. https://doi.org/10.48550/arXiv.2601.21337
20. Stepachev, P., Chen, P., & Haddow, B. (2024). Context and system fusion in post–ASR emotion recognition with large language models. IEEE Spoken Language Technology Work-shop (SLT), GenSEC Challenge. https://doi.org/10.48550/arXiv.2410.03312
21. Zheng, L., Chiang, W.–L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., & Stoica, I. (2023). Judging LLM–as–a–judge with MT–Bench and chatbot arena. Neural Information Processing Systems, 46595–46623. https://doi.org/10.52202/075280–2020
Downloads
Published
Issue
Section
License
Copyright (c) 2026 System technologies
This work is licensed under a Creative Commons Attribution 4.0 International License.
