Automatic assessment model of the adequacy of the request-response computer systems with using of the text generation

Abstract

The article deals with the question of evaluation of the results of the work of the request-response systems with the use of IR-based architecture, namely the system with the use of text generation, which was developed on the basis of algorithm of construction of semantic model of the document. Because the algorithms created are innovative, the development of methods for automatically testing the adequacy of a system based on them is a very relevant topic for research. Despite the fact that the query-response system is a kind of search engine, a common way to automate the testing of systems using reference tables-sets of query and answer, which is well proven in solving the classic problems of searching collections of query-relevant documents, is not suitable in the case of generation text response. Therefore, two methods of researching the results of the system operation were formulated. The first is an algorithm based on the value of the semantic correspondence coefficient, which allowed to organize the automatic evaluation of the results of the system. 520 system queries were automatically generated and executed, resulting in a value of the multiplicity of document formation correctness for the semantically appropriate mode was 0.67 versus the value of 0.49 for the semantically inappropriate mode, which indicates the feasibility of applying a semantic model-based approach to the request-response systems. The second one is based on the scoring method, for which the criteria for evaluating the answers received were described, namely: presence of the answer, degree of coincidence with the subject of the request, completeness of presentation, presence of thematic breaks and presence of meaning breaks. A total of 100 tests were conducted in the course of the study with different complexity of questions and different thematic direction of inquiries. The average of all expert assessments was 0.839, which indicates satisfactory results of the system of inquiry-response and confirms the conclusion that was obtained through the automatic testing of the system. The obtained estimates allow us to confirm the adequacy of both the developed model of the system of inquiry-response on the basis of text generation, and the adequacy of the created subsystem of evaluation of IR-based systems. It is advisable to use the developed system testing approach to test the operation of IR-based query response systems using automatic text generation.