Lexical analysis of program code

Abstract

The growing volume of technologies, the end of actively used development tools support, outdated API etc., entails the need of program codes conversion. In IT compa-nies and not only, often begged the question of deprecated software support, which cus-tomers continue to use, or translation of current software to actual technologies. It is more rational for programmers to use the conversion and save most of code base, than rewriting all software by hand, even if manual adjustment is needed. At this moment, there are few high-quality code conversion systems. Largely, conversion systems work well only with similar programming languages. The task of program codes conversion is a deep and complex topic. To convert the software code, you must first analyze, select components and form a structural representation. Any analysis of program code begins with lexical analysis. Although lexical analysis is considered a relatively simple step, it plays a key role in the entire system of analysis and transformation of software code, and also has a large number of both theoretical and practical features that require careful study. This article considers the definition of the lexical analyzer, its functional composition and principles of construction, provides key differences between the lexeme and the token. Two approaches have been proposed and considered to solve the search for tokens in the program code: regular expression search and finite state machine search. For these approaches, examples of the formation of search templates under cer-tain rules of vocabulary were given. As a result, the optimality of the use of determinis-tic finite state machines during the practical implementation of the lexical analyzer on real computing machines was substantiated.