In order for a machine to understand what needs to be done when reading a program, it first needs to break the text into components called tokens. A token is an object which has a type and value. This process of breaking apart the text into tokens is called lexical analysis. This component is represented in the lexer.py script. The implementation of the token object is as follows.

class Token(object):
    def __init__(self, type, value):
        self.type = type
        self.value = value
                                    

The lexer (or lexical analyser) is represented as an object which contains the entire text of the Cuneiform script, the current position being analysed in the text (index of the current character in the string), and the character in the current position. A Lexer object is represented as follows.

class Lexer(object):
    def __init__(self, text):
        self.text = text
        self.pos = 0
        self.current_char = self.text[self.pos]
                                    

The lexer and Token objects have a relationship as shown below.

Consider the following operation in Cuneiform.

var x = 1 + 2;

When this expression is obtained by the lexer from a Cuneiform script, it gets a string "var x = 1 + 2;". In order for the lexer to actually understand what needs to be done with this string, it is broken apart into components called tokens.

The function get_next_token in the Lexer class is the lexical analyser. Each time this is called, the next token from the script of characters is obtained.

Consider the above code being passed into the lexer. The code is stored in the variable text, that holds the code string. pos is an index into that string. The value of pos is initially set to 0, and therefore, points to the character 'v' in var.

In this text, it identifies that 'var' is a reserved keyword in the Cuneiform programming language, and assigns it the token object Token(VAR, 'var')

The lexer identifies x to be a variable ID, and as a result assigns it the token Token(ID, 'x').

Similarly, the entire text is divided into the following set of tokens.

• Token(VAR, 'var')
• Token(ID, 'x')
• Token(ASSIGN, '=')
• Token(INTEGER_CONST, 3)
• Token(PLUS, '+')
• Token(INTEGER_CONST, 5)
• Token(SEMI, ';')

Cuneiform consists of three types of tokens:

1. Regular tokens
2. Reserved keywords
3. System operations

Regular Tokens

Reserved Keywords

System Operations

System operations are assigned tokens of type SYSOP, and values relevant to the type of operation. The possible values are:

• Response
• InternalDatabase
• File
• HTTP
• DateTime
• ExitIntent
• Initiate

The next section discusses implementation details of the Parser.