Build tokens from Layers input.

Consider the following Layers text:

Host: "huntrsoftware.com"

Providing the above as input, the lexer produces the following tokens:

1           2       3                            4         5
[WORD Host] [COLON] [STRING "huntrsoftware.com"] [NEWLINE] [ENDFILE]

It is not the job of the lexer to deduce meaning from a set of tokens; that's for the parser (see LParser).

Converting input into a token stream

The lexer provides build_tokens() which returns a vector of LToken objects from the input provided when constructing the lexer.

LLexer lexer("Host: \"huntrsoftware.com\"");
vector<LToken> tokens = lexer.build_tokens();
    // `tokens` contains {WORD,COLON,STRING,NEWLINE,ENDFILE}

After lexing, acquire and log errors

Call errors() to acquire the errors that were found while lexing the input. Consider logging them with Layers::log() (LEngine logs its lexer errors after loading each file).

for (const LError& lexer_error : lexer.errors())
    Layers::log("lexer: " + std::to_string(lexer_error.line) + ": " +
        lexer_error.message);