You will use
- Lexie as a development dependency
- An ignored
.lexie/lexie.tomlspecification - Generated token classes and half-open source spans
- Poke assertions against the generated lexer
The whole program
module main
import tiny_lexer/scanner/lexer
import tiny_lexer/scanner/location.{Location}
import pooka/list
main() {
let found = lexer.lex("let answer = 42")
echo list.map(found, fn(lexeme) {
#(lexeme.token, Location("input.tiny", lexeme.span).format())
})
}
module lexer_test
import tiny_lexer/scanner/lexer
import tiny_lexer/scanner/token
import tiny_lexer/scanner/location.{Position}
import poke/should
import pooka/list
test "a binding has tokens and an exact source position" {
let found = lexer.lex("let answer = 42")
should.equal(
list.map(found, fn(lexeme) { lexeme.token }),
[
token.Let(),
token.Ident("answer"),
token.Equal(),
token.IntLiteral("42"),
token.End()
]
)
let answer = list.at(found, 1) |> should.be_some()
should.equal(answer.span.start, Position(1, 5, 4))
should.equal(answer.span.end, Position(1, 11, 10))
}
Give the language its words
pooka add lexie --dev
pooka add poke --dev
pooka run -m lexie init
In a project named tiny_lexer, edit .lexie/lexie.toml:
location-info = true
[output]
dir = "src"
[module]
name = "tiny_lexer/scanner"
[keywords]
items = ["let"]
[operators]
items = ["="]
Identifiers and numbers have defaults, so this is enough for let answer = 42.
The file is ignored by Git. Change module.name to choose the generated folder;
commit the generated Pooka modules, rather than the local specification.
Generate and ask it to read
pooka run -m lexie
pooka run
pooka test
Lexie writes location.pooka, token.pooka and lexer.pooka under
src/tiny_lexer/scanner. The program prints the token and its source address.
answer begins at line 1, column 5, byte 4. Its span ends just after its last
character; the final End token marks the end of the source.
Look at the tokens
For let answer = 42, the program prints these token/address pairs (wrapped here
so each token is easy to follow):
[
#(Let(), "input.tiny:1:1"),
#(Ident("answer"), "input.tiny:1:5"),
#(Equal(), "input.tiny:1:12"),
#(IntLiteral("42"), "input.tiny:1:14"),
#(End(), "input.tiny:1:16")
]
let is a keyword, answer is an identifier, and the integer keeps its source
spelling. The final token makes end-of-input explicit for a parser. The test
checks both the token sequence and the identifier's start and end positions.
Use location-info = false when only token values are needed. The explicit
lex_with_locations entry point remains available for tools that need spans.
See the Lexie guide for strings, comments,
number formats and semantic highlighting.