Struct Lexer 

pub struct Lexer {
    file: BufReader<File>,
}

The Anillo Lexer

The Anillo Lexer is purposefully simple. Lexemes (or Tokens here) are only delimited by whitespace and “Special Characters” (these are defined in the techincal specification, but generally punctuation like brackets, or special semantic tokens like ‘$’ count as “Special Characters”)

Fields

file: BufReader<File>

Implementations

impl Lexer

pub fn new(path: &Path) -> Result<Lexer>

pub fn tokenize(&mut self) -> Result<VecDeque<TokenInfo>, Box<dyn Error>>

Eagerly lex the ModuleSource and return a buffer of TokenInfos to the caller.

Lexing here is done as simply as possible (and the language was designed with this in mind). There should be absolutely no look ahead peeking done at this level. Compilation errors at this level are supported, but rare (only really possible if an invalid unicode character is found).

fn parse_str(tok: &str) -> Token

Helper function for tokenize.

Essentially anything that isn’t a Special Character will be lexed here. Where possible, we will store the lexeme in a rich token type (Like keywords). Everything else however will be parsed as identifiers (even numbers). The parser handles converting these to their expected types at AST generation time in parser.rs