Function pomelo_lex::lex

pub fn lex(src: &str) -> Vec<LexToken>

Convert the input src to a sequence of tokens.

Lexing errors either result in LexKind::Unknown or are encoded in extra data in the variant (e.g., LexKind::String { terminated: bool }). Error handling is deferred to the parsing stage.

(Note: we can improve this in the future to be a little more intelligent at the lexing stage, e.g., in lexing numeric literals.)