Struct BasicLexer 

pub struct BasicLexer { /* private fields */ }

Lark’s BasicLexer: scanner + ignore/newline sets + composed callbacks + the name↔id tables (ids cover all conf terminals, including scanner-removed absorbed literals).

Everything shareable is behind Arc, so deriving a variant lexer copies pointers rather than tables.

Implementations

impl BasicLexer

pub fn new(conf: LexerConf) -> Result<Self, LexerBuildError>

The build pipeline, in Lark’s order — steps 0+2 prepare the terminal set ([PreparedTerminals::new]); the rest are one method per step:

1. validate ([PreparedTerminals::validate]) unless conf.skip_validation — then newline_types ([PreparedTerminals::classify_newlines]) from each terminal’s final to_regexp() (step 1 of the plan’s build sketch);
2. sort ([sort_terminals]) by (-priority, -max_width, -len(value), name);
3. UNLESS absorption ([PreparedTerminals::create_unless]): refinement callback built from the full unless list; scanner-removal gated separately on the flag-subset test;
4. compose user callbacks ([PreparedTerminals::compose_callbacks]) via the gated CallChain;
5. compile the scanner ([PreparedTerminals::compile_scanner]), minus UNLESS-embedded literals.

pub fn names(&self) -> &[String]

Terminal names by id (conf order).

pub fn name_to_id(&self, name: &str) -> Option<TokenId>

pub fn next_token( &self, text: &str, state: &mut LexerState, ) -> Result<Option<Token>, LexError>

Lark’s next_token flow: scan at the byte cursor; on a match, feed the tracker before filling end positions; run the callback after end positions; a token is built when it is not ignored or has a callback, returned only when not ignored; last_token updates only on a returned token. Ok(None) = input exhausted — no synthetic EOF token (boundary inputs).

pub fn next_token_dont_ignore( &self, text: &str, state: &mut LexerState, ) -> Result<Option<Token>, LexError>

Self::next_token re-emitting %ignored terminals — the lex(dont_ignore = true) path. Same terminals, same positions, same callbacks; only the drop of an ignored token is suppressed.

pub fn lex<'l, 't>(&'l self, text: &'t str) -> Lex<'l, 't>

Iterate all tokens of text from a fresh state.

The state is checked out of Self::ls_pool, so the lazy-DFA scan cache is WARM — a repeated lex (the Earley pre-scan and the public Lark.lex both drive this) never re-warms it from cold, the same win the LALR path already took. Observable state is reset per call (LexerState::reset_keep_cache), so a pooled pass is byte-identical to one over a fresh LexerState.

pub fn lex_with<'l, 't>( &'l self, text: &'t str, dont_ignore: bool, ) -> Lex<'l, 't>

Self::lex, re-emitting %ignored terminals when dont_ignore.

One lexer serves both: the flag reaches the loop as a const parameter, so neither path pays for the other, and BOTH share this lexer’s warm state pool. A second lexer with an emptied ignore set would own a second, cold pool and re-warm the lazy DFA on every call.

pub fn is_ignored(&self, id: TokenId) -> bool

Whether id is an %ignore terminal of THIS lexer.

Exposed for the bindings’ lexer_callbacks= bridges, which need it to reproduce Lark’s split return contract (lexer.py:637-643): the callback runs either way, but its result is DISCARDED for an ignored terminal (any type is legal there) and must be a token for a live one. A binding callback is handed a Token with no other channel to ask, so it asks here — via the root crate::LarkInstance::basic_lexer, whose id space is the global one every callback sees (a contextual lexer maps its per-state local ids to global before calling; see contextual::wrap_callbacks_to_global).

This reports the lexer’s CONFIGURED ignore set, which is what a callback bound to the root lexer must observe — unaffected by whether a particular Self::lex_with pass is re-emitting ignored terminals.

pub fn without_ignore(&self) -> Self

👎Deprecated since 0.1.0-beta.2:

use lex_with(text, dont_ignore); this clone owns a cold state pool

The dont_ignore variant: same terminals and callbacks, ignore set emptied. Callbacks survive (they’re Arc).

PREFER Self::lex_with. This clone owns a SEPARATE, COLD state pool, so one built per call re-warms the lazy DFA on every lex — measured 2.9x on a small input. lex_with reaches the same behaviour through a const parameter on this lexer, keeping the warm pool.

Trait Implementations

impl Debug for BasicLexer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.