Struct Scanner 

pub struct Scanner { /* private fields */ }

The compiled two-partition scanner.

Implementations

impl Scanner

pub fn new( specs: Vec<ScanSpec>, g_regex_flags: u32, ) -> Result<Self, ScannerBuildError>

Compile specs, which MUST already be in Lark’s four-part sort order — rank is the index. Partitioning is by compile capability: try the fast engine, fall back to fancy-regex; failing both is ScannerBuildError::Uncompilable.

g_regex_flags is Python’s re bitmask (I=2 M=8 S=16 X=64), applied at compile time only: the fast partition via the syntax config, the slow partition by wrapping each spec in its own (?FLAGS:…) scope ([slow_source]) — never by editing the spec strings (grammar-wide flags).

pub fn new_with_engine( specs: Vec<ScanSpec>, g_regex_flags: u32, engine: ScannerEngine, ) -> Result<Self, ScannerBuildError>

As Scanner::new, but with an explicit fast-partition ScannerEngine instead of the process default. The slow partition, ranks, and merge semantics are identical across engines — only the fast matcher differs.

pub fn match_at(&self, text: &str, pos: usize) -> Option<ScanMatch>

The highest-ranked match starting exactly at byte offset pos (rank-first cross-engine merge; None = no terminal matches here). pos must be within text and on a UTF-8 char boundary; a pos past the end or mid-character yields None (no token can start there) rather than panicking — the slow anchored path slices text[pos..]. Internal callers always pass a boundary; this keeps the public method total.

pub fn match_at_cached( &self, text: &str, pos: usize, cache: &mut ScanCache, ) -> Option<ScanMatch>

As Scanner::match_at, but reuses cache for the Lazy fast engine’s lazy DFA instead of acquiring a pooled cache per call — the hot lex path threads one ScanCache through the whole token stream (~3-4% of parse). Byte-identical to match_at; the cache is pure memoization.

pub fn fullmatch(&self, value: &str) -> Option<TokenId>

The highest-ranked spec matching all of value — Lark’s Scanner.fullmatch, used by the UNLESS re-typing sub-scanner (which is itself a ranked Scanner over the absorbed literals, honoring their flags).

Precondition

Every fast-partition spec must be prefix-free with respect to itself: no pattern may match a proper prefix of value in leftmost-first preference order while also being able to match all of it. The fast probe below asks each pattern for its leftmost-first match, which is the preference-order match, not the longest one — so for a pattern like a|ab it reports a on "ab" and this method answers None, where Python’s re.fullmatch("a|ab", "ab") matches. Backtracking to a longer alternative is exactly what leftmost-first does not do.

The one production caller upholds this by construction: the UNLESS sub-scanner is built solely from re_escaped PatternKind::Str literals (lexer.rs, build_refiner), so | is always escaped and every pattern matches exactly one fixed string — a language of one string cannot contain a proper prefix of itself. fullmatch_prefix_free_precondition pins the behaviour outside that envelope.

A caller that needs the general Python semantics must not use this; the fix would be a second multi-pattern engine over \z-terminated patterns, deliberately not built here because it costs a compile per Scanner to serve a case no caller has.

pub fn token_ids(&self) -> impl Iterator<Item = TokenId> + '_

Token ids present in this scanner (minus UNLESS-embedded literals) — feeds the UnexpectedCharacters.allowed set.