Skip to main content

TokenSource

hyperlark

Trait TokenSource 

pub trait TokenSource {
    // Required method
    fn next_token(
        &mut self,
        parser_state: Option<usize>,
        expects: Option<&[u32]>,
    ) -> Result<Option<CustomToken>, CustomLexError>;
}
Expand description

A caller-implemented token source — the Rust custom lexer (Lark’s Lexer ABC with __future_interface__ = 2, pulled per token instead of a Python generator).

parser_state is the live parser state key on the LALR paths (the current LALR state index — Lark’s parser_state.position, the same key the contextual arm lexes by), None on the Earley path. expects is the live Earley expects set (Lark hands the custom lexer the mutated-per-pull {i.expect for i in to_scan}): the table terminal ids the next column can scan, None on the LALR paths. Ok(None) ends the stream (Lark’s generator return / EOFError).

The engines take the source as &Mutex<dyn TokenSource + Send> — shared so crate::LexerSource stays Copy and an interactive copy() shares the source (sanctioned divergence: Lark’s forks replay the remaining stream independently; see the note in LexerCursor::copy).

Required Methods§

fn next_token( &mut self, parser_state: Option<usize>, expects: Option<&[u32]>, ) -> Result<Option<CustomToken>, CustomLexError>

Dyn Compatibility§

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§