Trait PostlexStream 

pub trait PostlexStream: Send + Sync {
    // Required method
    fn next_into(
        &mut self,
        src: &mut PostlexSource<'_, '_, '_>,
        out: &mut VecDeque<Token>,
    ) -> Result<bool, ParseError>;

    // Provided method
    fn clone_stream(&self) -> Option<Box<dyn PostlexStream>> { ... }
}

One live postlex stream — THE contract, native or bindings.

The stream owns the pulling: it asks PostlexSource::pull for upstream tokens as it needs them and appends downstream ones to out. That is what lets a Python process() generator interleave with the lexer the way lark’s does, so each raw token is lexed against the live parser state.

Within one next_into call the parser state is FROZEN across however many pulls the stream makes — identical to lark, where the parser is blocked while process() pulls.

Required Methods

fn next_into( &mut self, src: &mut PostlexSource<'_, '_, '_>, out: &mut VecDeque<Token>, ) -> Result<bool, ParseError>

Append >=1 downstream token to out and return Ok(true), or return Ok(false) when the stream is complete.

out is EMPTY on entry — a driver that accumulates passes a scratch buffer — and must be left empty on Err, which makes the rollback exact rather than “appends only”. Ok(false) MAY append final flush tokens on that same call, so a driver drains out after a false before it feeds $END; it must not call again after false.

A stream with nothing to emit yet PULLS AGAIN — it does not return Ok(true) empty. That is the non-obvious half: filtering a token, or holding one back for lookahead, means looping over pull until something lands in out. IndenterSession::next_into does exactly this for an NL swallowed inside parens. Returning Ok(true) without appending drops a token silently in release, and trips a debug_assert! in every driver.

Provided Methods

fn clone_stream(&self) -> Option<Box<dyn PostlexStream>>

A fork for a copy() cursor — accepts() trial-feeds the parser state and never forks the lexer. None when the stream cannot be cloned (a Python generator); the fork then continues without postlex.

Dyn Compatibility

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors

impl PostlexStream for IndenterSession