Struct Indenter
pub struct Indenter {
pub nl_type: String,
pub open_paren_types: Vec<String>,
pub close_paren_types: Vec<String>,
pub indent_type: String,
pub dedent_type: String,
pub tab_len: usize,
}Expand description
Lark’s Indenter, configured by terminal names.
State machine per token: an NL at paren_level == 0
emits NL then INDENT/DEDENTs by comparing the post-newline whitespace width
(count(' ') + count('\t') * tab_len, code points) against the indent
stack; open/close types track paren_level; EOF flushes remaining DEDENTs.
Confirmed-and-kept deviations from indenter.py, all reviewer calls:
- A newline-free NL token yields an empty indent (Lark
IndexErrors, after already yielding the NL). - Paren underflow is left to the parser (Lark hard-asserts; its assert also
vanishes under
python -O). - An unknown nl/indent/dedent NAME is a
PostlexErroratPostlex::session; Lark runs on, emitting a token of a type the grammar never declared. tab_len == 0is aPostlexErrorat session time, where Lark asserts atIndenter()construction.- nl/indent/dedent are matched and emitted by id, so if two terminals share a
name only the first is recognized. Lark compares name strings, so both are.
The paren types, which are only ever matched, do follow Lark and mark every
matching id (
a_duplicated_name_resolves_first_but_marks_every_paren_id). Unreachable through the compiler, which interns names.
Not a deviation: Lark’s closing assert self.indent_level == [0]
(indenter.py:76) has no counterpart here because it cannot fail — the stack’s
bottom is invariantly 0 and the pop loop stops there.
Fields§
§nl_type: String§open_paren_types: Vec<String>§close_paren_types: Vec<String>§indent_type: String§dedent_type: String§tab_len: usizeTrait Implementations§
§impl Postlex for Indenter
impl Postlex for Indenter
§fn session(
&self,
names: &[String],
) -> Result<Box<dyn PostlexStream>, PostlexError>
fn session( &self, names: &[String], ) -> Result<Box<dyn PostlexStream>, PostlexError>
Start a session for one lex pass.
names is the terminal-name table
indexed by TokenId (the session resolves its configured names once).
An unknown nl/indent/dedent name is a PostlexError (they are emitted
or consumed, so they must exist); unknown paren names are dropped — Lark
treats the paren types as matched-only string sets (Lark parity). Read more§fn always_accept(&self) -> Vec<String>
fn always_accept(&self) -> Vec<String>
Terminal names accepted in every parser state (indenter:
[NL_type]).Auto Trait Implementations§
impl Freeze for Indenter
impl RefUnwindSafe for Indenter
impl Send for Indenter
impl Sync for Indenter
impl Unpin for Indenter
impl UnsafeUnpin for Indenter
impl UnwindSafe for Indenter
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more