Crate hyperlark 

Hyperlark core — the pure-Rust parsing library.

The shipped surface is the lexer stack (BasicLexer, Scanner, ContextualLexer), the LALR and Earley engines, the grammar model, the in-Rust .lark compiler, the serialized-grammar loader/saver, the custom- lexer and postlex seams, and the Lark facade every binding wraps. The differential comparison format lives in the hyperlark-conformance crate — dev tooling, deliberately not part of this shipped library.

Structs

Action

A LALR action cell, packed into a u32 (POC-proven encoding): Action::NONE = u32::MAX; high bit set = reduce (low 31 bits = rule index); high bit clear = shift/goto target state. A repr(transparent) newtype so a cell can’t be used as a bare integer (indexed with, arithmetic’d) — the packed word is reachable only through the constructors/queries below.

ArenaIdx

Index into crate::model::ParsedTree::arena (the child-node store) — a crate::model::Tree::children_start. NOT a length: children_len is a plain count, not an index into any space, so it stays a bare u32/usize.

BasicLexer

Lark’s BasicLexer: scanner + ignore/newline sets + composed callbacks + the name↔id tables (ids cover all conf terminals, including scanner-removed absorbed literals).

BasicTokenSource

A TokenSource over the shared BasicLexer — the analogue of the Lark test classes’ BasicLexer(copy(lexer_conf)) wrapper (CustomLexerNew), and the building block the conformance custom-lexer port drives. Owns its cursor (LexerState); ignores the parser-state key like Lark’s BasicLexer.next_token.

CompileOptions

Compile-time options (the facade threads Lark kwargs here; defaults match Lark’s).

CompiledGrammar

The compiled grammar — same downstream shape as json_loader::LoadedGrammar.

ContextualLexer

Per-parser-state lexer set.

Cursor

A borrowed walk cursor over a ParseResult — the core tree-walk primitive the deferred Visitor/Interpreter/Transformer classes will build on (Provisional-but-churn-averse). Resolves display names through the result’s table.

CustomToken

A token produced by a TokenSource, typed by terminal NAME — exactly the seam Lark exposes (a custom lexer yields Token(type: str, value)); the engine resolves the name through the parse table per token. Positions are the caller’s verbatim (the engines never recompute them; $END borrows them as-is).

DisplayId

Index into crate::model::ParseTable::display_names; a Tree’s data label (a rule’s origin, or alias when present). Shares no space with RuleId/NontermId — display names are their own dense id space in loader-preserved order.

EarleyDynamicMatchers

The dynamic-Earley lexer built once at construction (the seam mirroring Lark’s EarleyRegexpMatcher.__init__): the per-terminal forward matchers + parallel terminal-priority vector. A zero-width / bad regexp raises the GrammarError here (construction), never mid-parse — every subsequent parse borrows this by reference (see super::EarleyLexerSource).

EarleyParseJob

The invariant inputs to an Earley parse — the peer to the LALR engine’s crate::lalr::ParseJob, keeping the two entry surfaces parallel (this engine’s parse_with deliberately mirrors LALR’s). Bundles what to parse and how to lex it; the strategy (transform) stays a separate argument.

EarleyParseOptions

Per-parse Earley options. Mirrors crate::ParseOptions (LALR) plus the ambiguity field the Earley engine alone honors (Lark’s ambiguity=). Earley owns its own options type — LALR’s has no ambiguity and lives behind the engine boundary.

EmbeddedStdlib

The embedded Lark stdlib: all four lark/grammars/*.lark files (embed all four — %import python.X / %import lark.X are working Lark features via its package loader). Baked in via include_str!; no filesystem needed in native/WASM/C.

EvalNode

One node under ParseResult::eval: dispatch on name, fold children on demand with eval.

FnFold

Fold from two closures — for when a one-off fold doesn’t warrant a type. See fold_fn.

ForeignError

The opaque payload a ParseError::CustomLex carries out of a failed custom-lexer pull (crate::custom::CustomLexError::Custom): an owned, thread-safe Any a binding downcasts back to its host exception with identity intact (in Lark, the custom lexer’s exception escapes parse unwrapped — a stored PyErr round-trips through here). Arc keeps ParseError Clone; equality is payload identity (Arc::ptr_eq), the only equivalence an opaque value supports.

ForeignValue

The opaque payload carried by NodeValue::Foreign: an owned, thread-safe Any a binding downcasts back to its host value. Arc keeps NodeValue Clone (the SPPF walk clones freely) and Send + Sync (no core public item is !Send/!Sync; an owned PyObject/JS handle is itself Send + Sync). Manual Debug — dyn Any is not Debug.

GrammarError

A grammar-compile failure — Lark’s GrammarError class surface (message-detail parity deferred; the raise/no-raise decision is v1).

GrammarErrorReport

One entry of find_grammar_errors’s result: the recorded UnexpectedInput plus Lark’s _error_repr string. The binding maps error to a hyperlark.exceptions.UnexpectedInput subclass and keeps repr as the tuple’s second element.

Handle

A batching-callback slot handle: a binding-private slot index (into the hook’s pending-value table) packed with an optional splice-flag high bit. A repr(transparent) u32 newtype so the packed word — like Action — is reachable only through its accessors, never as a bare integer.

ImmutableInteractiveParser

Lark’s ImmutableInteractiveParser: the copy-on-feed wrapper. Some methods are overridden in Lark (feed_token, exhaust_lexer, as_mutable) so they return a new immutable and never mutate self; others are inherited unmodified (resume_parse, feed_eof, pretty, accepts, choices, copy) and keep their base semantics — including the quirk that resume_parse runs the base parse_from_state over the shared state and mutates it in place. This surface mirrors Lark faithfully, documenting each quirk with its citation rather than idealizing it away.

ImportedFile

One resolved imported file — Lark’s used_files KEY: a filesystem/package joined_path plus whether it came from the embedded stdlib (FromPackageLoader), which the binding surfaces as a PackageResource rather than a plain path string (list_grammar_imports).

Indenter

Lark’s Indenter, configured by terminal names.

IndenterSession

The live indenter state (Lark’s instance fields, indenter.py:29-34): paren_level, the indent_level stack, plus the resolved token ids and the last upstream token’s positions (the EOF flush’s borrow source, gated on that token’s value being non-empty). Fresh per session: paren_level 0, indent_level [0], no last token.

InteractiveParser

The mutable interactive parser. Construction: crate::lalr::parse_interactive.

Lark

A compiled grammar ready to parse — Lark’s Lark object. Holds the shared ParseTable (behind an Arc, so ParseResult can carry a cheap clone) plus the concrete lexer the resolved frontend needs. Send + Sync by construction.

LarkOptions

Construction options — Lark’s Lark(grammar, **options) kwargs as a flat Default-able struct (no fluent builder in core; a builder is binding sugar). Field-init the ones you need over Default::default.

LeafCursor

lark scan_values: live pre-order over leaf children — every non-Tree child value (Token, None placeholder, Foreign), left-to-right, exactly the recursive generator’s order (design P5). Yields the leaf’s arena slot.

Lex

Iterator over a lex pass. Yields tokens until exhaustion; a lex error ends the stream after being yielded once.

LexerConf

Everything BasicLexer::new consumes — never global state, so one is built per accept-set from a filtered conf. always_accept is deliberately absent: it is a ContextualLexer-only param whose effect arrives pre-folded into the terminal subset. Clone (callbacks are Arc-shared) — a custom-lexer instance retains a copy for the source to build from, Lark’s lexer_type(lexer_conf) handoff.

LexerConfReadError

A structural problem in the serialized grammar JSON.

LexerState

Mutable lex-pass state (Lark’s LexerState minus the text, which callers borrow): the tracker + last_token, which Lark updates only on a returned token and error reporting later consumes.

LineCounter

Line/column/offset tracker (Lark’s LineCounter), with both cursors: byte_pos is where the scanner reads; char_pos (and every derived position) counts code points, exactly as Python len/rindex do.

LoadError

A structural problem in the serialized grammar JSON.

LoadOptions

Loader-time options (grammar-shaping kwargs that Lark applies at build time, not parse time).

LoadedGrammar

The loaded grammar: the lexer conf + the parse table (which owns the rules). start symbols come from parser_conf.start.

MaybePos

A Pos whose fields may individually be unknown.

MaybeSpan

A start..end pair of MaybePos.

MetaIdx

Index into crate::model::ParsedTree::meta_arena — a crate::model::Tree::meta. Hand-written rather than [define_idx]: it needs a NONE sentinel (mirrors crate::model::Action::NONE / crate::model::Handle’s packed-word sentinels), so it does not fit the plain-newtype macro shape.

NontermId

A nonterminal id: index into crate::model::ParseTable::nonterm_names and the COLUMN of a goto_actions row. A rule’s origin_id is one (the LHS nonterminal that a reduce gotos on).

OnErrorContext

The context handed to a parse(on_error=) callback: the escaping error plus a mutable recovery handle onto the live parse.

OwnedInteractiveParser

An interactive LALR parser that owns its backing Lark and input text, so it can be stored in a binding handle and driven token-by-token across calls. Build one with Lark’s interactive entry (bindings call OwnedInteractiveParser::new).

ParseJob

The invariant inputs to a parse — what to parse and how to lex it — bundled so the six values that otherwise thread identically through every LALR entry (parse_with, parse_discard, parse_interactive, and the private [drive_to_accept]) travel as one. It is deliberately orthogonal to the strategy — what each reduction yields (build a tree / discard / transform) — which stays a separate argument (transform on parse_with, the reduce mode inside [drive_to_accept]). That strategy axis is the one thing that actually differs between the entries, so it is the one thing left un-bundled.

ParseOptions

Per-parse options.

ParseResult

A finished parse — self-describing: the owned tree plus the table its data_ids index, so a Cursor resolves display names from the result alone. Both fields are Arc (O(1) clone, Send, C-expressible).

ParseTable

The LALR parse table. token_actions is a locked cross-phase contract: per-state rows, dense state ids, carrying both shift and reduce-lookahead terminals — accept-sets and accepts() read through it; a shift-only row would silently shrink them. goto_actions stays per-state addressable too: choices() re-merges both maps. Loader-preserved order is canon — state ids, rule order, __ANON ids pass through exactly as serialized, never renumbered.

ParsedTree

A finished parse: root + arenas (POC-proven layout — children in one growing buffer, tokens in another; meta_arena parallel, empty in off-mode).

Pattern

A terminal’s pattern: raw value + per-terminal flags (chars: i m s x l u, as Lark serializes its frozenset) + the optional raw grammar-source form.

PendingMeta

A Meta::Pending payload: the endpoint plans plus — for a ?rule collapse onto an already-stamped child — the child’s prior meta (base), whose plain fields survive and whose container fields are the fallback when an endpoint resolves to no contribution (lark’s per-endpoint hasattr update).

PendingSpan

The raw PRE-FILTER span plan of one transform-mode reduce — what lark’s PropagatePositions (wrapping outside the child filter) reads: filtered-out anonymous tokens included, _-splice children read through their carried plans.

Pos

One point in the input.

PostlexError

A postlex configuration/stream error (unknown terminal name, or the indenter’s dedent-mismatch — Lark’s DedentError).

PostlexSource

What a crate::postlex::PostlexStream pulls upstream tokens from — one token per call, against the LIVE parser state.

PostorderCursor

Post-order DFS, children left-to-right — the Visitor_Recursive.visit and post-parse Transformer.transform order (design P6). Live: frames descend the arena as they go.

ReduceCx

The arena/token context handed to a ReduceCallback::reduce call. Both engines build one over their live child arena + token arena, so the hook resolves NodeValue::Token(idx) leaves and constructs arena-backed NodeValue::Trees (the representation a _-splice parent can drain) without engine-specific glue.

RouteSet

Which reductions still call the user’s ReduceCallback vs. build natively — a transform pays the callback (host-boundary) cost only for rules the user customized; the rest build as a plain parse. Keyed by display (a rule’s callback identity — aliased alternatives get their own; design A1), not rule id. A hooked display fires the callback; the rest take the native BuildTree/splice arms in apply_shape — no hook call (children still build, into the arena or a Splice). Built once by RouteSet::build; fetched per parse via ReduceCallback::routes (None = all hooked, today’s behavior).

Rule

A grammar rule. Lark’s RuleOptions fields are flattened onto the rule (keep flattened): a rule with multiple BNF expansions would otherwise SHARE one options object, which is exactly the aliasing that makes Lark’s priority='invert' negate a multi-expansion rule once per expansion (a net no-op — see the tp_prio_plus_invert note in the conformance corpus). Owning the fields by value makes that class of bug unrepresentable.

RuleId

Index into crate::model::ParseTable::rules — the reduce target a reduce action decodes to (crate::model::Action::rule_id). NOT a nonterminal id (that is origin_id, a distinct space).

RuleShape

Per-rule reduce-time instructions — one shared build_rule_shape is the tree-shaping contract both engines reuse (the POC’s duplicate copy is a named drop). Mirrors Lark’s maybe_create_child_filter

SaveError

A save-emit failure (mirrors Lark’s save() NotImplementedError for the non-LALR guard — a table only exists for LALR).

SaveOptions

The scalar, compile-invariant options that populate data.options. The callable/hook options (transformer, postlex, lexer_callbacks, edit_terminals) are deliberately absent — they are re-supplied at load (Lark’s _LOAD_ALLOWED_OPTIONS) and cannot cross a portable JSON boundary; the Python binding layers them into a pickle envelope wrapping this JSON.

SavedOptions

The compile-invariant half of a save’s data.options block — the options the loader/compiler BAKED into the serialized rules and table, so a faithful load must restore them from the file rather than re-accept them from the caller (Lark’s _LOAD_ALLOWED_OPTIONS complement, lark.py:248 / :570-575).

ScanCache

Per-lex-session scratch for the Lazy fast engine’s lazy DFA, threaded through LexerState so the hot scan reuses a cache instead of acquiring a pooled one per token (the pool exists for &self thread-safe access; a &mut-threaded cache is cheaper on the single-threaded lex path — worth ~3-4% of parse). Keyed by DFA identity because a contextual lexer’s per-state lexers each have their own DFA and a lazy Cache is valid only for the DFA that built it. A no-op for the Meta/Dense engines. Empty until the first Lazy scan; a fresh one costs nothing.

ScanMatch

A match starting exactly at the scan position.

ScanSpec

One ranked, compiled-ready terminal: its id + final to_regexp() output.

Scanner

The compiled two-partition scanner.

SentinelOption

Option<T> niche-packed into T::NONE. #[repr(transparent)] so it is layout-identical to T (4 B for u32, vs 8 B for Option<u32>). The point is the size, not a compiler niche — all the None/Some logic is explicit.

ShapeError

An empty_indices/expansion mismatch or other shape-construction problem.

Span

A start..end pair of Pos — the unit that actually travels.

StateId

An LALR state id: the ROW index of a token_actions/goto_actions table, and the value carried by crate::model::ParseTable::start_states / end_states and the engine’s state_stack. Bounded dense state numbering from the compiler/loader (canonical BFS renumbering).

SubtreesCursor

lark iter_subtrees order: process a growing queue (append each processed node’s Tree children reversed), then yield the whole thing reversed. The snapshot is taken eagerly at construction, matching lark’s eager queue walk — later host mutations never change the yielded position sequence.

TerminalDef

A terminal definition: name + pattern + lexer priority. Priorities may be negative; default 0.

Token

A lexed token — Lark’s 8 slots. Positions are code points: 0-based *_pos, 1-based line/column. SentinelOption<u32> (4 B each; u32::MAX = position-less) rather than Option<u32> (8 B) — a token can be constructed position-less (callbacks, $END), though the lexer always sets all six. BasicLexer::next_token guards inputs via [input_exceeds_u32] (LexError::InputTooLarge), which reserves u32::MAX, so every position here is provably < u32::MAX and round-trips through the sentinel losslessly.

TokenArenaIdx

Index into crate::model::ParsedTree::token_arena — a crate::model::NodeValue::Token’s payload.

TokenId

A terminal id: index into crate::model::ParseTable::id_to_token, the COLUMN of a token_actions row, and a crate::lexer::Token’s type_id. One id space shared by the lexer and the table (conf order; $END last).

TopdownCursor

lark iter_subtrees_topdown: live pre-order, left-to-right, no dedup. lark yields a node before reading its children (design P4), so the last-yielded node’s children are pushed lazily at the next next() call — the host’s yield window can prune the descent (skip_children) or hand off (stack) without the stale children already being on the stack.

Tree

A tree node: children live in the shared arena (slice descriptor), meta — when propagate_positions is on — in the parallel meta arena.

WidthError

A pattern that neither width route can analyze. The lexer build treats this as an uncompilable terminal (validation).

Enums

Ambiguity

Lark(..., ambiguity=…). v1 = Resolve (default: pick one derivation by summed priority) | Explicit (wrap every derivation in _ambig/_iambig/_inter Trees). 'forest' (raw SPPF) is deferred from v1; #[non_exhaustive] lets Forest land additively later.

ChoiceAction

A merged choices() entry — Lark returns the raw states[position] map whose keys include NON-TERMINAL goto names: hyperlark re-merges the split token_actions/goto_actions rows to reproduce it.

CustomLexError

A TokenSource failure.

EarleyLexerSource

Which scan strategy drives an Earley parse (Lark’s resolved lexer; the frontends facade’s ResolvedLexer). Earley defines its own source enum rather than importing LALR’s LexerSource (engine boundary): the basic path takes a pre-lexing BasicLexer token stream; the dynamic paths take the construction-built matchers.

FoldLeaf

A leaf handed to Fold::leaf: a lexed token, or the None placeholder a [...] optional inserts under maybe_placeholders (Lark’s None child).

FrontendError

A construction-time frontend/engine-configuration failure. Lives in the shared model (beside ParseError) so both the crate::frontends facade and the crate::earley engine can raise it without the engine reaching across the engine boundary into the facade layer (mirrors the sanctioned AMBIG_DATA_ID retrofit). crate::frontends re-exports it for the public surface. Carries Lark’s exception class so the conformance runner can assert both the class and the verbatim message: the postlex×dynamic and parser×lexer rejects raise ConfigurationError (FrontendError::Configuration); the dynamic×lexer_callbacks reject and the dynamic Earley zero-width/bad-regexp matcher-build reject raise GrammarError (FrontendError::Grammar).

LarkBuildError

A construction failure — Lark’s build-time exception classes, carrying the class name + verbatim message so a binding reproduces the Python exception. Provisional richness.

LexError

Runtime lex errors. Construction-time problems are LexerBuildError.

LexerBuildError

Construction-time errors, mirroring Lark’s validation, all gated by skip_validation.

LexerSource

Which lexer drives a parse (contextual becomes the real LALR default; the basic path stays for bootstrap/diagnostics). Copy — it holds only immutable references, so a caller can drive the same source through more than one entry (e.g. the tree vs. no-tree parity A/B).

LexerSpec

The user-facing lexer choice: Lark’s lexer= strings, plus Custom for a caller-supplied lexer. Like Lark, a custom lexer is exempt from the parser×lexer matrix entirely — every parser accepts one.

Meta

Node position info under propagate_positions. Not a generic Tree<M>.

NodeKind

What a Cursor points at. Provisional: the deferred visitor classes build on this, so #[non_exhaustive].

NodeValue

A parse-tree node value. Splice is a reduce-time intermediate produced and drained inside the LALR engine (LALR-private). The engine never returns one — finish_parse re-wraps a root Splice and a debug_assert guards it (lalr/state.rs). The variant is pub only because this enum is shared cross-engine; a hand-built ParsedTree containing a Splice is unsupported input that violates the tree walkers’ expectations (they may panic).

ParseError

Structured parse failure, mirroring Lark’s UnexpectedInput subclasses. $END rejection raises UnexpectedToken with the $END token exactly as Lark’s LALR does — UnexpectedEOF is Earley-only and the POC’s rewrite is a named parity bug we do not replicate.

ParserKind

The engine the grammar is configured for. hyperlark implements Lark’s lalr and earley; Lark’s cyk parser is not supported.

PatternKind

Which kind of pattern a terminal carries (Lark’s two Pattern subclasses).

PriorityLiteral

Lark’s priority= literal ('auto' | 'normal' | 'invert') — Rust None (absence) maps to Lark’s priority=None. Only Self::Invert and Rust None trigger the pre-serialize negate/strip; Auto/Normal are the Earley-runtime knob, no-ops at compile time.

PriorityMode

Lark’s priority= kwarg values.

ResolvedLexer

What auto (or an explicit spec) resolves to. Dynamic/DynamicComplete are Earley-only; the LALR engine never sees them (the matrix rejects the pairing first).

ScannerBuildError

ScannerEngine

Which concrete regex-automata engine drives the fast (look-around-free) partition. Provisional knob (see LarkOptions::scanner_engine); the merge semantics (MatchKind::LeftmostFirst, anchored-at-pos span search) and the resulting token stream are byte-identical across all three — only the build-time / warm-scan / resident-memory trade differs.

SpanPlan

One endpoint of a transform-mode raw-children span scan.

SpanSource

One candidate in a pending endpoint walk, in lark’s _pp_get_meta order.

SpliceSpanPayload

The boxed SpliceSpan payload: the _-rule’s span, the poison marker (see Meta::Poisoned), or — under a batching transform — the deferred endpoint plans a scanning parent inlines (lark reads the spliced Tree(_rule)’s meta, whose value may depend on callback results).

Symbol

A grammar symbol in a rule expansion (Terminal/NonTerminal).

Constants

AMBIG_DATA_ID

Ambiguity tree-data sentinels. Earley ambiguity='explicit' wraps derivations in _ambig, _iambig/_inter Trees whose data names are not rule origins, so they have no slot in ParseTable::display_names. They intern as reserved data_ids at the top of the u32-backed DisplayId range — the POC used one such sentinel (AMBIG_DATA_ID = usize::MAX, parser.rs:154); this transposes the idea into DisplayId space for all three. Resolve names via ParseTable::data_name, never by indexing display_names. (Sentinels sit far above any real display_id; a grammar with u32::MAX - 2 distinct tree names is not representable regardless.)

CURSOR_ROOT

The position addressing the root node (ParsedTree::root, which lives outside the arena). Unreachable as a real slot: Tree::children_start is u32, so an arena never has a valid slot at u32::MAX.

IAMBIG_DATA_ID

INTER_DATA_ID

SAVE_FORMAT_VERSION

The canonical-JSON save format version. Bump the (single) integer on any wire-incompatible change to the envelope the loader consumes. The loader reads a top-level hyperlark_format_version and rejects a value greater than this (a newer save cannot be trusted to parse under an older build); an absent header is accepted — the corpus fixtures and the loader’s own input format predate the header and are, by construction, the current pin’s shape.

WIDTH_UNBOUNDED

Unbounded-width sentinel: strictly greater than every finite code-point width. Only the ordering matters — do NOT mirror CPython’s MAXREPEAT or its 2**64 (width sentinel).

Traits

Fold

A bottom-up fold algebra: what a leaf is worth, and how a rule combines its children’s values. The Rust sibling of Lark’s Transformer — both methods dispatch on a display name, with the same shape: rule is called with the rule’s DISPLAY name (aliases included) and the already-folded children in order; leaf with the token TYPE name ("NUMBER" — or "" for the hole a [...] optional inserts under maybe_placeholders).

FoldSink

A monomorphic, UNBOXED reduce-time fold — the Rust-native peer to ReduceCallback that does NOT route values through the type-erased NodeValue::Foreign (Arc) box the language bindings need. The concrete sink owns its own typed value stack (e.g. Vec<f64>), so a pure-Rust fold pays no per-reduction heap allocation and no per-reduction Vec<NodeValue> children shaping: Self::shift converts a token leaf to a value and pushes it; Self::reduce keeps the shaped children of one rule (shape.to_include, ?-collapse via expand1_inline), folds them, and replaces them with the single result. It drives the identical token stream as the tree build (the reduce value never steers the parser), so it accepts/rejects identically — only the value representation differs. Driven by crate::Lark::parse_fold; the concrete sink exposes its own accessor for the accepted root value.

GrammarLoader

One %import source. Returns (joined_path, text) — the joined path threads back as the base for nested imports — or None = try the next source (mirrors except IOError: continue).

InteractiveHandle

The recovery handle the on-error callback drives — a lifetime-erased view over the live InteractiveParser (so the callback type carries no engine lifetime). The callback feeds corrective tokens through this, then returns true to have the façade retry resume_parse. Provisional surface.

Postlex

A postlex configuration — the factory for per-parse sessions plus the always_accept names the contextual lexer unions into every state and the compiler threads into terminal pruning.

PostlexStream

One live postlex stream — THE contract, native or bindings.

ReduceCallback

The public reduce-time transform hook — Lark’s embedded transformer= (create_callback), the one core seam every binding (Python Transformer, WASM JS fn, C fn-ptr) boxes as Box<dyn ReduceCallback + Send>. Object-safe (no generics, no Self return) so it is dyn-compatible; Send at the box site (an owned PyObject/JS handle is Send). One trait serves both engines: LALR drives it from apply_shape, Earley from the per-rule chain’s node builder.

Sentinel

A type that reserves one value of its domain to mean None, so SentinelOption<Self> needs no discriminant word and stays size_of::<Self>().

TokenSource

A caller-implemented token source — the Rust custom lexer (Lark’s Lexer ABC with __future_interface__ = 2, pulled per token instead of a Python generator).

Functions

build_rule_shape

Build a RuleShape. The single shared implementation — the bridge must call this, never re-derive.

compile_grammar

Compile .lark source. from_lark_source-style construction also routes through here.

earley_build_dynamic_matchers

Eagerly build the dynamic-Earley matchers at construction, surfacing a zero-width / bad-regexp FrontendError::Grammar (Lark’s construction-time GrammarError) before any parse runs. g_regex_flags is the grammar-wide bitmask (0 for the common no-g_regex_flags grammar).

earley_parse

Parse text from start with the basic lexer — thin delegate to parse_with, mirroring the LALR engine’s parse. (Convenience for the basic path; the dynamic paths go through parse_with directly.)

earley_parse_with

The Earley parse entry — signature mirrors the LALR engine’s parse_with (including the postlex param) plus Earley’s ParseOptions with its ambiguity field. Flow (Algorithm Sketch): build recognizer tables from the shared model → run the [recognizer] over the [scan] strategy chosen by lexer (postlex, when present, wraps the basic token stream via the connector) → gate + run the [prioritizer] sum-cascade → [extract] by ambiguity through the per-rule [chains]. A no-parse maps to the per-lexer error class (basic: UnexpectedToken/UnexpectedEOF; dynamic: UnexpectedCharacters). Dynamic matcher construction (with its zero-width/bad-regexp GrammarError) happens earlier, in build_dynamic_matchers; this entry only borrows the built matchers.

find_grammar_errors

find_grammar_errors(text, start='start'). Parses source (Lark appends a trailing newline) error-tolerantly over the meta-grammar, collecting (UnexpectedInput, repr) pairs, then keeps the first error per line in ascending-line order.

fold_fn

Build a Fold from two closures — two matches with one shape: on_leaf(token_type_name, leaf) -> T and on_rule(rule_name, children) -> T. (A placeholder hole reaches on_leaf with the name "".)

list_grammar_imports

list_grammar_imports(grammar, import_paths) — see [super::builder::list_grammar_imports].

load_grammar

Load a full serialized grammar document.

node_at

Resolve a cursor position to its node (CURSOR_ROOT = the root).

optimize_substitutions

The enabled (original, substitute) pairs, for the conformance span-fixture and flag-on corpus cross-checks (a compact, stable read of the table without exposing the internal struct).

parse

Parse text from start using the basic lexer. Thin delegate to parse_with over LexerSource::Basic so the shift/reduce loop lives in one place (the basic path stays for bootstrap/diagnostics — Lark.lex is basic — while the contextual lexer is now the real LALR default).

parse_discard

No-tree parse: drive the same lexer/postlex/feed loop as parse_with but under [Reduce::Discard], so every reduce yields NodeValue::None with no shaping and no arena write. Returns Ok(()) on accept, the same ParseError on reject. Because the reduce value never steers feed_token, this consumes the byte-identical token stream and reaches the identical accept/reject as the tree path — only tree construction is elided. The peer to the POC’s parse_no_arena for no-tree benches (and a discard-parse API candidate: recognition-only validation without paying for a tree).

parse_fold

Monomorphic reduce-time fold: drive the same lexer/feed loop as parse_with but under [Reduce::Fold], forwarding every shift/reduce to sink (which owns its own typed value stack). No tree and no Foreign(Arc) box is built — the Rust-native peer to a transformer= hook, without the per-reduction heap box. Returns Ok(()) on accept (the root value is left on the sink); the same ParseError on reject.

parse_interactive

Begin an interactive parse (Lark’s parse_interactive): returns immediately with an interactive::InteractiveParser holding the start state and a lazy lexer cursor over text; nothing is lexed until the caller drives it (exhaust_lexer/resume_parse/manual feeds). The returned parser borrows lexer and text for 'p. postlex, when present, is started once (its session interposes between lexer and parser, buffering the fan-out); a non-clonable session is dropped on copy().

parse_with

Parse with an explicit lexer source (04: the single shared shift/reduce loop).

re_escape

Byte-exact equivalent of CPython’s re.escape (as behaving on the pinned interpreter): escapes exactly the special-character set Python does — including space, #, &, -, ~ (escaping) — and passes everything else (incl. all non-ASCII) through untouched. Verify the exact set against the source-of-truth venv, not from memory.

read_lexer_conf

Read data.parser.lexer_conf out of a Lark tools.serialize JSON document into a LexerConf (callbacks empty, skip_validation false).

read_saved_options

Read the compile-invariant options a save recorded in data.options. A separate pass over the document, like [read_lexer_conf] — from_json_str needs maybe_placeholders BEFORE load_grammar runs, because it shapes the rules the loader builds.

regexp_matches_newline

True iff regexp (a final crate::Pattern::to_regexp output) can match text containing a newline (U+000A) — the exact newline_types predicate.

regexp_width

The static (min, max) match width of regexp in code points.

resolve_lexer

Resolve lexer='auto' and validate the parser×lexer pairing, in Lark’s evaluation order:

resolve_pending_metas

Resolve every Meta::Pending in metas in dependency order (MetaRef edges form a forest — each meta is referenced by at most one parent walk; explicit stack, so deep ?rule chains cannot recurse out). classify reads a drained slot’s host value (peek, never consume) and MUST be callable for every slot a plan references — the binding calls this only when drains guarantee those slots are filled (a crossing’s consuming drain; parse finish). Unresolvable plans (opaque residuals) demote to Meta::Poisoned, which surfaces as Empty.

to_canonical_json

Serialize a compiled grammar to the canonical envelope, as a pretty-printed (indent-2, key-sorted) string with a trailing newline — matching the corpus fixtures’ shape and byte-stable across repeated saves.

validate_dynamic_lexer_callbacks

The dynamic/dynamic_complete lexer forbids lexer_callbacks — Lark raises GrammarError("Earley's dynamic lexer doesn't support lexer_callbacks.") at Earley-parser construction, after resolve_lexer. Callers pass the resolved lexer; the check is a no-op for any non-dynamic lexer.

Type Aliases

EditTerminals

The edit_terminals hook: run over every compiled TerminalDef between compile and the priority transform. Boxed + Send so LarkOptions stays Send.

LarkInstance

The pre-rename spelling of Lark, kept so existing embedders keep compiling. Every sibling binding (Python, JS, C) already calls its entry type Lark; the core converged on the same noun.

OnError

The on_error callback type: return true to resume, false to re-raise.

SlotContribution

What one host (slot) value contributes to one endpoint of a pending span: None = lark’s _pp_get_meta skips it (not a Token/Tree/__lark_meta__, or a Tree with empty meta); Some = the container-preferred point, whose individual fields may still be unknown.

SpliceSpan

The reduce-time container span a NodeValue::Splice carries up to the parent that drains it: a (start, end) pair of (line, column, pos) triples. Lark keeps the spliced _-rule as a Tree(_rule) and its parent reads that Tree’s container_* meta; our flat Splice carries the same span here instead. None when the _-rule has no positioned raw child (an Empty _rule meta — the parent skips it). Meaningful only mid-reduce: it is None in propagate_positions=off (zero-cost) and irrelevant once a Splice is drained.

TokenCallback

A per-terminal token callback (lexer_callbacks). Arc so one callback can be held by more than one composed chain, unlike the POC.