Lexers

Before the parser sees your input, a lexer turns raw text into a stream of tokens. hyperlark gives you a choice of lexers, control over which terminals produce tokens, and hooks to reshape or replace the token stream entirely — the same grammar and options across Python, Rust, TypeScript, and C.

Terminals and rules

A grammar is built from two kinds of symbols:

• Terminals are UPPERCASE. They match text and produce the tokens that become the leaves of the tree. Define one with a string literal or a /regex/.
• Rules are lowercase. They combine terminals and other rules into tree nodes.

start:  NUMBER (OP NUMBER)*
OP:     "+" | "-" | "*" | "/"      // string alternatives
NUMBER: /[0-9]+/                    // a regex
%ignore " "                        // drop spaces between tokens

%ignore names a terminal (or an inline pattern) that the lexer matches and then drops — the usual home for whitespace and comments. Ignored text never reaches the parser, so it never appears in the tree.

Inspect the token stream directly with lex (ignored tokens are absent):

Python

import hyperlark as lark
p = lark.Lark(GRAMMAR, parser="lalr")
for t in p.lex("1 + 22"):
    print(t.type, repr(t.value))   # NUMBER '1'  OP '+'  NUMBER '22'

TypeScript

import { Lark } from "hyperlark";
const p = new Lark(GRAMMAR, { parser: "lalr" });
for (const t of p.lex("1 + 22")) console.log(t.type, t.value);

Rust exposes the same entry, lark.lex(text, dont_ignore), returning a Vec<Token>.

Terminal priority

When several terminals can match at the same position, the lexer keeps the longest match; ties are broken by priority (higher wins), and among equal priorities a string literal is preferred over a /regex/. Set a priority explicitly with .N after the terminal name:

NAME:  /[a-z]+/
IF.2:  "if"        // "if" matches NAME too — the higher priority makes it IF

Priorities are 32-bit integers. A magnitude beyond the i32 range is rejected at build time rather than silently truncated (see the feature matrix).

keep_all_tokens

By default the tree drops tokens that carry no information — filtered anonymous literals like "+" and punctuation. Set keep_all_tokens to retain every token as a child. It is a compile option available on all four targets:

Python

p = lark.Lark(GRAMMAR, parser="lalr", keep_all_tokens=True)

Rust

use hyperlark::{Lark, LarkOptions, ParserKind};
let p = Lark::from_lark_source(
    GRAMMAR,
    LarkOptions { parser: ParserKind::Lalr, keep_all_tokens: true, ..Default::default() },
)?;

TypeScript

const p = new Lark(GRAMMAR, { parser: "lalr", keep_all_tokens: true });

C

Lark *p = NULL;
lark_from_source(GRAMMAR, strlen(GRAMMAR), LARK_OPT_KEEP_ALL_TOKENS, &p);

Choosing a lexer

lexer="auto" (the default) picks the right lexer for your parser:

Parser	auto resolves to	Also available
LALR	contextual	basic
Earley	dynamic	basic, dynamic_complete

• basic — one context-free tokenization pass; works under either parser.
• contextual (LALR only) — the lexer consults parser state to decide which terminals are legal next, so overlapping tokens (a keyword vs. an identifier) disambiguate by position.
• dynamic / dynamic_complete (Earley only) — the lexer explores the token possibilities the parse can still accept; dynamic_complete is the exhaustive variant.

Set it explicitly when you want to override auto:

Python

p = lark.Lark(GRAMMAR, parser="earley", lexer="dynamic")

Rust

use hyperlark::{Lark, LarkOptions, LexerSpec, ParserKind};
let p = Lark::from_lark_source(
    GRAMMAR,
    LarkOptions { parser: ParserKind::Earley, lexer: LexerSpec::Dynamic, ..Default::default() },
)?;

TypeScript

const p = new Lark(GRAMMAR, { parser: "earley", lexer: "dynamic" });

The C surface is LALR-only and always uses the built-in (basic/contextual) lexer — dynamic is not available there.

Indentation: postlex and Indenter

Indentation-sensitive languages (Python-like block syntax) need INDENT and DEDENT tokens that a plain lexer cannot produce. A postlex sits between the lexer and the parser and rewrites the token stream; Indenter is the built-in postlex that emits _INDENT / _DEDENT by tracking column depth (and ignoring newlines inside brackets). The indent tokens are %declared — injected by the indenter, never lexed:

?start: _NEWLINE* tree
tree: NAME _NEWLINE [_INDENT tree+ _DEDENT]
NAME: /\w+/
_NEWLINE: /(\r?\n[\t ]*)+/
%declare _INDENT _DEDENT
%import common.WS_INLINE
%ignore WS_INLINE

Python

from hyperlark import Lark, PythonIndenter
p = Lark(GRAMMAR, parser="lalr", postlex=PythonIndenter())
p.parse("root\n    branch\n        leaf\n")

Rust

use hyperlark::{Indenter, Lark, LarkOptions, ParserKind};
let indenter = Indenter {
    nl_type: "_NEWLINE".into(),
    open_paren_types: vec!["LPAR".into(), "LSQB".into(), "LBRACE".into()],
    close_paren_types: vec!["RPAR".into(), "RSQB".into(), "RBRACE".into()],
    indent_type: "_INDENT".into(),
    dedent_type: "_DEDENT".into(),
    tab_len: 8,
};
let p = Lark::from_lark_source(
    GRAMMAR,
    LarkOptions { parser: ParserKind::Lalr, postlex: Some(Box::new(indenter)), ..Default::default() },
)?;

TypeScript

// The Python-style indenter is a built-in preset.
const p = new Lark(GRAMMAR, { parser: "lalr", indenter: "python" });
p.parse("root\n    branch\n        leaf\n");

Python and Rust accept any postlex (subclass Indenter, or implement the PostLex / Postlex interface); TypeScript ships the indenter: "python" preset. Postlex is not available in the C binding.

A postlex composes with the contextual lexer: its process is resumed between parser steps, so each raw token is still lexed against the live parser state and lexer="contextual" keeps working. One caveat is inherent to the design rather than to any binding — a postlexer that reads ahead (pulling the next token before yielding the current one) outruns the parser, so terminals that only disambiguate contextually can fail for it. Holding a token conditionally, as most postlexers do, is unaffected.

Custom lexers

To tokenize by rules a grammar cannot express, supply your own token stream. A custom lexer is called once per parse and pulled one token at a time, interleaved with the parser. The built-in scanner is exposed (BasicLexer / BasicTokenSource) so you can wrap and decorate it rather than start from scratch:

Python

from hyperlark import Lark
from hyperlark.lexer import BasicLexer, Lexer

class MyLexer(Lexer):
    def __init__(self, lexer_conf):
        self.lexer = BasicLexer(lexer_conf)          # reuse the built-in scan

    def lex(self, lexer_state, parser_state):
        for tok in self.lexer.lex(lexer_state, parser_state):
            yield tok                                # inspect / reshape / drop here

    __future_interface__ = 2

p = Lark(GRAMMAR, parser="lalr", lexer=MyLexer)

Rust

use hyperlark::{CustomLexError, CustomToken, Lark, LarkOptions, LexerSpec, ParserKind, TokenSource};
use std::sync::Mutex;

struct MySource { /* your cursor over the input */ }

impl TokenSource for MySource {
    fn next_token(&mut self, _state: Option<usize>, _expects: Option<&[u32]>)
        -> Result<Option<CustomToken>, CustomLexError>
    {
        // Ok(None) ends the stream; otherwise yield the next token:
        Ok(Some(CustomToken {
            type_name: "NUMBER".into(), value: "42".into(),
            start_pos: Some(0), end_pos: Some(2),
            line: Some(1), end_line: Some(1),
            column: Some(1), end_column: Some(3),
        }))
    }
}

let p = Lark::from_lark_source(
    GRAMMAR,
    LarkOptions { parser: ParserKind::Lalr, lexer: LexerSpec::Custom, ..Default::default() },
)?;
let src = Mutex::new(MySource { /* ... */ });
let tree = p.parse_custom(&src, None, None)?;

TypeScript

import { Lark, BasicLexer } from "hyperlark";

class MyLexer {
  constructor(conf) { this.lexer = new BasicLexer(conf); }
  *lex(input, parserState) {
    for (const tok of this.lexer.lex(input)) yield tok;   // reshape tokens here
  }
}
const p = new Lark(GRAMMAR, { parser: "lalr", lexer: MyLexer });

Custom lexers are not available in the C binding.

Per-target support

• Python, TypeScript, Rust — all lexers (basic, contextual, dynamic, dynamic_complete), postlex / indenter, and custom lexers.
• C — LALR with the built-in basic/contextual lexer only: no dynamic, no postlex, no custom lexer. %ignore, terminal priority, and keep_all_tokens are grammar/compile features and work everywhere.

hyperlark implements Lark’s lexer semantics natively. See the feature matrix for the full per-target breakdown.