Divergences from Lark

hyperlark implements the vast majority of lark’s features, and most programs run unchanged. This page is the detailed list of where it does not match — for the Python binding specifically.

• To migrate an existing lark project, start at Coming from Lark.
• For what each target (Rust, Python, TypeScript, C) supports, see the feature matrix.

Not implemented

Absent entirely: the reconstructor, tree templates, the standalone tool, and TextSlice inputs.

These options are recognized but not supported. Each raises a clear ConfigurationError at construction, rather than silently doing nothing:

Option	Note
strict=
cache_grammar=	cache= itself works
use_bytes=
edit_terminals=
regex=	the third-party regex module
ambiguity='forest'	'resolve' and 'explicit' work
parser='cyk'	'lalr' and 'earley' work
parser=None	lark’s lexer-only mode; build a parser and call .lex()

Lexer callbacks

lexer_callbacks= is supported, including the comment-collection recipe — the callbacks run at lex time, so they see %ignored tokens:

import hyperlark

comments = []
parser = hyperlark.Lark(r"""
    start: WORD+
    COMMENT: /#[^\n]*/
    %import common.WORD
    %ignore " "
    %ignore /\n/
    %ignore COMMENT
""", parser="lalr", lexer_callbacks={"COMMENT": comments.append})

parser.parse("hello # a note\nworld")
print(comments)          # [Token('COMMENT', '# a note')]

lark’s return contract is reproduced as-is, including its split on whether the terminal is %ignored: the callback always runs, but its result is discarded unchecked for an ignored terminal (any type is legal — that is what makes comments.append work), and must be a token for a live one, else LexError.

Three details differ:

• Returning a token whose type is not a terminal of the compiled grammar raises LexError. lark’s lexer does not validate the name, so the bogus token reaches the parser and comes back as UnexpectedToken; a hyperlark token needs a terminal id, so the failure has to happen at the callback. Declaring the name is enough to make it retypeable: a %declared terminal gets an id whether or not a rule references it.
• Position fields a returned token leaves unset fall back to the input token’s span, where lark leaves them None. So a callback that returns a bare Token('X', 'y') keeps the original span here instead of losing it.
• Under lex(text, dont_ignore=True) an %ignored terminal becomes live, so lark type-checks its callback’s return there; this bridge still discards it.

A callable lexer= (a custom lexer class) behaves as it does in lark, which depends on how it is written: one that wraps the compiled conf (BasicLexer(copy(lexer_conf)), lark’s documented shape) does run the callbacks, because the conf carries them; one that emits tokens itself never consults them, silently. Only the Earley dynamic lexer rejects them loudly.

Under the default fast_tokens=True your callback receives a FastToken. Its .type and .value are writable, as lark’s are — assign directly or build a replacement with tok.update(value=...) / type(tok).new_borrow_pos(...). A .value write becomes the emitted token’s text (an engine token has one value; lark’s tree token keeps the original text and carries the write only in its separate .value slot), and a non-str .value (lark’s lex-time coercion trick, t.update(value=int(t))) is flattened to str(value) where lark keeps the object — convert in a transformer to keep an object value.

lexer_callbacks is not available under Earley’s dynamic lexer (lark raises there too); use lexer="basic".

The parser object

parser.options is present and reports the same normalized values lark does. These attributes of lark.Lark are not available: rules, grammar, lexer_conf, ignore_tokens, and the serialize / deserialize / memo_serialize trio.

Grammar persistence is Lark.save() / Lark.load(), which do work — and cache= builds on them: constructor-level caching of the compiled parser, lark-style (LALR only).

Class identity, and compat mode

hyperlark’s Token / Tree / exception classes are its own, not lark’s. Code that only uses hyperlark never notices. Mixed-import code that also does import lark and relies on class identity (isinstance(tok, lark.Token), except lark.exceptions.UnexpectedToken) needs the optional compat mode:

pip install hyperlark[compat]

import lark
from hyperlark.compat import Lark   # returns real lark.Tree / lark.Token objects

tree = Lark(r"""
    start: WORD+
    %import common.WORD
    %ignore " "
""", parser="lalr").parse("hello world")

assert isinstance(tree.children[0], lark.Token)   # the real lark class

Installing the extra alone changes nothing — you must import from hyperlark.compat explicitly. Conversion happens after parsing, so compat mode gives up part of the speed advantage.

Compat mode is incomplete in this beta, and the gaps are silent rather than loud — check them against your usage before relying on it:

• Only parse() converts. lex(), parse_interactive() and parse_eager() return hyperlark objects, so isinstance(tok, lark.Token) is False on their results.
• compat.Lark has no open() / open_from_package(); construct from a grammar string.

Exceptions raised out of parse() are real lark.exceptions.* classes.

Recursion inside your own callbacks can crash the interpreter

An object whose attribute access reaches hyperlark — or a postlex — calling back into the parser without a base case can kill the process with SIGSEGV instead of raising RecursionError, where lark raises. Re-entering parse() itself is the exception: that path is bounded, see below.

The reason is structural, not a tuning problem. sys.setrecursionlimit() counts interpreter frames, and since 3.11 those cost almost no C stack. Each Python → hyperlark → Python round trip costs several KiB of real C stack while the counter treats it as free, so the stack runs out long before the limit trips. CPython 3.14 checks the real stack pointer and catches some of these; it does not catch all of them.

You will not hit this by accident:

• Deep input is fine — the parser core is iterative, and 200,000-deep nested input parses under both engines.
• Deep trees are fine — repr, pretty and Transformer raise a normal RecursionError exactly as lark does.
• Callback-driven parse() re-entry is bounded: a custom lexer callback that parses with the same instance in a loop raises a clean RecursionError after 8 nested levels (each level costs a native parse frame the interpreter’s own limit cannot see), on every build profile and down to the smallest thread stack CPython accepts. A transformer callback cannot get that far — a recursive parse() on a transformer= instance is refused with NotImplementedError.
• What remains is unbounded recursion in code you wrote that re-enters hyperlark through some other surface (attribute access, tree materialization); that can still exhaust the C stack, and it is much easier to reach on a small thread stack (under ~4 MiB) than on the 8 MiB main thread. Give such a callback a base case or an explicit depth counter.

Interactive parsing

parse_interactive is LALR-only, as in lark, and hyperlark additionally requires a built-in lexer. It also offers last_token, peek_next() and feed_next() beyond lark’s surface. See Interactive parsing for the full picture and the remaining differences.