Meaningful indentation?

[tslocke]

Hi, I'm working on a DSL that uses indentation to delimit block structure, like Python or Haskell. Is that something Langium can cope with?

I see you are using Chevrotain under the hood - my current implementation is also in Chevrotain, and makes use of the pattern matching callbacks in the lexer.

I've been putting off writing a VS Code editing mode, and if I understand right, Langium could make that a lot easier. My understanding was that VC Code syntax highlighting is driven not by a full parser but by a bunch of regexes, like in the old TextMate modes. Is that something Langium can derive from the grammar? Seems difficult!

[msujew]

Hi @tslocke, you're getting into uncharted waters here, as we haven't tried that out yet, but as long as it's possible in Chevrotain, it should be possible in Langium as well.

By default, Langium will create relatively simple TokenType objects from each keyword and terminal that's found in the Langium grammar definition of your language. All of them go through the TokenBuilder service, where you can fine-tune each TokenType object. You would probably need to create dummy INDENT and DEDENT tokens, reference those in your parser rules and add a pattern matching callback to them in your own TokenBuilder implementation.

My understanding was that VC Code syntax highlighting is driven not by a full parser but by a bunch of regexes, like in the old TextMate modes.

vscode syntax highlighting is twofold: For one we can simply let vscode do syntactical highlighting, which is provided based on TextMate grammars. Additionally, the language server protocol also offers semantic highlighting, which enables far more fine-grained control over your highlighting.

Is that something Langium can derive from the grammar? Seems difficult!

We actually already generate TextMate grammars from Langium grammars, see the textmate-generator.ts. It suffices for a lot of use cases, but isn't recommended for production use of more complicated languages, as these probably require a bit fine tuning. It does these kinds of highlighting all by itself:

• Keywords
• Comments
• Strings

All the TextMate grammars for our example languages are generated using the provided generator.

Semantic highlighting can be implemented using the AbstractSemanticTokenProvider and allows to perform highlighting based on AST info.

[msujew]

@tslocke By the way the Chevrotain documentation contains an example on how one would go on about python style indentation lexing using token callbacks.

[tslocke]

Thanks for the advice. Yes I already have it working in chevrotain based on that example. I think I'm firmly in the realm of "more complicated languages" in terms of generating a useful textmate grammar.

[aabounegm]

@msujew Great! Thank you very much for the reference, seems like I simply had an incomplete RegEx and an issue with one if condition 😁.
Anyway, seeing as this issue is relatively common, I would like to contribute my solution so others don't have to copy the same Token Builder and Lexer to their projects. Would you prefer a PR to the Langium repo (simply adding the new implementation to be imported by users who need them) or a separate npm module with these 2 classes?

[msujew]

Sure, a PR would be appreciated, thanks! See also #1016 (comment) for a bit of inspiration on the API of the feature. You can directly link the issue in your PR, since that would effectively close the issue.

[aabounegm]

@msujew I submitted a PR now at #1578. Hope it helps

[aabounegm]

The IndentationAwareTokenBuilder has been released in v3.2.0, and there is now a recipe explaining how to use it.